[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155858138 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155857955 **[Test build #45645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45645/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155858141 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155868423 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread aray
Github user aray commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44545811 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala --- @@ -385,6 +385,20 @@ case class Rollup(

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44546180 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala --- @@ -385,6 +385,20 @@ case class Rollup(

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155819695 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155819728 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155820135 **[Test build #45645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45645/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread aray
Github user aray commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155985411 @rxin sure I'll put together a PR for the python API tonight --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155969944 @aray do you want to submit a pull request for python api too? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/7841 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155954990 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155954842 **[Test build #45673 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45673/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155954992 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155955761 Thanks! Merging to master and branch 1.6. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155868392 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread aray
Github user aray commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44566886 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/GroupedData.scala --- @@ -273,6 +280,60 @@ class GroupedData protected[sql]( def sum(colNames:

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread aray
Github user aray commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44572982 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -248,6 +253,43 @@ class Analyzer( } }

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread aray
Github user aray commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155871674 @yhuai RE your questions (3 was already addressed above): >1. Should we always ask users to provide pivot values? The argument for not requiring values I think is

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44571373 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -248,6 +253,43 @@ class Analyzer( } }

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155870575 **[Test build #45659 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45659/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44571747 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/GroupedData.scala --- @@ -273,6 +280,60 @@ class GroupedData protected[sql]( def sum(colNames:

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread aray
Github user aray commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155916926 @yhuai I think this addresses everything we discussed, let me know if I missed anything or if there is anything else I can do. Again, thanks for the code review. --- If

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44573504 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -248,6 +253,43 @@ class Analyzer( } }

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155889032 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155888949 **[Test build #45659 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45659/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155889034 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155916465 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155916497 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155918774 **[Test build #45673 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45673/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155931475 LGTM pending jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-10 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155594566 **[Test build #45565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45565/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155595966 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155595968 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-10 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155595961 **[Test build #45565 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45565/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-10 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155596856 To give a status update -- I'm hoping to merge this this week. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-10 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44489682 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -247,6 +252,38 @@ class Analyzer( } }

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-10 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44489613 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -247,6 +252,38 @@ class Analyzer( } }

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-10 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44500250 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -248,6 +253,38 @@ class Analyzer( } }

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-10 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44500328 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -248,6 +253,38 @@ class Analyzer( } }

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-10 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44500352 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -248,6 +253,38 @@ class Analyzer( } }

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-10 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155661297 @aray This is very cool! Here are a few things I'd like to discuss. 1. Should we always ask users to provide pivot values? 2. For the type of pivot values, should

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155069845 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155069873 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155070539 **[Test build #45366 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45366/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155204885 @aray I talked to a few more people about this. Most like the 2nd API more (groupBy.pivot.agg). I think it'd also be better to remove the requirement to specify

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155211191 BTW we can also later add a variant that allows users to specify values directly, in order to avoid materializing the intermediate data. --- If your project is set up

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155255172 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155270117 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155270096 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155295277 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155270221 **[Test build #45486 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45486/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155295081 **[Test build #45486 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45486/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155295276 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155105052 **[Test build #45366 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45366/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155105393 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155255175 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155255062 **[Test build #45432 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45432/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155223213 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread aray
Github user aray commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155223109 @rxin Updated, the values are now optional. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44348066 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala --- @@ -989,6 +989,41 @@ class DataFrame private[sql]( } /** + *

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread aray
Github user aray commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44352381 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala --- @@ -989,6 +989,41 @@ class DataFrame private[sql]( } /** + *

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155223235 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155224361 **[Test build #45432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45432/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-154858826 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-154858835 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-154859147 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-154859143 **[Test build #45316 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45316/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-154859006 **[Test build #45316 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45316/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-06 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-154477941 @aray sorry was away for spark summit - back now and will get to this today. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-152343672 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-152343703 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-152344322 **[Test build #44643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44643/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-152362042 **[Test build #44643 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44643/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-152362142 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-152362138 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-23 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-150663288 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-150664411 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-150664375 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-150665042 **[Test build #44249 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44249/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-23 Thread aray
Github user aray commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-150620321 @rxin here is my summary of other frameworks API's I'm going to use an example dataset form the pandas doc for all the examples (as df) |A|B|C|D|

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-23 Thread aray
Github user aray commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-150745807 @rxin, Not requiring the values would necessitate doing a separate query for the distinct values of the column before the pivot query. It looks like at least some DF

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-23 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-150713518 I like your 2nd interface more (group by and then pivot), since it is easier to get that working for both Java and Scala. We can implement a simpler interface for Python/R

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-150691855 **[Test build #44249 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44249/consoleFull)** for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-150691984 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-150691985 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-23 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-150511647 @aray Thanks a lot for updating this. To help api design, can you take a look at other frameworks and see what their signatures look like? --- If your project is set up

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-22 Thread aray
Github user aray commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-150464038 @rxin and @JoshRosen, this is ready for review now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-10-20 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-149657987 @rxin, do you want to revisit this now for 1.6? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-08-11 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-130176629 @aray FYI this didn't make it into the 1.5 release (was submitted too close to the feature freeze deadline), but we will try to include it in Spark 1.6. --- If your

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-08-03 Thread aray
Github user aray commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-127265753 @rxin it looks like Jenkins forgot about building this. Can you help trigger the build again? --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-127408423 [Test build #1319 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1319/console) for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-127404566 [Test build #1319 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1319/consoleFull) for PR 7841 at commit

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-07-31 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-126860589 Jenkins, ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-126810607 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-126811200 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-07-31 Thread aray
GitHub user aray opened a pull request: https://github.com/apache/spark/pull/7841 [SPARK-8992] [SQL] Add pivot to dataframe api This adds a pivot method to the dataframe api. Following the lead of cube and rollup this adds a Pivot operator that is translated into an