[GitHub] spark issue #14897: [SPARK-17338][SQL] add global temp view

2016-09-01 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14897 @srinathshankar also want to review this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14911: [SPARK-17355] Workaround for HIVE-14684 / HiveResultSetM...

2016-09-01 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14911 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #14915: [SPARK-17356][SQL] Fix out of memory issue when g...

2016-09-01 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14915#discussion_r77265668 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala --- @@ -274,6 +274,13 @@ abstract class QueryTest extends PlanTest { val

[GitHub] spark pull request #14915: [SPARK-17356][SQL] Fix out of memory issue when g...

2016-09-01 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14915#discussion_r77265626 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala --- @@ -274,6 +274,13 @@ abstract class QueryTest extends PlanTest { val

[GitHub] spark issue #14915: [SPARK-17356][SQL] Fix out of memory issue when generati...

2016-09-01 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14915 btw, we also need to merge it to branch 1.6, which also have toJSON (https://github.com/apache/spark/blob/branch-1.6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala#L533

[GitHub] spark pull request #14915: [SPARK-17356][SQL] Fix out of memory issue when g...

2016-09-01 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14915#discussion_r77217091 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala --- @@ -274,6 +274,13 @@ abstract class QueryTest extends PlanTest { val

[GitHub] spark pull request #14915: [SPARK-17356][SQL] Fix out of memory issue when g...

2016-09-01 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14915#discussion_r77216938 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala --- @@ -274,6 +274,13 @@ abstract class QueryTest extends PlanTest { val

[GitHub] spark pull request #14915: [SPARK-17356][SQL] Fix out of memory issue when g...

2016-09-01 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14915#discussion_r77216735 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala --- @@ -604,6 +604,7 @@ abstract class TreeNode[BaseType <: TreeN

[GitHub] spark pull request #14876: showcase, DO NOT MERGE

2016-08-30 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14876#discussion_r76900104 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/MergePartialAggregate.scala --- @@ -0,0 +1,55 @@ +/* + * Licensed

[GitHub] spark issue #10896: [SPARK-12978][SQL] Skip unnecessary final group-by when ...

2016-08-30 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/10896 @maropu Thank you for working on this. Sorry that I did not get time to look at it after you updated the pr. I looked at it today. I think this optimization deserves a feature flag since

[GitHub] spark pull request #14850: [SPARK-17279][SQL] better error message for NPE d...

2016-08-27 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14850#discussion_r76518705 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala --- @@ -1057,5 +1057,17 @@ case class ScalaUDF

[GitHub] spark issue #14846: [SPARK-17274][SQL] Move join optimizer rules into a sepa...

2016-08-26 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14846 lgtm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #14845: [SPARK-17273][SQL] Move expression optimizer rules into ...

2016-08-26 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14845 lgtm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #14844: [SPARK-17272][SQL] Move subquery optimizer rules into it...

2016-08-26 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14844 lgtm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #14839: [SPARK-17270][SQL] Move object optimization rules into i...

2016-08-26 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14839 Seems the conflicts are caused by https://github.com/apache/spark/commit/d063898bebaaf4ec2aad24c3ac70aabdbf97a190#diff-a636a87d8843eeccca90140be91d4faf. --- If your project is set up for it, you can

[GitHub] spark issue #14839: [SPARK-17270][SQL] Move object optimization rules into i...

2016-08-26 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14839 hmm. There are conflicts in branch 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

spark git commit: [SPARK-17270][SQL] Move object optimization rules into its own file

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master a6bca3ad0 -> cc0caa690 [SPARK-17270][SQL] Move object optimization rules into its own file ## What changes were proposed in this pull request? As part of breaking Optimizer.scala apart, this patch moves various Dataset object optimization

[GitHub] spark issue #14838: [SPARK-17269][SQL] Move finish analysis optimization sta...

2016-08-26 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14838 LGTM. Merging to master and branch 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14839: [SPARK-17270][SQL] Move object optimization rules into i...

2016-08-26 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14839 LGTM. Merging to master and branch 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

spark git commit: [SPARK-17266][TEST] Add empty strings to the regressionTests of PrefixComparatorsSuite

2016-08-26 Thread yhuai
y. But, let's this test case in the regressionTests. Author: Yin Huai <yh...@databricks.com> Closes #14837 from yhuai/SPARK-17266. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a6bca3ad Tree: http://git-wip-us.apache.org/r

[GitHub] spark issue #14837: [SPARK-17266] [TEST] Add empty strings to the regression...

2016-08-26 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14837 Thanks. I will merge this and watch the build. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

spark git commit: [SPARK-17244] Catalyst should not pushdown non-deterministic join conditions

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/branch-2.0 dfdfc3092 -> 9c0ac6b53 [SPARK-17244] Catalyst should not pushdown non-deterministic join conditions ## What changes were proposed in this pull request? Given that non-deterministic expressions can be stateful, pushing them down the

spark git commit: [SPARK-17244] Catalyst should not pushdown non-deterministic join conditions

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master f64a1ddd0 -> 540e91280 [SPARK-17244] Catalyst should not pushdown non-deterministic join conditions ## What changes were proposed in this pull request? Given that non-deterministic expressions can be stateful, pushing them down the query

[GitHub] spark issue #14815: [SPARK-17244] Catalyst should not pushdown non-determini...

2016-08-26 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14815 LGTM. Merging to master and branch 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #14837: [SPARK-17266] [TEST] Add empty strings to the reg...

2016-08-26 Thread yhuai
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/14837 [SPARK-17266] [TEST] Add empty strings to the regressionTests of PrefixComparatorsSuite ## What changes were proposed in this pull request? This PR adds a regression test

spark git commit: [SPARK-17250][SQL] Remove HiveClient and setCurrentDatabase from HiveSessionCatalog

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master fd4ba3f62 -> 261c55dd8 [SPARK-17250][SQL] Remove HiveClient and setCurrentDatabase from HiveSessionCatalog ### What changes were proposed in this pull request? This is the first step to remove `HiveClient` from `HiveSessionState`. In the

[GitHub] spark issue #14821: [SPARK-17250] [SQL] Remove HiveClient and setCurrentData...

2016-08-26 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14821 I like this. We do not need to set db to hive client. I am merging this to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

spark git commit: [SPARK-17192][SQL] Issue Exception when Users Specify the Partitioning Columns without a Given Schema

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 188321623 -> fd4ba3f62 [SPARK-17192][SQL] Issue Exception when Users Specify the Partitioning Columns without a Given Schema ### What changes were proposed in this pull request? Address the comments by yhuai in the original PR: ht

[GitHub] spark issue #14572: [SPARK-17192] [SQL] Issue Exception when Users Specify t...

2016-08-26 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14572 LGTM. Thanks. Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

spark git commit: [SPARK-17187][SQL][FOLLOW-UP] improve document of TypedImperativeAggregate

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 28ab17922 -> 970ab8f6d [SPARK-17187][SQL][FOLLOW-UP] improve document of TypedImperativeAggregate ## What changes were proposed in this pull request? improve the document to make it easier to understand and also mention window operator.

[GitHub] spark issue #14822: [SPARK-17187][SQL][follow-up] improve document of TypedI...

2016-08-26 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14822 LGTM. Merging to master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

spark git commit: [SPARK-17260][MINOR] move CreateTables to HiveStrategies

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 6063d5963 -> 28ab17922 [SPARK-17260][MINOR] move CreateTables to HiveStrategies ## What changes were proposed in this pull request? `CreateTables` rule turns a general `CreateTable` plan to `CreateHiveTableAsSelectCommand` for hive serde

[GitHub] spark issue #14825: [SPARK-17260][minor] move CreateTables to HiveStrategies

2016-08-26 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14825 LGTM. Merging to master. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

spark git commit: [SPARK-17187][SQL] Supports using arbitrary Java object as internal aggregation buffer object

2016-08-25 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 9b5a1d1d5 -> d96d15156 [SPARK-17187][SQL] Supports using arbitrary Java object as internal aggregation buffer object ## What changes were proposed in this pull request? This PR introduces an abstract class `TypedImperativeAggregate` so

[GitHub] spark issue #14753: [SPARK-17187][SQL] Supports using arbitrary Java object ...

2016-08-25 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14753 Thanks. Overall looks good. I am merging this to master. Let me tweak the interface later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #14816: [SPARK-17245] [SQL] [BRANCH-1.6] Do not rely on Hive's s...

2016-08-25 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14816 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #14816: [SPARK-17245] [SQL] [BRANCH-1.6] Do not rely on Hive's s...

2016-08-25 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14816 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #14816: [SPARK-17245] [SQL] [BRANCH-1.6] Do not rely on H...

2016-08-25 Thread yhuai
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/14816 [SPARK-17245] [SQL] [BRANCH-1.6] Do not rely on Hive's session state to retrieve HiveConf ## What changes were proposed in this pull request? Right now, we rely on Hive's `SessionState.get

[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

2016-08-25 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14537#discussion_r76305079 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -237,21 +237,27 @@ private[hive] class HiveMetastoreCatalog

[GitHub] spark pull request #13852: [SPARK-16200][SQL] Rename AggregateFunction#suppo...

2016-08-25 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/13852#discussion_r76291119 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala --- @@ -45,7 +45,7 @@ abstract class Collect extends

[GitHub] spark issue #14753: [SPARK-17187][SQL] Supports using arbitrary Java object ...

2016-08-25 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14753 @hvanhovell This is supposed to work with window functions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-25 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76264528 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/TypedImperativeAggregateSuite.scala --- @@ -0,0 +1,300 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-25 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76263947 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,144 @@ abstract class

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76181435 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/TypedImperativeAggregateSuite.scala --- @@ -0,0 +1,300 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76176167 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/TypedImperativeAggregateSuite.scala --- @@ -0,0 +1,300 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76175290 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregationIterator.scala --- @@ -234,7 +234,24 @@ abstract class

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76158137 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,175 @@ abstract class

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76157178 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/TypedImperativeAggregateSuite.scala --- @@ -0,0 +1,261 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76156969 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/TypedImperativeAggregateSuite.scala --- @@ -0,0 +1,261 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76155973 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/TypedImperativeAggregateSuite.scala --- @@ -0,0 +1,261 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76154134 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregationIterator.scala --- @@ -234,7 +234,20 @@ abstract class

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76154000 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,144 @@ abstract class

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76152853 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,144 @@ abstract class

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76001378 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/TypedImperativeAggregateSuite.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76000196 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/TypedImperativeAggregateSuite.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r76000216 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/TypedImperativeAggregateSuite.scala --- @@ -0,0 +1,234 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r75999634 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,153 @@ abstract class

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-24 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r75999559 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,146 @@ abstract class

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-23 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r75997361 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortBasedAggregationIterator.scala --- @@ -131,6 +150,11 @@ class

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-23 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r75997312 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortBasedAggregationIterator.scala --- @@ -131,6 +150,11 @@ class

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-23 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r75994509 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,175 @@ abstract class

[GitHub] spark issue #14014: [SPARK-16344][SQL] Decoding Parquet array of struct with...

2016-08-23 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14014 @deepakmk9 For now, you need to rename the field. As long as it is not called `element`, it should work. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-22 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r75776471 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,148 @@ abstract class

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-22 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14753#discussion_r75776503 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,148 @@ abstract class

[GitHub] spark issue #14572: [SPARK-17192] [SQL] Issue Exception when Users Specify t...

2016-08-22 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14572 LGTM. I left one comment about the error message --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #14572: [SPARK-17192] [SQL] Issue Exception when Users Sp...

2016-08-22 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14572#discussion_r75774405 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala --- @@ -72,29 +72,19 @@ case class PreprocessDDL(conf: SQLConf

spark git commit: [MINOR][SQL] Fix some typos in comments and test hints

2016-08-22 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 6f3cd36f9 -> 929cb8bee [MINOR][SQL] Fix some typos in comments and test hints ## What changes were proposed in this pull request? Fix some typos in comments and test hints ## How was this patch tested? N/A. Author: Sean Zhong

[GitHub] spark issue #14755: [MINOR][SQL] Fix some typos in comments and test hints

2016-08-22 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14755 lgtm. Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #14572: [SPARK-16552] [FOLLOW-UP] [SQL] Store the Inferre...

2016-08-22 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14572#discussion_r75743457 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala --- @@ -72,29 +72,19 @@ case class PreprocessDDL(conf: SQLConf

[GitHub] spark issue #14732: [SPARK-16320] [DOC] Document G1 heap region's effect on ...

2016-08-22 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14732 ah sorry... Feel free to send out a small pr to add the link back :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

spark git commit: [SPARK-16320][DOC] Document G1 heap region's effect on spark 2.0 vs 1.6

2016-08-22 Thread yhuai
Repository: spark Updated Branches: refs/heads/branch-2.0 79195982a -> 94eff0875 [SPARK-16320][DOC] Document G1 heap region's effect on spark 2.0 vs 1.6 ## What changes were proposed in this pull request? Collect GC discussion in one section, and documenting findings about G1 GC heap region

spark git commit: [SPARK-16320][DOC] Document G1 heap region's effect on spark 2.0 vs 1.6

2016-08-22 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 209e1b3c0 -> 342278c09 [SPARK-16320][DOC] Document G1 heap region's effect on spark 2.0 vs 1.6 ## What changes were proposed in this pull request? Collect GC discussion in one section, and documenting findings about G1 GC heap region

[GitHub] spark issue #14732: [SPARK-16320] [DOC] Document G1 heap region's effect on ...

2016-08-22 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14732 LGTM. Merging to master and branch 2.0. Thanks @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #14732: [SPARK-16320] [DOC] Document G1 heap region's eff...

2016-08-22 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14732#discussion_r75728831 --- Diff: docs/tuning.md --- @@ -217,14 +204,22 @@ temporary objects created during task execution. Some steps which may be useful * Check

[GitHub] spark pull request #14732: [SPARK-16320] [DOC] Document G1 heap region's eff...

2016-08-22 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14732#discussion_r75722532 --- Diff: docs/tuning.md --- @@ -122,21 +122,8 @@ large records. `R` is the storage space within `M` where cached blocks immune to being evicted

[GitHub] spark pull request #14732: [SPARK-16320] [DOC] Document G1 heap region's eff...

2016-08-22 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14732#discussion_r75722287 --- Diff: docs/tuning.md --- @@ -217,14 +204,22 @@ temporary objects created during task execution. Some steps which may be useful * Check

[GitHub] spark pull request #14732: [SPARK-16320] [DOC] Document G1 heap region's eff...

2016-08-22 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14732#discussion_r75722253 --- Diff: docs/tuning.md --- @@ -217,14 +204,22 @@ temporary objects created during task execution. Some steps which may be useful * Check

[GitHub] spark issue #14572: [SPARK-16552] [FOLLOW-UP] [SQL] Store the Inferred Schem...

2016-08-22 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14572 sorry. I missed this PR. Can you update? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[1/2] spark git commit: [SPARK-16498][SQL] move hive hack for data source table into HiveExternalCatalog

2016-08-21 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 91c239768 -> b2074b664 http://git-wip-us.apache.org/repos/asf/spark/blob/b2074b66/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala -- diff

[2/2] spark git commit: [SPARK-16498][SQL] move hive hack for data source table into HiveExternalCatalog

2016-08-21 Thread yhuai
[SPARK-16498][SQL] move hive hack for data source table into HiveExternalCatalog ## What changes were proposed in this pull request? Spark SQL doesn't have its own meta store yet, and use hive's currently. However, hive's meta store has some limitations(e.g. columns can't be too many, not

[GitHub] spark issue #14155: [SPARK-16498][SQL] move hive hack for data source table ...

2016-08-21 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14155 LGTM. There are two things that we need to address in follow-up prs. The first one is if we can consolidate location in CatalogStorageFormat and `path` in options. The second one is to read the conf

[GitHub] spark pull request #14155: [SPARK-16498][SQL] move hive hack for data source...

2016-08-21 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14155#discussion_r75617761 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -200,22 +375,77 @@ private[spark] class HiveExternalCatalog

[GitHub] spark issue #14723: [SQL][WIP][Test] Supports object-based aggregation funct...

2016-08-20 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14723 Can you create a jira? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586776 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/AggregateWithObjectAggregateBufferSuite.scala --- @@ -0,0 +1,156 @@ +/* + * Licensed

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586764 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586760 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586661 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586622 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586350 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586238 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586232 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586233 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586183 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14155: [SPARK-16498][SQL] move hive hack for data source...

2016-08-19 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14155#discussion_r75523912 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -200,22 +375,77 @@ private[spark] class HiveExternalCatalog

[GitHub] spark pull request #14155: [SPARK-16498][SQL] move hive hack for data source...

2016-08-19 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14155#discussion_r75523842 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -200,22 +375,77 @@ private[spark] class HiveExternalCatalog

[GitHub] spark pull request #14155: [SPARK-16498][SQL] move hive hack for data source...

2016-08-19 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14155#discussion_r75432710 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -264,10 +261,8 @@ case class AlterTableUnsetPropertiesCommand

[GitHub] spark pull request #14155: [SPARK-16498][SQL] move hive hack for data source...

2016-08-19 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14155#discussion_r75432300 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -144,16 +163,172 @@ private[spark] class HiveExternalCatalog

[GitHub] spark pull request #14155: [SPARK-16498][SQL] move hive hack for data source...

2016-08-19 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14155#discussion_r75432051 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala --- @@ -985,35 +987,37 @@ class MetastoreDataSourcesSuite extends

[GitHub] spark pull request #14155: [SPARK-16498][SQL] move hive hack for data source...

2016-08-19 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14155#discussion_r75432071 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala --- @@ -49,6 +49,8 @@ class MetastoreDataSourcesSuite extends

[GitHub] spark pull request #14155: [SPARK-16498][SQL] move hive hack for data source...

2016-08-19 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14155#discussion_r75432039 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala --- @@ -985,35 +987,37 @@ class MetastoreDataSourcesSuite extends

<    2   3   4   5   6   7   8   9   10   11   >