[GitHub] spark issue #15807: [SPARK-18147][SQL] do not fail for very complex aggregat...

2016-11-08 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/15807 For wholestage codegen, I think that a life time of sub-expressions is within an iteration for a row. Thus, `isInitialized` and `subExpr1` should be initialized at the beginning of each iteration.

[GitHub] spark issue #15822: [Minor][PySpark] Improve error message when running PySp...

2016-11-08 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15822 yeah, as per the discussion at https://github.com/apache/spark/pull/15659#issuecomment-259157516. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #13909: [SPARK-16213][SQL] Reduce runtime overhead of a program ...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13909 **[Test build #68396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68396/consoleFull)** for PR 13909 at commit

[GitHub] spark issue #15822: [Minor][PySpark] Improve error message when running PySp...

2016-11-08 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15822 Thanks - did this come from a discussion somewhere? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #15816: SPARK-18368: Fix regexp_replace with task seriali...

2016-11-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15816 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #15816: SPARK-18368: Fix regexp_replace with task serialization.

2016-11-08 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15816 I'm surprised too that we haven't caught this earlier ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15816: SPARK-18368: Fix regexp_replace with task serialization.

2016-11-08 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15816 Merging in master/branch-2.1/branch-2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15823: [SPARK-18191][CORE][FOLLOWUP] Call `setConf` if `OutputF...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15823 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15823: [SPARK-18191][CORE][FOLLOWUP] Call `setConf` if `OutputF...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15823 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68391/ Test PASSed. ---

[GitHub] spark issue #15823: [SPARK-18191][CORE][FOLLOWUP] Call `setConf` if `OutputF...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15823 **[Test build #68391 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68391/consoleFull)** for PR 15823 at commit

[GitHub] spark issue #15627: [SPARK-18099][YARN] Fail if same files added to distribu...

2016-11-08 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/15627 @kishorvpatil Thank you for fixing this! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15807: [SPARK-18147][SQL] do not fail for very complex aggregat...

2016-11-08 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15807 even we modify it to hold the results of subexpressions in member variables, the above code example should not work under wholestage codegen. The above code example is similar to non

[GitHub] spark issue #15807: [SPARK-18147][SQL] do not fail for very complex aggregat...

2016-11-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15807 why whole stage codegen can't use member variables to keep the result of subexpression? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request #15799: [SPARK-18333] [SQL] Revert hacks in parquet and o...

2016-11-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15799 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #15823: [SPARK-18191][CORE][FOLLOWUP] Call `setConf` if `OutputF...

2016-11-08 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15823 LGTM. Not at my laptop, would be great if you can merge @rxin, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #15799: [SPARK-18333] [SQL] Revert hacks in parquet and orc read...

2016-11-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15799 merging to master/2.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15825: [SPARK-18377][SQL] warehouse path should be a static con...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15825 **[Test build #68395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68395/consoleFull)** for PR 15825 at commit

[GitHub] spark issue #15807: [SPARK-18147][SQL] do not fail for very complex aggregat...

2016-11-08 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15807 > isn't the result of subexpression kept in member variables? For non-wholestage codegen, yes. For wholestage codegen, no. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request #15797: [SPARK-17990][SPARK-18302][SQL] correct several p...

2016-11-08 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15797#discussion_r87141007 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -810,13 +825,44 @@ private[spark] class

[GitHub] spark issue #15799: [SPARK-18333] [SQL] Revert hacks in parquet and orc read...

2016-11-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15799 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #15233: [SPARK-17659] [SQL] Partitioned View is Not Supported By...

2016-11-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15233 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #15825: [SPARK-18377][SQL] warehouse path should be a static con...

2016-11-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15825 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15807: [SPARK-18147][SQL] do not fail for very complex aggregat...

2016-11-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15807 isn't the result of subexpression kept in member variables? What I am talking about is something like: ``` private boolean isInitialized = false; private Int subExpr1 = 0;

[GitHub] spark issue #15807: [SPARK-18147][SQL] do not fail for very complex aggregat...

2016-11-08 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15807 E.g., if (isNull(subexpr)) { ... } else { AssertNotNull(subexpr) // subexpr2, first used. SomeExpr(AssertNotNull(subexpr)) //

[GitHub] spark issue #15807: [SPARK-18147][SQL] do not fail for very complex aggregat...

2016-11-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15807 > we can't access the subexpression outside later I don't quite understand it, can you give an example? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #15824: [SPARK-18376][SQL] Skip subexpression elimination for co...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15824 **[Test build #68394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68394/consoleFull)** for PR 15824 at commit

[GitHub] spark issue #15824: [SPARK-18376][SQL] Skip subexpression elimination for co...

2016-11-08 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15824 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #15824: [SPARK-18376][SQL] Skip subexpression elimination for co...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15824 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15824: [SPARK-18376][SQL] Skip subexpression elimination for co...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15824 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68393/ Test FAILed. ---

[GitHub] spark issue #15824: [SPARK-18376][SQL] Skip subexpression elimination for co...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15824 **[Test build #68393 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68393/consoleFull)** for PR 15824 at commit

[GitHub] spark pull request #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE...

2016-11-08 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15814#discussion_r87138232 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -87,25 +120,40 @@ class

[GitHub] spark pull request #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE...

2016-11-08 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/15814#discussion_r87138139 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -87,25 +120,40 @@ class

[GitHub] spark issue #15803: [SPARK-18298][Web UI]change gmt time to local zone time ...

2016-11-08 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the issue: https://github.com/apache/spark/pull/15803 I agree with showing the timezone with date string. But always using GMT/UTC time is not a good choice, logs of application(using log4j) usually are printed using local

[GitHub] spark issue #15807: [SPARK-18147][SQL] do not fail for very complex aggregat...

2016-11-08 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15807 @cloud-fan Then once the first expression to use the subexpression is in a if/else branch, we can't access the subexpression outside later. Evaluate it again? --- If your project is set up for

[GitHub] spark issue #15824: [SPARK-18376][SQL] Skip subexpression elimination for co...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15824 **[Test build #68393 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68393/consoleFull)** for PR 15824 at commit

[GitHub] spark issue #15824: [SPARK-18376][SQL] Skip subexpression elimination for co...

2016-11-08 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15824 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #15825: [SPARK-18377][SQL] warehouse path should be a static con...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15825 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68390/ Test FAILed. ---

[GitHub] spark issue #15825: [SPARK-18377][SQL] warehouse path should be a static con...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15825 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15825: [SPARK-18377][SQL] warehouse path should be a static con...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15825 **[Test build #68390 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68390/consoleFull)** for PR 15825 at commit

[GitHub] spark issue #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE TABLE ...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15814 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68388/ Test FAILed. ---

[GitHub] spark issue #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE TABLE ...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15814 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE TABLE ...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15814 **[Test build #68388 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68388/consoleFull)** for PR 15814 at commit

[GitHub] spark issue #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE TABLE ...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15814 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE TABLE ...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15814 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68387/ Test FAILed. ---

[GitHub] spark issue #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE TABLE ...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15814 **[Test build #68387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68387/consoleFull)** for PR 15814 at commit

[GitHub] spark pull request #15748: [SPARK-18240][ML][PySpark] Add Summary of BiKMean...

2016-11-08 Thread zhengruifeng
Github user zhengruifeng closed the pull request at: https://github.com/apache/spark/pull/15748 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #15233: [SPARK-17659] [SQL] Partitioned View is Not Supported By...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15233 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15233: [SPARK-17659] [SQL] Partitioned View is Not Supported By...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15233 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68386/ Test PASSed. ---

[GitHub] spark issue #15233: [SPARK-17659] [SQL] Partitioned View is Not Supported By...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15233 **[Test build #68386 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68386/consoleFull)** for PR 15233 at commit

[GitHub] spark pull request #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE...

2016-11-08 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15814#discussion_r87135950 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala --- @@ -226,6 +238,34 @@ case class

[GitHub] spark issue #15823: [SPARK-18191][CORE][FOLLOWUP] Call `setConf` if `OutputF...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15823 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68384/ Test PASSed. ---

[GitHub] spark issue #15823: [SPARK-18191][CORE][FOLLOWUP] Call `setConf` if `OutputF...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15823 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15823: [SPARK-18191][CORE][FOLLOWUP] Call `setConf` if `OutputF...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15823 **[Test build #68384 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68384/consoleFull)** for PR 15823 at commit

[GitHub] spark issue #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE TABLE ...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15814 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68385/ Test FAILed. ---

[GitHub] spark issue #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE TABLE ...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15814 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE TABLE ...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15814 **[Test build #68385 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68385/consoleFull)** for PR 15814 at commit

[GitHub] spark issue #15797: [SPARK-17990][SPARK-18302][SQL] correct several partitio...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15797 **[Test build #68392 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68392/consoleFull)** for PR 15797 at commit

[GitHub] spark issue #15823: [SPARK-18191][CORE][FOLLOWUP] Call `setConf` if `OutputF...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15823 **[Test build #68391 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68391/consoleFull)** for PR 15823 at commit

[GitHub] spark issue #15824: [SPARK-18376][SQL] Skip subexpression elimination for co...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15824 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68389/ Test FAILed. ---

[GitHub] spark issue #15824: [SPARK-18376][SQL] Skip subexpression elimination for co...

2016-11-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15824 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15824: [SPARK-18376][SQL] Skip subexpression elimination for co...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15824 **[Test build #68389 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68389/consoleFull)** for PR 15824 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL].Staging directory fail to be removed

2016-11-08 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15819 Can you add some documentation? The current code is very difficult to follow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #15823: [SPARK-18191][CORE][FOLLOWUP] Call `setConf` if `OutputF...

2016-11-08 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15823 LGTM otherwise. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark pull request #15823: [SPARK-18191][CORE][FOLLOWUP] Call `setConf` if `...

2016-11-08 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15823#discussion_r87134013 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -42,7 +43,13 @@ class

[GitHub] spark issue #13758: [SPARK-16043][SQL] Prepare GenericArrayData implementati...

2016-11-08 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13758 @cloud-fan yes, we could take the same approach as #15044. When I have just implement it in my local environment, it can achieve similar performance improvement. I will submit that approach to

[GitHub] spark issue #15807: [SPARK-18147][SQL] do not fail for very complex aggregat...

2016-11-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15807 can we just evaluate subexpression like a scala lazy val? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #15825: [SPARK-18377][SQL] warehouse path should be a static con...

2016-11-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15825 CC @yhuai @rxin @srown @gatorsmile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15825: [SPARK-18377][SQL] warehouse path should be a static con...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15825 **[Test build #68390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68390/consoleFull)** for PR 15825 at commit

[GitHub] spark pull request #15825: [SPARK-18377][SQL] warehouse path should be a sta...

2016-11-08 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/15825 [SPARK-18377][SQL] warehouse path should be a static conf ## What changes were proposed in this pull request? it's weird that every session can set its own warehouse path at runtime, we

[GitHub] spark pull request #15262: [SPARK-17690][STREAMING][SQL] Add mini-dfs cluste...

2016-11-08 Thread ScrapCodes
Github user ScrapCodes closed the pull request at: https://github.com/apache/spark/pull/15262 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #15262: [SPARK-17690][STREAMING][SQL] Add mini-dfs cluster based...

2016-11-08 Thread ScrapCodes
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/15262 I was going to close this for now. @srowen Those deps should not have changed, I have not added anything to the compile scope. I have not analyzed the working of those deps generation,

[GitHub] spark issue #15824: [SQL] Skip subexpression elimination for conditional exp...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15824 **[Test build #68389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68389/consoleFull)** for PR 15824 at commit

[GitHub] spark issue #15824: [SQL] Skip subexpression elimination for conditional exp...

2016-11-08 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15824 cc @cloud-fan @kiszk --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #15824: [SQL] Skip subexpression elimination for conditio...

2016-11-08 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/15824 [SQL] Skip subexpression elimination for conditional expressions ## What changes were proposed in this pull request? As per discussion at #15807, we should disallow subexpression

[GitHub] spark issue #15807: [SPARK-18147][SQL] do not fail for very complex aggregat...

2016-11-08 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/15807 @viirya @cloud-fan It looks reasonable to me that to skip subexpression elimination for the expressions wrapped in condition expressions such as `if`. This is because we have only a place at top

[GitHub] spark pull request #15563: [SPARK-16759][CORE] Add a configuration property ...

2016-11-08 Thread weiqingy
Github user weiqingy commented on a diff in the pull request: https://github.com/apache/spark/pull/15563#discussion_r87131269 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -2587,17 +2589,16 @@ private[spark] class CallerContext( taskId: Option[Long]

[GitHub] spark issue #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE TABLE ...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15814 **[Test build #68388 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68388/consoleFull)** for PR 15814 at commit

[GitHub] spark pull request #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE...

2016-11-08 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15814#discussion_r87109861 --- Diff: core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala --- @@ -86,6 +86,16 @@ abstract class FileCommitProtocol { def

[GitHub] spark pull request #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE...

2016-11-08 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15814#discussion_r87111922 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -42,17 +44,21 @@ class

[GitHub] spark pull request #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE...

2016-11-08 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15814#discussion_r87112095 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala --- @@ -350,13 +350,15 @@ case class

[GitHub] spark pull request #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE...

2016-11-08 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15814#discussion_r87113460 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala --- @@ -182,41 +182,53 @@ case class

[GitHub] spark pull request #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE...

2016-11-08 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15814#discussion_r87129853 --- Diff: core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala --- @@ -86,6 +86,16 @@ abstract class FileCommitProtocol { def

[GitHub] spark pull request #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE...

2016-11-08 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15814#discussion_r87112188 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -418,6 +418,8 @@ case class DataSource( val

[GitHub] spark pull request #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE...

2016-11-08 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15814#discussion_r87111758 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -42,17 +44,21 @@ class

[GitHub] spark pull request #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE...

2016-11-08 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15814#discussion_r87112037 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala --- @@ -178,18 +178,13 @@ class AstBuilder extends

[GitHub] spark issue #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE TABLE ...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15814 **[Test build #68387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68387/consoleFull)** for PR 15814 at commit

[GitHub] spark issue #15233: [SPARK-17659] [SQL] Partitioned View is Not Supported By...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15233 **[Test build #68386 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68386/consoleFull)** for PR 15233 at commit

[GitHub] spark issue #15820: [SPARK-18373][SS][Kafka]Make failOnDataLoss=false work w...

2016-11-08 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15820 Wow, looks like the new github comment interface did all kinds of weird things, apologies about that. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request #15820: [SPARK-18373][SS][Kafka]Make failOnDataLoss=false...

2016-11-08 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/15820#discussion_r87129126 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaConsumer.scala --- @@ -83,6 +86,113 @@ private[kafka010] case

[GitHub] spark pull request #15233: [SPARK-17659] [SQL] Partitioned View is Not Suppo...

2016-11-08 Thread gatorsmile
GitHub user gatorsmile reopened a pull request: https://github.com/apache/spark/pull/15233 [SPARK-17659] [SQL] Partitioned View is Not Supported By SHOW CREATE TABLE ### What changes were proposed in this pull request? `Partitioned View` is not supported by SPARK SQL. For

[GitHub] spark pull request #15820: [SPARK-18373][SS][Kafka]Make failOnDataLoss=false...

2016-11-08 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/15820#discussion_r87129981 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaConsumer.scala --- @@ -83,6 +86,113 @@ private[kafka010] case

[GitHub] spark pull request #15820: [SPARK-18373][SS][Kafka]Make failOnDataLoss=false...

2016-11-08 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/15820#discussion_r87129817 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaConsumer.scala --- @@ -83,6 +86,113 @@ private[kafka010] case

[GitHub] spark pull request #15820: [SPARK-18373][SS][Kafka]Make failOnDataLoss=false...

2016-11-08 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/15820#discussion_r87129927 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaConsumer.scala --- @@ -83,6 +86,113 @@ private[kafka010] case

[GitHub] spark issue #15233: [SPARK-17659] [SQL] Partitioned View is Not Supported By...

2016-11-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15233 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15233: [SPARK-17659] [SQL] Partitioned View is Not Supported By...

2016-11-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15233 Sure, let me reopen it. : ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #15820: [SPARK-18373][SS][Kafka]Make failOnDataLoss=false...

2016-11-08 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/15820#discussion_r87127811 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaConsumer.scala --- @@ -83,6 +86,113 @@ private[kafka010] case

[GitHub] spark pull request #15820: [SPARK-18373][SS][Kafka]Make failOnDataLoss=false...

2016-11-08 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/15820#discussion_r87130059 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaConsumer.scala --- @@ -83,6 +86,113 @@ private[kafka010] case

[GitHub] spark pull request #15820: [SPARK-18373][SS][Kafka]Make failOnDataLoss=false...

2016-11-08 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/15820#discussion_r87128373 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaConsumer.scala --- @@ -83,6 +86,113 @@ private[kafka010] case

[GitHub] spark pull request #15820: [SPARK-18373][SS][Kafka]Make failOnDataLoss=false...

2016-11-08 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/15820#discussion_r87129204 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaConsumer.scala --- @@ -83,6 +86,113 @@ private[kafka010] case

[GitHub] spark issue #15814: [SPARK-18185] Fix all forms of INSERT / OVERWRITE TABLE ...

2016-11-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15814 **[Test build #68385 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68385/consoleFull)** for PR 15814 at commit

  1   2   3   4   5   6   >