[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...

[gatorsmile]

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/12081

[SPARK-14124] [SQL] [FOLLOWUP] Implement Database-related DDL Commands 

 What changes were proposed in this pull request?
This PR is to correct the generated database path when users issue the 
command `Create Database`. For example, when users specify the `Location` in 
the command, the generated path should be `path/databaseName.db` instead of 
`path`. 

cc @yhuai @andrewor14 Sorry, I did not catch this issue when submitting the 
PR. Could you review this follow-up?

 How was this patch tested?
Added a test case for it and correct the previous test cases.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark mkdir

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12081.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12081


commit 7cd839e81f8c5b157c832a16ef5355378cc7b7c0
Author: gatorsmile 
Date:   2016-03-31T06:52:31Z

correct the database path.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...

[davies]

Github user davies commented on a diff in the pull request:

https://github.com/apache/spark/pull/11517#discussion_r58008150
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala ---
@@ -194,6 +194,71 @@ case class Sample(
   child.execute().randomSampleWithRange(lowerBound, upperBound, seed)
 }
   }
+
+  override def upstreams(): Seq[RDD[InternalRow]] = {
+child.asInstanceOf[CodegenSupport].upstreams()
+  }
+
+  private var rowBuffer: String = _
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: 
ExprCode): String = {
+val sampler = ctx.freshName("sampler")
+
+if (withReplacement) {
+  val samplerClass = classOf[PoissonSampler[UnsafeRow]].getName
+  val classTag = ctx.freshName("classTag")
--- End diff --

Do we really need this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203788406
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203788409
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54595/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203788243
  
**[Test build #54595 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54595/consoleFull)**
 for PR 12060 at commit 
[`f1407c0`](https://github.com/apache/spark/commit/f1407c0bb302355f7f06aad9ece00541063bde6e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...

[davies]

Github user davies commented on a diff in the pull request:

https://github.com/apache/spark/pull/11517#discussion_r58007758
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala ---
@@ -194,6 +194,71 @@ case class Sample(
   child.execute().randomSampleWithRange(lowerBound, upperBound, seed)
 }
   }
+
+  override def upstreams(): Seq[RDD[InternalRow]] = {
+child.asInstanceOf[CodegenSupport].upstreams()
+  }
+
+  private var rowBuffer: String = _
--- End diff --

Where is this used?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...

[davies]

Github user davies commented on a diff in the pull request:

https://github.com/apache/spark/pull/11517#discussion_r58007651
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala ---
@@ -275,8 +340,8 @@ case class Range(
   | // initialize Range
   | if (!$initTerm) {
   |   $initTerm = true;
-  |   if ($input.hasNext()) {
-  | initRange(((InternalRow) $input.next()).getInt(0));
+  |   if (partitionIndex != -1) {
--- End diff --

?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14098][SQL][WIP] Generate Java code tha...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11956#issuecomment-203787738
  
**[Test build #54604 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54604/consoleFull)**
 for PR 11956 at commit 
[`35a352a`](https://github.com/apache/spark/commit/35a352a2ab24ed31a8c5a5c54e940ba32f807601).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [DO_NOT_MERGE]Try to reproduce StateStoreRDDSu...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12005#issuecomment-203787652
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [DO_NOT_MERGE]Try to reproduce StateStoreRDDSu...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12005#issuecomment-203787661
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54597/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [DO_NOT_MERGE]Try to reproduce StateStoreRDDSu...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12005#issuecomment-203787488
  
**[Test build #54597 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54597/consoleFull)**
 for PR 12005 at commit 
[`9a4d8b3`](https://github.com/apache/spark/commit/9a4d8b3f9a388505f156115919a13949e48d0bc6).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14124] [SQL] Implement Database-related...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/12009#discussion_r58006853
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -45,46 +45,135 @@ abstract class NativeDDLCommand(val sql: String) 
extends RunnableCommand {
 
 }
 
+/**
+ * A command for users to create a new database.
+ *
+ * It will issue an error message when the database with the same name 
already exists,
+ * unless 'ifNotExists' is true.
+ * The syntax of using this command in SQL is:
+ * {{{
+ *CREATE DATABASE|SCHEMA [IF NOT EXISTS] database_name
+ * }}}
+ */
 case class CreateDatabase(
 databaseName: String,
 ifNotExists: Boolean,
 path: Option[String],
 comment: Option[String],
-props: Map[String, String])(sql: String)
-  extends NativeDDLCommand(sql) with Logging
+props: Map[String, String])
+  extends RunnableCommand {
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val catalog = sqlContext.sessionState.catalog
+catalog.createDatabase(
+  CatalogDatabase(
+databaseName,
+comment.getOrElse(""),
+path.getOrElse(catalog.getDefaultDBPath(databaseName)),
+props),
+  ifNotExists)
+Seq.empty[Row]
+  }
--- End diff --

I tried it in `spark-sql`. If the directory is not created, Hive will do it 
for us. I am wondering if we still should create directory in Spark?

However, this PR has an issue when users specify the location in the 
`Create Database` command. The generated path should be `path/databaseName.db` 
instead of `path`. Will fix it soon. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14123][SQL] Implement function related ...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/12036#discussion_r58006735
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
 ---
@@ -67,9 +77,14 @@ class SimpleFunctionRegistry extends FunctionRegistry {
   }
 
   override def lookupFunction(name: String, children: Seq[Expression]): 
Expression = {
+val builder = functionBuilders.get(name)
+if (builder.isEmpty) {
+  throw new AnalysisException(s"undefined function $name")
+}
 val func = synchronized {
-  functionBuilders.get(name).map(_._2).getOrElse {
-throw new AnalysisException(s"undefined function $name")
+  Try(builder.map(_._2)) match {
--- End diff --

Hive's exception will be thrown. I faced the exception thrown in working on 
the changes. One jenkins test checks the error message. We will miss the actual 
exception and message and only get AnalysisException with `undefined function` 
here. I think it is proper to show real error message.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203783507
  
**[Test build #54603 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54603/consoleFull)**
 for PR 12057 at commit 
[`72a5ec0`](https://github.com/apache/spark/commit/72a5ec08123f5f7b8c515256a09cd0a87e05cc9f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203779351
  
**[Test build #54602 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54602/consoleFull)**
 for PR 12057 at commit 
[`8597bba`](https://github.com/apache/spark/commit/8597bbaf7d29de234f3f29e2929ce79c1ec99075).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14123][SQL] Implement function related ...

[viirya]

Github user viirya commented on the pull request:

https://github.com/apache/spark/pull/12036#issuecomment-203776390
  
@andrewor14 Thanks for reviewing this. The current changes might need 
further refactoring. As you said, `HiveFunctionRegistry` now refers to 
`HiveContext` and `SessionCatalog`. It is a messy one but I  just want it to be 
functional and prove this approach work. I think I would do refactoring today 
to make it more clear. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14191][SQL] Remove invalid Expand opera...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11995#issuecomment-203775452
  
**[Test build #54601 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54601/consoleFull)**
 for PR 11995 at commit 
[`ab89e62`](https://github.com/apache/spark/commit/ab89e620883f581b2104fc60ffb32f77501f94c4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13796] Redirect error message to logWar...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12052#issuecomment-203774529
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14270][SQL] whole stage codegen support...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12061#issuecomment-203774865
  
**[Test build #54593 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54593/consoleFull)**
 for PR 12061 at commit 
[`aa95fd6`](https://github.com/apache/spark/commit/aa95fd6fd0a6146cd9f41cd59dbfb7f58b6c17da).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  s\"Unable to generate an encoder for inner class `$`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13796] Redirect error message to logWar...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12052#issuecomment-203774534
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54587/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11987#issuecomment-203773703
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13796] Redirect error message to logWar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12052#issuecomment-203773804
  
**[Test build #54587 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54587/consoleFull)**
 for PR 12052 at commit 
[`4dc9c79`](https://github.com/apache/spark/commit/4dc9c79a2d1d101226b415689663f0572adccb2d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11987#issuecomment-203773709
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54592/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13112]CoarsedExecutorBackend register t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12078#issuecomment-203773443
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54586/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13112]CoarsedExecutorBackend register t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12078#issuecomment-203773436
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11888#issuecomment-203772958
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11987#issuecomment-203772943
  
**[Test build #54592 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54592/consoleFull)**
 for PR 11987 at commit 
[`38ea348`](https://github.com/apache/spark/commit/38ea348cab7ffcf9a107bb56fb905bab7a6f2d00).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13112]CoarsedExecutorBackend register t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12078#issuecomment-203772900
  
**[Test build #54586 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54586/consoleFull)**
 for PR 12078 at commit 
[`1b04630`](https://github.com/apache/spark/commit/1b046304313c7663015667ab9cc8fe4201d17eb2).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11888#issuecomment-203772960
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54588/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11888#issuecomment-203772472
  
**[Test build #54588 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54588/consoleFull)**
 for PR 11888 at commit 
[`39dd0ee`](https://github.com/apache/spark/commit/39dd0eed15d9c67604cb3430350c4f0f57c5f7e7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11987#issuecomment-203772332
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11987#issuecomment-203772338
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54594/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11987#issuecomment-203771892
  
**[Test build #54594 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54594/consoleFull)**
 for PR 11987 at commit 
[`dd34529`](https://github.com/apache/spark/commit/dd34529ea4f15b627cdf4dd921bf5a726d543b6d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203770747
  
cc @cloud-fan 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14270][SQL] whole stage codegen support...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12061#issuecomment-203770531
  
**[Test build #54600 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54600/consoleFull)**
 for PR 12061 at commit 
[`bf9f5b5`](https://github.com/apache/spark/commit/bf9f5b53c28566674e47f8146ed9f53248dfcea6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203768625
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203768640
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54591/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-14138][SQL] Fix generated SpecificColum...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11984#issuecomment-203768144
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54599/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-14138][SQL] Fix generated SpecificColum...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11984#issuecomment-203768128
  
**[Test build #54599 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54599/consoleFull)**
 for PR 11984 at commit 
[`16cf602`](https://github.com/apache/spark/commit/16cf602b506c37c9e5cd9cbb48d577b7e66f14d4).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-14138][SQL] Fix generated SpecificColum...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11984#issuecomment-203768142
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203767874
  
**[Test build #54591 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54591/consoleFull)**
 for PR 12077 at commit 
[`cec6902`](https://github.com/apache/spark/commit/cec6902fc6bb5b501aa5b1c536b034508f7a5c75).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-14138][SQL] Fix generated SpecificColum...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11984#issuecomment-203766948
  
**[Test build #54599 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54599/consoleFull)**
 for PR 11984 at commit 
[`16cf602`](https://github.com/apache/spark/commit/16cf602b506c37c9e5cd9cbb48d577b7e66f14d4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11987#issuecomment-203764881
  
**[Test build #54598 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54598/consoleFull)**
 for PR 11987 at commit 
[`48aec92`](https://github.com/apache/spark/commit/48aec92480ec59ed4a965941d56126d9222cb853).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14191][SQL] Fix Expand operator constra...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/11995#discussion_r58003817
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
 ---
@@ -514,12 +524,13 @@ private[sql] object Expand {
  * @param projections to apply
  * @param output of all projections.
  * @param child operator.
+ * @param groupByAttrs the attributes used in group by.
  */
 case class Expand(
 projections: Seq[Seq[Expression]],
 output: Seq[Attribute],
-child: LogicalPlan) extends UnaryNode {
-
+child: LogicalPlan,
+groupByAttrs: Seq[Attribute]) extends UnaryNode {
--- End diff --

Ok. Let me remove the constraints first.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-13784][ML][WIP] Model export/import for...

[GayathriMurali]

Github user GayathriMurali commented on the pull request:

https://github.com/apache/spark/pull/12023#issuecomment-203763228
  
@jkbradley I am sorry, I am afraid I will not be able to complete tonight. 
Can you please help me with reusing Splitdata/build code from DecisionTrees in 
RandomForests? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203761313
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54589/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203761312
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/12077#discussion_r58003342
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/typedaggregators.scala
 ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.aggregate
+
+import org.apache.spark.sql.expressions.Aggregator
+

+
+// This file defines internal implementations for aggregators.

+
+
+
+class TypedSum[IN, OUT : Numeric](f: IN => OUT) extends Aggregator[IN, 
OUT, OUT] {
+  val numeric = implicitly[Numeric[OUT]]
+  override def zero: OUT = numeric.zero
+  override def reduce(b: OUT, a: IN): OUT = numeric.plus(b, f(a))
+  override def merge(b1: OUT, b2: OUT): OUT = numeric.plus(b1, b2)
+  override def finish(reduction: OUT): OUT = reduction
+}
+
+
+class TypedSumDouble[IN](f: IN => Double) extends Aggregator[IN, Double, 
Double] {
--- End diff --

in the future we should just turn these into aggregate expressions without 
aggregator


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203761179
  
**[Test build #54589 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54589/consoleFull)**
 for PR 12080 at commit 
[`7dd88a3`](https://github.com/apache/spark/commit/7dd88a369c375009715de4d0c604d956dab69002).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [DO_NOT_MERGE]Try to reproduce StateStoreRDDSu...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12005#issuecomment-203757779
  
**[Test build #54597 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54597/consoleFull)**
 for PR 12005 at commit 
[`9a4d8b3`](https://github.com/apache/spark/commit/9a4d8b3f9a388505f156115919a13949e48d0bc6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12343][YARN] Simplify Yarn client and c...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11603#issuecomment-203756317
  
**[Test build #54596 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54596/consoleFull)**
 for PR 11603 at commit 
[`3bb44b4`](https://github.com/apache/spark/commit/3bb44b4b1b84f9a972ad8ea4876b70369ba07d0c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203753133
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54584/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203753127
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14191][SQL] Fix Expand operator constra...

[marmbrus]

Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/11995#discussion_r58001590
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
 ---
@@ -514,12 +524,13 @@ private[sql] object Expand {
  * @param projections to apply
  * @param output of all projections.
  * @param child operator.
+ * @param groupByAttrs the attributes used in group by.
  */
 case class Expand(
 projections: Seq[Seq[Expression]],
 output: Seq[Attribute],
-child: LogicalPlan) extends UnaryNode {
-
+child: LogicalPlan,
+groupByAttrs: Seq[Attribute]) extends UnaryNode {
--- End diff --

Yeah, I do think it would be good if it just took a 
`Seq[Seq[NamedExpression]]` (or at least I can't come up with anything better). 
 I'd still consider breaking this into two PRs.  Simple fix for now that just 
removes invalid constraints and a refactoring that add back in valid ones.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203752798
  
**[Test build #54595 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54595/consoleFull)**
 for PR 12060 at commit 
[`f1407c0`](https://github.com/apache/spark/commit/f1407c0bb302355f7f06aad9ece00541063bde6e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203752828
  
**[Test build #54584 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54584/consoleFull)**
 for PR 12057 at commit 
[`dd71ba9`](https://github.com/apache/spark/commit/dd71ba90cd41722c8fd3811d2c698da344aeb690).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...

[ueshin]

Github user ueshin commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203752562
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/11987#discussion_r58001426
  
--- Diff: 
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/ng/SqlBase.g4 
---
@@ -74,12 +75,16 @@ statement
 SET SKEWED LOCATION skewedLocationList 
#setTableSkewLocations
 | ALTER TABLE tableIdentifier ADD (IF NOT EXISTS)?
 partitionSpecLocation+ 
#addTablePartition
+| ALTER VIEW tableIdentifier ADD (IF NOT EXISTS)?
+partitionSpec+ 
#addViewPartition
--- End diff --

@viirya Yeah, I am doing it now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/11987#discussion_r58001378
  
--- Diff: 
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/ng/SqlBase.g4 
---
@@ -74,12 +75,16 @@ statement
 SET SKEWED LOCATION skewedLocationList 
#setTableSkewLocations
 | ALTER TABLE tableIdentifier ADD (IF NOT EXISTS)?
 partitionSpecLocation+ 
#addTablePartition
+| ALTER VIEW tableIdentifier ADD (IF NOT EXISTS)?
+partitionSpec+ 
#addViewPartition
--- End diff --

re-use label?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/11987#discussion_r58001379
  
--- Diff: 
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/ng/SqlBase.g4 
---
@@ -74,12 +75,16 @@ statement
 SET SKEWED LOCATION skewedLocationList 
#setTableSkewLocations
 | ALTER TABLE tableIdentifier ADD (IF NOT EXISTS)?
 partitionSpecLocation+ 
#addTablePartition
+| ALTER VIEW tableIdentifier ADD (IF NOT EXISTS)?
+partitionSpec+ 
#addViewPartition
--- End diff --

@hvanhovell Still trying to combine `addViewPartition` and 
`addTablePartition`. Will keep you posted. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11987#issuecomment-203751599
  
**[Test build #54594 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54594/consoleFull)**
 for PR 11987 at commit 
[`dd34529`](https://github.com/apache/spark/commit/dd34529ea4f15b627cdf4dd921bf5a726d543b6d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14270][SQL] whole stage codegen support...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12061#issuecomment-203751600
  
**[Test build #54593 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54593/consoleFull)**
 for PR 12061 at commit 
[`aa95fd6`](https://github.com/apache/spark/commit/aa95fd6fd0a6146cd9f41cd59dbfb7f58b6c17da).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14191][SQL] Fix Expand operator constra...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/11995#discussion_r58000861
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
 ---
@@ -514,12 +524,13 @@ private[sql] object Expand {
  * @param projections to apply
  * @param output of all projections.
  * @param child operator.
+ * @param groupByAttrs the attributes used in group by.
  */
 case class Expand(
 projections: Seq[Seq[Expression]],
 output: Seq[Attribute],
-child: LogicalPlan) extends UnaryNode {
-
+child: LogicalPlan,
+groupByAttrs: Seq[Attribute]) extends UnaryNode {
--- End diff --

Thanks for pointing the problem. Agreed after re-thinking about it. As 
separating `projections` and `output` causes the problem. How about we get the 
output from `projections`?

As there are more than one projection, we can just get the output from the 
first projection and verify its consistency with other projections.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14182] [SQL] Parse DDL Command: Alter V...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11987#issuecomment-203750876
  
**[Test build #54592 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54592/consoleFull)**
 for PR 11987 at commit 
[`38ea348`](https://github.com/apache/spark/commit/38ea348cab7ffcf9a107bb56fb905bab7a6f2d00).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203750523
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54582/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203750521
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12060#issuecomment-203750120
  
**[Test build #54582 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54582/consoleFull)**
 for PR 12060 at commit 
[`f1407c0`](https://github.com/apache/spark/commit/f1407c0bb302355f7f06aad9ece00541063bde6e).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14242][CORE][Network] avoid copy in com...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12038#issuecomment-203749671
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54580/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14242][CORE][Network] avoid copy in com...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12038#issuecomment-203749668
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14133][SQL] Throws exception for unsupp...

[sureshthalamati]

Github user sureshthalamati commented on the pull request:

https://github.com/apache/spark/pull/12069#issuecomment-203749355
  
Thanks for the input, Herman.  Addressed the review comments , and fixed 
the test failure in my latest commit. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14242][CORE][Network] avoid copy in com...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12038#issuecomment-203749214
  
**[Test build #54580 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54580/consoleFull)**
 for PR 12038 at commit 
[`80f7573`](https://github.com/apache/spark/commit/80f75737d6dd25fa859b06e876f16f21d5b6b247).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14277] UnsafeSorterSpillReader should d...

[xerial]

Github user xerial commented on the pull request:

https://github.com/apache/spark/pull/12074#issuecomment-203748806
  
Released snappy-java-1.1.2.4 with this fix. Thanks for letting me know. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14133][SQL] Throws exception for unsupp...

[sureshthalamati]

Github user sureshthalamati commented on a diff in the pull request:

https://github.com/apache/spark/pull/12069#discussion_r58000282
  
--- Diff: 
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/ng/SqlBase.g4 
---
@@ -167,6 +167,11 @@ unsupportedHiveNativeCommands
 | kw1=SHOW kw2=TRANSACTIONS
 | kw1=SHOW kw2=INDEXES
 | kw1=SHOW kw2=LOCKS
+| kw1=CREATE kw2=INDEX
+| kw1=DROP kw2=INDEX
+| kw1=LOCK kw2=TABLE
--- End diff --

Add lock/unlock database also.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203748254
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54581/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203748302
  
**[Test build #54591 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54591/consoleFull)**
 for PR 12077 at commit 
[`cec6902`](https://github.com/apache/spark/commit/cec6902fc6bb5b501aa5b1c536b034508f7a5c75).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203748250
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203748158
  
**[Test build #54581 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54581/consoleFull)**
 for PR 12057 at commit 
[`8dc1adf`](https://github.com/apache/spark/commit/8dc1adfb12a35280a01b4c8ab95b5aed346d8f0f).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14133][SQL] Throws exception for unsupp...

[sureshthalamati]

Github user sureshthalamati commented on a diff in the pull request:

https://github.com/apache/spark/pull/12069#discussion_r5740
  
--- Diff: 
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/ng/SqlBase.g4 
---
@@ -861,6 +866,7 @@ ROLES: 'ROLES';
 COMPACTIONS: 'COMPACTIONS';
 PRINCIPALS: 'PRINCIPALS';
 TRANSACTIONS: 'TRANSACTIONS';
+INDEX: 'INDEX';
--- End diff --

Thank you for reviewing the PR, Herman.  Added it to the non-Reserved list. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203747707
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203747708
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54590/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203747701
  
**[Test build #54590 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54590/consoleFull)**
 for PR 12077 at commit 
[`6fb03e5`](https://github.com/apache/spark/commit/6fb03e5163637e612bc6366583915ec612e7f0a0).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14285][SQL] Implement common type-safe ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12077#issuecomment-203747344
  
**[Test build #54590 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54590/consoleFull)**
 for PR 12077 at commit 
[`6fb03e5`](https://github.com/apache/spark/commit/6fb03e5163637e612bc6366583915ec612e7f0a0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203746656
  
**[Test build #54589 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54589/consoleFull)**
 for PR 12080 at commit 
[`7dd88a3`](https://github.com/apache/spark/commit/7dd88a369c375009715de4d0c604d956dab69002).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13796] Redirect error message to logWar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12052#issuecomment-203744494
  
**[Test build #54587 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54587/consoleFull)**
 for PR 12052 at commit 
[`4dc9c79`](https://github.com/apache/spark/commit/4dc9c79a2d1d101226b415689663f0572adccb2d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11888#issuecomment-203744481
  
**[Test build #54588 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54588/consoleFull)**
 for PR 11888 at commit 
[`39dd0ee`](https://github.com/apache/spark/commit/39dd0eed15d9c67604cb3430350c4f0f57c5f7e7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13112]CoarsedExecutorBackend register t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12078#issuecomment-203744495
  
**[Test build #54586 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54586/consoleFull)**
 for PR 12078 at commit 
[`1b04630`](https://github.com/apache/spark/commit/1b046304313c7663015667ab9cc8fe4201d17eb2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13112]CoarsedExecutorBackend register t...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/12078#issuecomment-203744234
  
add to whitelist @zsxwing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203744275
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54579/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203744274
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14267] [SQL] [PYSPARK] execute multiple...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12057#issuecomment-203744186
  
**[Test build #54579 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54579/consoleFull)**
 for PR 12057 at commit 
[`87f4bb4`](https://github.com/apache/spark/commit/87f4bb41096bbfbb8de49dbe4898e688ca3ccd6c).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13796] Redirect error message to logWar...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/12052#issuecomment-203744206
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203744084
  
**[Test build #54585 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54585/consoleFull)**
 for PR 12080 at commit 
[`7459a3c`](https://github.com/apache/spark/commit/7459a3c7293e2659aaf87485d4a937bae9fdd384).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203744088
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54585/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203744087
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/11888#issuecomment-203744003
  
LGTM retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12080#issuecomment-203743978
  
**[Test build #54585 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54585/consoleFull)**
 for PR 12080 at commit 
[`7459a3c`](https://github.com/apache/spark/commit/7459a3c7293e2659aaf87485d4a937bae9fdd384).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14287] isStreaming method for Dataset

[brkyvz]

GitHub user brkyvz opened a pull request:

https://github.com/apache/spark/pull/12080

[SPARK-14287] isStreaming method for Dataset

With the addition of StreamExecution (ContinuousQuery) to Datasets, data 
will become unbounded. With unbounded data, the execution of some methods and 
operations will not make sense, e.g. `Dataset.count()`.

A simple API is required to check whether the data in a Dataset is bounded 
or unbounded. This will allow users to check whether their Dataset is in 
streaming mode or not. ML algorithms may check if the data is unbounded and 
throw an exception for example.

The implementation of this method is simple, however naming it is the 
challenge. Some possible names for this method are:
 - isStreaming
 - isContinuous
 - isBounded
 - isUnbounded

I've gone with `isStreaming` for now. We can change it before Spark 2.0 if 
we decide to come up with a different name. For that reason I've marked it as 
`@Experimental`


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/brkyvz/spark is-streaming

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12080.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12080


commit 7459a3c7293e2659aaf87485d4a937bae9fdd384
Author: Burak Yavuz 
Date:   2016-03-31T03:56:29Z

added isStreaming method to Dataset




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14238][ML][MLLIB][PYSPARK] Add binary t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12079#issuecomment-203741023
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14238][ML][MLLIB][PYSPARK] Add binary t...

[yongtang]

GitHub user yongtang opened a pull request:

https://github.com/apache/spark/pull/12079

[SPARK-14238][ML][MLLIB][PYSPARK] Add binary toggle Param to PySpark 
HashingTF in ML & MLlib

## What changes were proposed in this pull request?

This fix tries to add binary toggle Param to PySpark HashingTF in ML & 
MLlib. If this toggle is set, then all non-zero counts will be set to 1.

## How was this patch tested?

This fix adds two tests to cover the code changes. One for HashingTF in 
PySpark's ML and one for HashingTF in PySpark's MLLib.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yongtang/spark SPARK-14238

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12079.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12079


commit e58d1a279aaded9045c9e7a7a161500163b81fd6
Author: Yong Tang 
Date:   2016-03-31T03:49:33Z

[SPARK-14238][ML][MLLIB][PYSPARK] Add binary toggle Param to PySpark 
HashingTF in ML & MLlib

This fix tries to add binary toggle Param to PySpark HashingTF in ML & 
MLlib.
If this toggle is set, then all non-zero counts will be set to 1.

This fix adds two tests to cover the code changes. One for HashingTF in 
PySpark's ML
and one for HashingTF in PySpark's MLLib.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14098][SQL][WIP] Generate Java code tha...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11956#issuecomment-203739715
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org