[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11410#issuecomment-193653823
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52645/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11410#issuecomment-193653822
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13671] [SQL] Use different physical pla...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11514#issuecomment-193651401
  
**[Test build #52644 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52644/consoleFull)**
 for PR 11514 at commit 
[`0278fd9`](https://github.com/apache/spark/commit/0278fd94a230108c37e1e9c17365bd37b30a5288).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13671] [SQL] Use different physical pla...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11514#issuecomment-193651407
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52644/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13671] [SQL] Use different physical pla...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11514#issuecomment-193651406
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13671] [SQL] Use different physical pla...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11514#issuecomment-193650798
  
**[Test build #2619 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2619/consoleFull)**
 for PR 11514 at commit 
[`d2d2062`](https://github.com/apache/spark/commit/d2d206249b16c1a019a294a42c3118e400a21da6).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13631] [CORE] Thread-safe getLocationsW...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/11505#issuecomment-193650717
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13671] [SQL] Use different physical pla...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11514#issuecomment-193650362
  
**[Test build #52644 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52644/consoleFull)**
 for PR 11514 at commit 
[`0278fd9`](https://github.com/apache/spark/commit/0278fd94a230108c37e1e9c17365bd37b30a5288).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13631] [CORE] Thread-safe getLocationsW...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11505#issuecomment-193649424
  
OK, well unless there's a moderately strong objection, I think we can go 
ahead and merge this, even for 1.6. It's a cheap defensive measure and I don't 
see a downside.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13671] [SQL] Use different physical pla...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11514#issuecomment-193648651
  
**[Test build #2619 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2619/consoleFull)**
 for PR 11514 at commit 
[`d2d2062`](https://github.com/apache/spark/commit/d2d206249b16c1a019a294a42c3118e400a21da6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-193648680
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-193648669
  
**[Test build #52642 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52642/consoleFull)**
 for PR 11443 at commit 
[`4dc95bc`](https://github.com/apache/spark/commit/4dc95bc5cfb0f0be26b580e075a7b608725fc01f).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-193648682
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52642/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13523] [SQL] Reuse exchanges in a query

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11403#issuecomment-193647799
  
**[Test build #52643 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52643/consoleFull)**
 for PR 11403 at commit 
[`679d669`](https://github.com/apache/spark/commit/679d6696ec4ccf7d22fe836837cdf9320a2c06c7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13671] [SQL] Use different physical pla...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11514#issuecomment-193647377
  
**[Test build #2618 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2618/consoleFull)**
 for PR 11514 at commit 
[`d2d2062`](https://github.com/apache/spark/commit/d2d206249b16c1a019a294a42c3118e400a21da6).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL] Parse Hive DDL commands our...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/11573#discussion_r55324768
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.Logging
+import org.apache.spark.sql.{Row, SQLContext}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.expressions.{Attribute, 
AttributeReference}
+import org.apache.spark.sql.execution.datasources.BucketSpec
+import org.apache.spark.sql.types._
+
+
+/**
+ * A DDL command expected to be run in the underlying system without Spark 
parsing the
+ * query text.
+ */
+abstract class NativeDDLCommands(val sql: String) extends RunnableCommand {
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+sqlContext.runNativeSql(sql)
+  }
+
+  override val output: Seq[Attribute] = {
+Seq(AttributeReference("result", StringType, nullable = false)())
+  }
+
+}
+
+case class CreateDatabase(
+databaseName: String,
+allowExisting: Boolean,
+path: Option[String],
+comment: Option[String],
+props: Map[String, String])(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class CreateFunction(
+functionName: String,
+alias: String,
+resourcesMap: Map[String, String],
+isTemp: Boolean)(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableRename(
+tableName: TableIdentifier,
+renameTableName: TableIdentifier)(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableSetProperties(
+tableName: TableIdentifier,
+setProperties: Map[String, Option[String]])(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableDropProperties(
+tableName: TableIdentifier,
+dropProperties: Map[String, Option[String]],
+allowExisting: Boolean)(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableSerDeProperties(
+tableName: TableIdentifier,
+serdeClassName: Option[String],
+serdeProperties: Option[Map[String, Option[String]]],
+partition: Option[Map[String, Option[String]]])(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableStoreProperties(
+tableName: TableIdentifier,
+buckets: Option[BucketSpec],
+// TODO: use `clustered` and `sorted` instead for simplicity
+noClustered: Boolean,
+noSorted: Boolean)(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableSkewed(
+tableName: TableIdentifier,
+skewedCols: Seq[String],
+skewedValues: Seq[Seq[String]],
+storedAsDirs: Boolean,
+notSkewed: Boolean,
+// TODO: what??
+notStoredAsDirs: Boolean)(sql: String)
--- End diff --

`notStoredAsDirs` is used when the command is `ALTER TABLE table_name NOT 
STORED AS DIRECTORIES`. When it is true, `storedAsDirs` is false of course.

`storedAsDirs` is used with skewed arguments columns and values.

I think we can use just one variable for them.

When the command is `ALTER TABLE table_name NOT SKEWED`, then the values of 
 `notStoredAsDirs` and `storedAsDirs` don't matter. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: 

[GitHub] spark pull request: [SPARK-13523] [SQL] Reuse exchanges in a query

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/11403#issuecomment-193647125
  
@JoshRosen Do you have time to review this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13692][CORE][SQL] Fix trivial Coverity/...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the pull request:

https://github.com/apache/spark/pull/11530#issuecomment-193645680
  
Thank you, @zsxwing . 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11501#issuecomment-193643544
  
**[Test build #52641 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52641/consoleFull)**
 for PR 11501 at commit 
[`ea41707`](https://github.com/apache/spark/commit/ea417071f468ba3cdda6a6d273abd2fed2d4cf2b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11443#issuecomment-193643543
  
**[Test build #52642 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52642/consoleFull)**
 for PR 11443 at commit 
[`4dc95bc`](https://github.com/apache/spark/commit/4dc95bc5cfb0f0be26b580e075a7b608725fc01f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13671] [SQL] Use different physical pla...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11514#issuecomment-193642699
  
**[Test build #2618 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2618/consoleFull)**
 for PR 11514 at commit 
[`d2d2062`](https://github.com/apache/spark/commit/d2d206249b16c1a019a294a42c3118e400a21da6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11501#issuecomment-193642117
  
**[Test build #2617 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2617/consoleFull)**
 for PR 11501 at commit 
[`ea41707`](https://github.com/apache/spark/commit/ea417071f468ba3cdda6a6d273abd2fed2d4cf2b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL] Parse Hive DDL commands our...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/11573#discussion_r55324206
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.Logging
+import org.apache.spark.sql.{Row, SQLContext}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.expressions.{Attribute, 
AttributeReference}
+import org.apache.spark.sql.execution.datasources.BucketSpec
+import org.apache.spark.sql.types._
+
+
+/**
+ * A DDL command expected to be run in the underlying system without Spark 
parsing the
+ * query text.
+ */
+abstract class NativeDDLCommands(val sql: String) extends RunnableCommand {
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+sqlContext.runNativeSql(sql)
+  }
+
+  override val output: Seq[Attribute] = {
+Seq(AttributeReference("result", StringType, nullable = false)())
+  }
+
+}
+
+case class CreateDatabase(
+databaseName: String,
+allowExisting: Boolean,
+path: Option[String],
+comment: Option[String],
+props: Map[String, String])(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class CreateFunction(
+functionName: String,
+alias: String,
+resourcesMap: Map[String, String],
+isTemp: Boolean)(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableRename(
+tableName: TableIdentifier,
+renameTableName: TableIdentifier)(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableSetProperties(
+tableName: TableIdentifier,
+setProperties: Map[String, Option[String]])(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableDropProperties(
+tableName: TableIdentifier,
+dropProperties: Map[String, Option[String]],
+allowExisting: Boolean)(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableSerDeProperties(
+tableName: TableIdentifier,
+serdeClassName: Option[String],
+serdeProperties: Option[Map[String, Option[String]]],
+partition: Option[Map[String, Option[String]]])(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableStoreProperties(
+tableName: TableIdentifier,
+buckets: Option[BucketSpec],
+// TODO: use `clustered` and `sorted` instead for simplicity
+noClustered: Boolean,
+noSorted: Boolean)(sql: String)
--- End diff --

Just because the corresponding token is `TOK_NOT_CLUSTERED` and 
`TOK_NOT_SORTED`. We can use positive here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13692][CORE][SQL] Fix trivial Coverity/...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11530#issuecomment-193641581
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52631/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11501#issuecomment-193641668
  
**[Test build #52640 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52640/consoleFull)**
 for PR 11501 at commit 
[`5765b09`](https://github.com/apache/spark/commit/5765b096b105688af83311b429168c3df834c1d2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13692][CORE][SQL] Fix trivial Coverity/...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11530#issuecomment-193641575
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13692][CORE][SQL] Fix trivial Coverity/...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11530#issuecomment-193641123
  
**[Test build #52631 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52631/consoleFull)**
 for PR 11530 at commit 
[`9a0f8fa`](https://github.com/apache/spark/commit/9a0f8fabeccf56800dd8af74c39f14a99b8041a7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11572#issuecomment-193639421
  
**[Test build #52639 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52639/consoleFull)**
 for PR 11572 at commit 
[`dadd589`](https://github.com/apache/spark/commit/dadd589d11e2fa70ea756a9ba768a4663badf507).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL] Parse Hive DDL commands our...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11573#issuecomment-193637484
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL] Parse Hive DDL commands our...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11573#issuecomment-193637485
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52630/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/11501#issuecomment-193637383
  
Oops there is a conflict.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL] Parse Hive DDL commands our...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11573#issuecomment-193637270
  
**[Test build #52630 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52630/consoleFull)**
 for PR 11573 at commit 
[`a663b5c`](https://github.com/apache/spark/commit/a663b5ccca7718b41df8b8ede462ac6c9e0b8e8f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11501#issuecomment-193637100
  
**[Test build #2616 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2616/consoleFull)**
 for PR 11501 at commit 
[`c187554`](https://github.com/apache/spark/commit/c187554382ed1921c29b3a017a8ea6270eeec50f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13696][WIP] Remove BlockStore class & s...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/11534#issuecomment-193636595
  
Splitting up `doPut()` into two smaller methods, one concerned with putting 
bytes and another for putting iterators of objects, would let me simplify its 
return type and might lead to other simplifications (e.g. by eliminating 
impossible error-handling cases).  I might try this tomorrow.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13696][WIP] Remove BlockStore class & s...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11534#issuecomment-193636217
  
**[Test build #52638 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52638/consoleFull)**
 for PR 11534 at commit 
[`04f8d15`](https://github.com/apache/spark/commit/04f8d15b8877a508fba70ba1ee118bf9d199973b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13696][WIP] Remove BlockStore class & s...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/11534#issuecomment-193635954
  
FYI I rebased this to fix the merge conflicts, so the diff + commit log is 
now smaller.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...

[davies]

Github user davies commented on a diff in the pull request:

https://github.com/apache/spark/pull/11444#discussion_r55322763
  
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,139 @@ def __eq__(self, other):
   for v in [ArrayType, MapType, StructType])
 
 
+_FIXED_DECIMAL = re.compile("decimal\\((\\d+),(\\d+)\\)")
+
+
+def _parse_basic_datatype_string(s):
+if s == "null":
+return NullType()
+elif s == "boolean":
+return BooleanType()
+elif s == "byte":
+return ByteType()
+elif s == "short":
+return ShortType()
+elif s == "int":
+return IntegerType()
+elif s == "long":
+return LongType()
+elif s == "float":
+return FloatType()
+elif s == "double":
+return DoubleType()
+elif s == "decimal":
+return DecimalType()
+elif _FIXED_DECIMAL.match(s):
+m = _FIXED_DECIMAL.match(json_value)
+return DecimalType(int(m.group(1)), int(m.group(2)))
+elif s == "string":
+return StringType()
+elif s == "date":
+return DateType()
+elif s == "timestamp":
+return TimestampType()
+elif s == "binary":
+return BinaryType()
+else:
+raise ValueError("Cannot parse datatype string: %s" % s)
+
+
+def _ignore_brackets_split(s, separator):
+parts = []
+buf = ""
+level = 0
+for c in s:
+if c == "<":
+level += 1
+buf += c
+elif c == ">":
+if level == 0:
+raise ValueError("Cannot parse datatype string: %s" % s)
+level -= 1
+buf += c
+elif c == separator and level > 0:
+buf += c
+elif c == separator:
+parts.append(buf)
+buf = ""
+else:
+buf += c
+
+if len(buf) == 0:
+raise ValueError("Cannot parse datatype string: %s" % s)
+parts.append(buf)
+return parts
+
+
+def _parse_struct_type_string(s):
+parts = _ignore_brackets_split(s, ",")
+fields = []
+for part in parts:
+name_and_type = _ignore_brackets_split(part, ":")
+if len(name_and_type) != 2:
+raise ValueError("Cannot parse datatype string: %s" % s)
+field_name = name_and_type[0].strip()
+field_type = _parse_datatype_string(name_and_type[1])
+fields.append(StructField(field_name, field_type))
+return StructType(fields)
+
+
+def _parse_datatype_string(s):
--- End diff --

We could doc it for now, (and create a JIRA for it)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13740][SQL] add null check for _verify_...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11574#issuecomment-193634786
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13740][SQL] add null check for _verify_...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11574#issuecomment-193634791
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52633/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13740][SQL] add null check for _verify_...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11574#issuecomment-193634426
  
**[Test build #52633 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52633/consoleFull)**
 for PR 11574 at commit 
[`910315b`](https://github.com/apache/spark/commit/910315b0a8902805ad505a431b5bf940b66cf4ae).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13740][SQL] add null check for _verify_...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/11574#issuecomment-193634399
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11572#issuecomment-193634049
  
**[Test build #52637 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52637/consoleFull)**
 for PR 11572 at commit 
[`d83c2e2`](https://github.com/apache/spark/commit/d83c2e242fde0081c8215add1fb8381f5aeac2cb).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13695] Don't cache MEMORY_AND_DISK bloc...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11533#issuecomment-193632211
  
**[Test build #52636 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52636/consoleFull)**
 for PR 11533 at commit 
[`57d0bce`](https://github.com/apache/spark/commit/57d0bcec3348e2178b16da2e17d853529a6dc6d9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[gatorsmile]

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-193630940
  
Let me add the test cases in column pruning through `Windows` in 
`ColumnPruningSuite` tonight/tomorrow. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13695] Don't cache MEMORY_AND_DISK bloc...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/11533#issuecomment-193630155
  
Conflicts fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11572#issuecomment-193628454
  
**[Test build #52634 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52634/consoleFull)**
 for PR 11572 at commit 
[`6f6f099`](https://github.com/apache/spark/commit/6f6f0991a7a124390dc4da30c6a193a70fa901d3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13108][SQL] Support for ascii compatibl...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11016#issuecomment-193628447
  
**[Test build #52635 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52635/consoleFull)**
 for PR 11016 at commit 
[`264a1dc`](https://github.com/apache/spark/commit/264a1dc603164bd264e0c084608f31ffb8ad5f69).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-193628390
  
ah I see, makes sense. It would be better if there are some tests which 
guarantee the column pruning for window still works. Otherwise LGTM.
cc @yhuai 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8000][SQL] Support for auto-detecting d...

[HyukjinKwon]

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/11270#issuecomment-193627655
  
I will take an action as soon as I could have some feedback for this 
conflict.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13740][SQL] add null check for _verify_...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11574#issuecomment-193627008
  
**[Test build #52633 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52633/consoleFull)**
 for PR 11574 at commit 
[`910315b`](https://github.com/apache/spark/commit/910315b0a8902805ad505a431b5bf940b66cf4ae).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13740][SQL] add null check for _verify_...

[cloud-fan]

GitHub user cloud-fan opened a pull request:

https://github.com/apache/spark/pull/11574

[SPARK-13740][SQL] add null check for _verify_type in types.py

## What changes were proposed in this pull request?

This PR adds null check in `_verify_type` according to the nullability 
information.


## How was this patch tested?

new doc tests

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cloud-fan/spark py-null-check

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11574.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11574


commit 910315b0a8902805ad505a431b5bf940b66cf4ae
Author: Wenchen Fan 
Date:   2016-03-08T06:33:04Z

do null check in _verify_type




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13740][SQL] add null check for _verify_...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/11574#issuecomment-193626640
  
cc @yhuai @davies 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13527] [SQL] Prune Filters based on Con...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11406#issuecomment-193626228
  
**[Test build #52632 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52632/consoleFull)**
 for PR 11406 at commit 
[`7b7b411`](https://github.com/apache/spark/commit/7b7b41134257bf18169f28b648a00daf711bc2ab).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13695] Don't cache MEMORY_AND_DISK bloc...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/11533#issuecomment-193619570
  
LGTM2. There are conflicts now though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13659] Refactor BlockStore put*() APIs ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11502


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13659] Refactor BlockStore put*() APIs ...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/11502#issuecomment-193618865
  
LGTM I'm merging this into master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13659] Refactor BlockStore put*() APIs ...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/11502#discussion_r55319064
  
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -54,15 +54,12 @@ private[spark] class DiskStore(blockManager: 
BlockManager, diskManager: DiskBloc
 val finishTime = System.currentTimeMillis
 logDebug("Block %s stored as %s file on disk in %d ms".format(
   file.getName, Utils.bytesToString(bytes.limit), finishTime - 
startTime))
-PutResult(bytes.limit(), Right(bytes.duplicate()))
   }
 
   override def putIterator(
   blockId: BlockId,
   values: Iterator[Any],
-  level: StorageLevel,
-  returnValues: Boolean): PutResult = {
-
+  level: StorageLevel): Right[Iterator[Any], Long] = {
--- End diff --

This return type is really wonky. We can fix this later though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13665][SQL] Separate the concerns of Ha...

[tedyu]

Github user tedyu commented on a diff in the pull request:

https://github.com/apache/spark/pull/11509#discussion_r55318504
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -465,214 +379,165 @@ abstract class OutputWriter {
 }
 
 /**
- * ::Experimental::
- * A [[BaseRelation]] that provides much of the common code required for 
relations that store their
- * data to an HDFS compatible filesystem.
- *
- * For the read path, similar to [[PrunedFilteredScan]], it can eliminate 
unneeded columns and
- * filter using selected predicates before producing an RDD containing all 
matching tuples as
- * [[Row]] objects. In addition, when reading from Hive style partitioned 
tables stored in file
- * systems, it's able to discover partitioning information from the paths 
of input directories, and
- * perform partition pruning before start reading the data. Subclasses of 
[[HadoopFsRelation()]]
- * must override one of the four `buildScan` methods to implement the read 
path.
- *
- * For the write path, it provides the ability to write to both 
non-partitioned and partitioned
- * tables.  Directory layout of the partitioned tables is compatible with 
Hive.
- *
- * @constructor This constructor is for internal uses only. The 
[[PartitionSpec]] argument is for
- *  implementing metastore table conversion.
- *
- * @param maybePartitionSpec An [[HadoopFsRelation]] can be created with 
an optional
- *[[PartitionSpec]], so that partition discovery can be skipped.
- *
- * @since 1.4.0
+ * Acts as a container for all of the metadata required to read from a 
datasource. All discovery,
+ * resolution and merging logic for schemas and partitions has been 
removed.
+ *
+ * @param location A [[FileCatalog]] that can enumerate the locations of 
all the files that comprise
+ * this relation.
+ * @param partitionSchema The schmea of the columns (if any) that are used 
to partition the relation
+ * @param dataSchema The schema of any remaining columns.  Note that if 
any partition columns are
+ *   present in the actual data files as well, they are 
removed.
+ * @param bucketSpec Describes the bucketing (hash-partitioning of the 
files by some column values).
+ * @param fileFormat A file format that can be used to read and write the 
data in files.
+ * @param options Configuration used when reading / writing data.
  */
-@Experimental
-abstract class HadoopFsRelation private[sql](
-maybePartitionSpec: Option[PartitionSpec],
-parameters: Map[String, String])
-  extends BaseRelation with FileRelation with Logging {
-
-  override def toString: String = getClass.getSimpleName
+case class HadoopFsRelation(
+sqlContext: SQLContext,
+location: FileCatalog,
+partitionSchema: StructType,
+dataSchema: StructType,
+bucketSpec: Option[BucketSpec],
+fileFormat: FileFormat,
+options: Map[String, String]) extends BaseRelation with FileRelation {
+
+  val schema: StructType = {
+val dataSchemaColumnNames = dataSchema.map(_.name.toLowerCase).toSet
+StructType(dataSchema ++ partitionSchema.filterNot { column =>
+  dataSchemaColumnNames.contains(column.name.toLowerCase)
+})
+  }
 
-  def this() = this(None, Map.empty[String, String])
+  def partitionSchemaOption: Option[StructType] =
+if (partitionSchema.isEmpty) None else Some(partitionSchema)
+  def partitionSpec: PartitionSpec = 
location.partitionSpec(partitionSchemaOption)
 
-  def this(parameters: Map[String, String]) = this(None, parameters)
+  def refresh(): Unit = location.refresh()
 
-  private[sql] def this(maybePartitionSpec: Option[PartitionSpec]) =
-this(maybePartitionSpec, Map.empty[String, String])
+  override def toString: String =
+s"$fileFormat part: ${partitionSchema.simpleString}, data: 
${dataSchema.simpleString}"
 
-  private val hadoopConf = new 
Configuration(sqlContext.sparkContext.hadoopConfiguration)
+  /** Returns the list of files that will be read when scanning this 
relation. */
+  override def inputFiles: Array[String] =
+location.allFiles().map(_.getPath.toUri.toString).toArray
+}
 
-  private var _partitionSpec: PartitionSpec = _
+/**
+ * Used to read a write data in files to [[InternalRow]] format.
--- End diff --

nit: a write -> and write


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[gatorsmile]

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-193616509
  
We still can prune the child even if we do not have `projectList`. See the 
following code:

https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L400-L406

However, this does not cover all the cases. Let me submit a PR for 
improving it. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12718][SPARK-13720][SQL] SQL generation...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11555#issuecomment-193616416
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52626/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12718][SPARK-13720][SQL] SQL generation...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11555#issuecomment-193616413
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12718][SPARK-13720][SQL] SQL generation...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11555#issuecomment-193616299
  
**[Test build #52626 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52626/consoleFull)**
 for PR 11555 at commit 
[`c82229a`](https://github.com/apache/spark/commit/c82229a42efec9131652435b9543df81d1feab6c).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/11565#issuecomment-193614610
  
After think about it more, I think the `projectList` maybe useful to do 
column pruning for `Window`.

If there is a `Project` above `Window`, it's easy to filter out unnecessary 
window expressions. However, it's hard to propagate this information top down. 
The `projectList` can store this information and continue to do column pruning 
under `Window`.

Any ideas? cc @yhuai 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [HOT-FIX][BUILD] Use the new location of `chec...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11567#issuecomment-193613491
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [HOT-FIX][BUILD] Use the new location of `chec...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11567#issuecomment-193613494
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52627/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/11565#discussion_r55317197
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -345,8 +343,6 @@ object ColumnPruning extends Rule[LogicalPlan] {
 // Prunes the unused columns from child of 
Aggregate/Window/Expand/Generate
 case a @ Aggregate(_, _, child) if (child.outputSet -- 
a.references).nonEmpty =>
   a.copy(child = prunedChild(child, a.references))
-case w @ Window(_, _, _, _, child) if (child.outputSet -- 
w.references).nonEmpty =>
--- End diff --

I see, makes sense. But I think it's still possible to do column pruning 
for window, which may require a more complex rule. We can do it in follow-ups.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [HOT-FIX][BUILD] Use the new location of `chec...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11567#issuecomment-193612972
  
**[Test build #52627 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52627/consoleFull)**
 for PR 11567 at commit 
[`4a58fba`](https://github.com/apache/spark/commit/4a58fba530df6e4b665389804908d04da88e7d4f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11444#issuecomment-193611600
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11444#issuecomment-193611603
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52629/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11444#issuecomment-193611469
  
**[Test build #52629 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52629/consoleFull)**
 for PR 11444 at commit 
[`77ff36b`](https://github.com/apache/spark/commit/77ff36baa992f4350d7c2650bdda1d267cdc0e77).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13692][CORE][SQL] Fix trivial Coverity/...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11530#issuecomment-193611280
  
**[Test build #52631 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52631/consoleFull)**
 for PR 11530 at commit 
[`9a0f8fa`](https://github.com/apache/spark/commit/9a0f8fabeccf56800dd8af74c39f14a99b8041a7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL] Parse Hive DDL commands our...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/11573#issuecomment-193610715
  
Note: The only changes I made on top of #11048 is addressing the 
outstanding comments in that patch and some minor clean ups. It's entirely 
possible that there still are things that are missing or incorrect given the 
original patch was not reviewed completely yet.

@hvanhovell @yhuai PTAL.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13692][CORE][SQL] Fix trivial Coverity/...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/11530#issuecomment-193610488
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13711][Core]Don't call SparkUncaughtExc...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11566


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL] Parse Hive DDL commands our...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11573#issuecomment-193609366
  
**[Test build #52630 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52630/consoleFull)**
 for PR 11573 at commit 
[`a663b5c`](https://github.com/apache/spark/commit/a663b5ccca7718b41df8b8ede462ac6c9e0b8e8f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL] Parse Hive DDL commands our...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/11573#discussion_r55316476
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.Logging
+import org.apache.spark.sql.{Row, SQLContext}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.expressions.{Attribute, 
AttributeReference}
+import org.apache.spark.sql.execution.datasources.BucketSpec
+import org.apache.spark.sql.types._
+
+
+/**
+ * A DDL command expected to be run in the underlying system without Spark 
parsing the
+ * query text.
+ */
+abstract class NativeDDLCommands(val sql: String) extends RunnableCommand {
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+sqlContext.runNativeSql(sql)
+  }
+
+  override val output: Seq[Attribute] = {
+Seq(AttributeReference("result", StringType, nullable = false)())
+  }
+
+}
+
+case class CreateDatabase(
+databaseName: String,
+allowExisting: Boolean,
+path: Option[String],
+comment: Option[String],
+props: Map[String, String])(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class CreateFunction(
+functionName: String,
+alias: String,
+resourcesMap: Map[String, String],
+isTemp: Boolean)(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableRename(
+tableName: TableIdentifier,
+renameTableName: TableIdentifier)(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableSetProperties(
+tableName: TableIdentifier,
+setProperties: Map[String, Option[String]])(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableDropProperties(
+tableName: TableIdentifier,
+dropProperties: Map[String, Option[String]],
+allowExisting: Boolean)(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableSerDeProperties(
+tableName: TableIdentifier,
+serdeClassName: Option[String],
+serdeProperties: Option[Map[String, Option[String]]],
+partition: Option[Map[String, Option[String]]])(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableStoreProperties(
+tableName: TableIdentifier,
+buckets: Option[BucketSpec],
+// TODO: use `clustered` and `sorted` instead for simplicity
+noClustered: Boolean,
+noSorted: Boolean)(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableSkewed(
+tableName: TableIdentifier,
+skewedCols: Seq[String],
+skewedValues: Seq[Seq[String]],
+storedAsDirs: Boolean,
+notSkewed: Boolean,
+// TODO: what??
+notStoredAsDirs: Boolean)(sql: String)
--- End diff --

@viirya I'm really confused about this flag. We have another one called 
`storedAsDirs` and this one says `notStoredAsDirs`. It would seem that one is 
always the opposite of the other, but that's not actually the case in tests, 
e.g.

```
// DDLCommandSuite, "alter table: skewed"
val expected3 = AlterTableSkewed(
  TableIdentifier("table_name", None),
  Seq("dt", "country"),
  Seq(List("2008-08-08", "us"), List("2009-09-09", "uk")),
  storedAsDirs = false,
  notSkewed = false,
  notStoredAsDirs = false)(sql3)
```
seems contradictory?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-13139][SQL] Parse Hive DDL commands our...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/11573#discussion_r55316403
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.Logging
+import org.apache.spark.sql.{Row, SQLContext}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.expressions.{Attribute, 
AttributeReference}
+import org.apache.spark.sql.execution.datasources.BucketSpec
+import org.apache.spark.sql.types._
+
+
+/**
+ * A DDL command expected to be run in the underlying system without Spark 
parsing the
+ * query text.
+ */
+abstract class NativeDDLCommands(val sql: String) extends RunnableCommand {
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+sqlContext.runNativeSql(sql)
+  }
+
+  override val output: Seq[Attribute] = {
+Seq(AttributeReference("result", StringType, nullable = false)())
+  }
+
+}
+
+case class CreateDatabase(
+databaseName: String,
+allowExisting: Boolean,
+path: Option[String],
+comment: Option[String],
+props: Map[String, String])(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class CreateFunction(
+functionName: String,
+alias: String,
+resourcesMap: Map[String, String],
+isTemp: Boolean)(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableRename(
+tableName: TableIdentifier,
+renameTableName: TableIdentifier)(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableSetProperties(
+tableName: TableIdentifier,
+setProperties: Map[String, Option[String]])(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableDropProperties(
+tableName: TableIdentifier,
+dropProperties: Map[String, Option[String]],
+allowExisting: Boolean)(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableSerDeProperties(
+tableName: TableIdentifier,
+serdeClassName: Option[String],
+serdeProperties: Option[Map[String, Option[String]]],
+partition: Option[Map[String, Option[String]]])(sql: String)
+  extends NativeDDLCommands(sql) with Logging
+
+case class AlterTableStoreProperties(
+tableName: TableIdentifier,
+buckets: Option[BucketSpec],
+// TODO: use `clustered` and `sorted` instead for simplicity
+noClustered: Boolean,
+noSorted: Boolean)(sql: String)
--- End diff --

@viirya was there any reason why these have to be negative? It's much 
easier to understand if it's positive, i.e. `clustered` and `sorted`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12719][SQL] SQL generation support for ...

[dilipbiswal]

Github user dilipbiswal commented on the pull request:

https://github.com/apache/spark/pull/11563#issuecomment-193609126
  
@rxin Thanks for the input. Let me try to work on it and see if i encounter 
any issues


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13711][Core]Don't call SparkUncaughtExc...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/11566#issuecomment-193609085
  
Thanks. Merging to master and 1.6


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL] Parse Hive DDL commands our...

[andrewor14]

GitHub user andrewor14 opened a pull request:

https://github.com/apache/spark/pull/11573

[SPARK-13139][SQL] Parse Hive DDL commands ourselves

## What changes were proposed in this pull request?

This patch is ported over from @viirya's changes in #11048. Currently for 
most DDLs we just pass the query text directly to Hive. Instead, we should 
parse these commands ourselves and in the future (not part of this patch) use 
the `HiveCatalog` to process these DDLs. This is a pretext to merging 
`SQLContext` and `HiveContext`.

Note: As of this patch we still pass the query text to Hive. The difference 
is that we now parse the commands ourselves so in the future we can just use 
our own catalog.

## How was this patch tested?

Jenkins, new `DDLCommandSuite`, which comprises of about 40% of the changes 
here.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewor14/spark parser-plus-plus

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11573.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11573


commit fc3c1684ad8e24ab9b05f0f7e02659ea2e365ebd
Author: Andrew Or 
Date:   2016-03-04T22:20:07Z

Move things into new ParserUtils object

commit adcb561a6bcb91a946448b8a5601155c9d714675
Author: Andrew Or 
Date:   2016-03-07T22:40:48Z

Merge branch 'master' of github.com:apache/spark into parser-plus-plus

commit 010afddf40d776b3009cee98057e74d499c45012
Author: Andrew Or 
Date:   2016-03-07T23:56:09Z

Port over viirya's changes in #11048

commit 007907460d72a3aa82b222567c08589c62efb614
Author: Andrew Or 
Date:   2016-03-08T04:43:31Z

Address comments from #11408 + fix style

commit 02de9b771cc74abb6f1971081411bd6d8c4c4b5f
Author: Andrew Or 
Date:   2016-03-08T04:45:43Z

Merge branch 'master' of github.com:apache/spark into parser-plus-plus

Conflicts:

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/bucket.scala

commit 3766f83bb5551c0fd978117c3f41efb99257c984
Author: Andrew Or 
Date:   2016-03-08T04:49:22Z

Minor fixes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13737][SQL][wip]Add getOrCreate method ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11571#issuecomment-193608028
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52623/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13737][SQL][wip]Add getOrCreate method ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11571#issuecomment-193608023
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/11444#discussion_r55316023
  
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,139 @@ def __eq__(self, other):
   for v in [ArrayType, MapType, StructType])
 
 
+_FIXED_DECIMAL = re.compile("decimal\\((\\d+),(\\d+)\\)")
+
+
+def _parse_basic_datatype_string(s):
+if s == "null":
+return NullType()
+elif s == "boolean":
+return BooleanType()
+elif s == "byte":
+return ByteType()
+elif s == "short":
+return ShortType()
+elif s == "int":
+return IntegerType()
+elif s == "long":
+return LongType()
+elif s == "float":
+return FloatType()
+elif s == "double":
+return DoubleType()
+elif s == "decimal":
+return DecimalType()
+elif _FIXED_DECIMAL.match(s):
+m = _FIXED_DECIMAL.match(json_value)
+return DecimalType(int(m.group(1)), int(m.group(2)))
+elif s == "string":
+return StringType()
+elif s == "date":
+return DateType()
+elif s == "timestamp":
+return TimestampType()
+elif s == "binary":
+return BinaryType()
+else:
+raise ValueError("Cannot parse datatype string: %s" % s)
+
+
+def _ignore_brackets_split(s, separator):
+parts = []
+buf = ""
+level = 0
+for c in s:
+if c == "<":
+level += 1
+buf += c
+elif c == ">":
+if level == 0:
+raise ValueError("Cannot parse datatype string: %s" % s)
+level -= 1
+buf += c
+elif c == separator and level > 0:
+buf += c
+elif c == separator:
+parts.append(buf)
+buf = ""
+else:
+buf += c
+
+if len(buf) == 0:
+raise ValueError("Cannot parse datatype string: %s" % s)
+parts.append(buf)
+return parts
+
+
+def _parse_struct_type_string(s):
+parts = _ignore_brackets_split(s, ",")
+fields = []
+for part in parts:
+name_and_type = _ignore_brackets_split(part, ":")
+if len(name_and_type) != 2:
+raise ValueError("Cannot parse datatype string: %s" % s)
+field_name = name_and_type[0].strip()
+field_type = _parse_datatype_string(name_and_type[1])
+fields.append(StructField(field_name, field_type))
+return StructType(fields)
+
+
+def _parse_datatype_string(s):
--- End diff --

not now, should we support it? It will make the parser more complicated 
though...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13737][SQL][wip]Add getOrCreate method ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11571#issuecomment-193607352
  
**[Test build #52623 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52623/consoleFull)**
 for PR 11571 at commit 
[`a64a0a4`](https://github.com/apache/spark/commit/a64a0a4bb9dad43b837678f06f45e7a15215826f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11444#issuecomment-193604622
  
**[Test build #52629 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52629/consoleFull)**
 for PR 11444 at commit 
[`77ff36b`](https://github.com/apache/spark/commit/77ff36baa992f4350d7c2650bdda1d267cdc0e77).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13695] Don't cache MEMORY_AND_DISK bloc...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11533#issuecomment-193600272
  
**[Test build #2615 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2615/consoleFull)**
 for PR 11533 at commit 
[`8f332a7`](https://github.com/apache/spark/commit/8f332a7c14aff8aebfd8b36ec56fa33b8330605e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13732] [SQL] Remove projectList from Wi...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/11565#discussion_r55314448
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -345,8 +343,6 @@ object ColumnPruning extends Rule[LogicalPlan] {
 // Prunes the unused columns from child of 
Aggregate/Window/Expand/Generate
 case a @ Aggregate(_, _, child) if (child.outputSet -- 
a.references).nonEmpty =>
   a.copy(child = prunedChild(child, a.references))
-case w @ Window(_, _, _, _, child) if (child.outputSet -- 
w.references).nonEmpty =>
--- End diff --

Yeah. There is a todo mark at the end of that suite. I plan to do it when I 
finish the top items in the to-do list. If you want to take it, feel free to 
start it. Thanks!
```
// todo: add more tests for column pruning
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13657] [SQL] Support parsing very long ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11501#issuecomment-193595713
  
**[Test build #2616 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2616/consoleFull)**
 for PR 11501 at commit 
[`c187554`](https://github.com/apache/spark/commit/c187554382ed1921c29b3a017a8ea6270eeec50f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13404] [SQL] Create variables for input...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11274


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13404] [SQL] Create variables for input...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/11274#issuecomment-193595238
  
Merging this into master, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11572#issuecomment-193593900
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52625/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11572#issuecomment-193593899
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13667][SQL] Support for specifying cust...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11550#issuecomment-193593849
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11572#issuecomment-193593805
  
**[Test build #52625 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52625/consoleFull)**
 for PR 11572 at commit 
[`cf7c719`](https://github.com/apache/spark/commit/cf7c719b72896450affad9b866ad9077a6140e40).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13667][SQL] Support for specifying cust...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11550#issuecomment-193593851
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52619/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13667][SQL] Support for specifying cust...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11550#issuecomment-193593721
  
**[Test build #52619 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52619/consoleFull)**
 for PR 11550 at commit 
[`db27259`](https://github.com/apache/spark/commit/db27259629721f2e584457b4e5739baabfd851ea).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class DefaultSource extends FileFormat with DataSourceRegister `
  * `class InMemoryCatalog extends ExternalCatalog `
  * `abstract class ExternalCatalog `
  * `case class CatalogTablePartition(`
  * `case class WriteRelation(`
  * `class DefaultSource extends FileFormat with DataSourceRegister `
  * `class DefaultSource extends FileFormat with DataSourceRegister `
  * `  case class FileTypes(`
  * `class DefaultSource extends FileFormat with DataSourceRegister `
  * `case class HadoopFsRelation(`
  * `trait FileFormat `
  * `trait FileCatalog `
  * `class HDFSFileCatalog(`
  * `class HiveFileCatalog(`
  * `.doc(\"A comma-separated list of class names of services to add to 
the scheduler.\")`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-529] [sql] Modify SQLConf to use new co...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11570#issuecomment-193591065
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52622/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-529] [sql] Modify SQLConf to use new co...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11570#issuecomment-193591062
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-529] [sql] Modify SQLConf to use new co...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11570#issuecomment-193590521
  
**[Test build #52622 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52622/consoleFull)**
 for PR 11570 at commit 
[`884926c`](https://github.com/apache/spark/commit/884926c76e0403eca0aba43319eb28c37eca2e66).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `.doc(\"The output committer class used by Parquet. The specified 
class needs to be a \" +`
  * `.doc(\"A comma separated list of class prefixes that should be 
loaded using the classloader \" +`
  * `.doc(\"A comma separated list of class prefixes that should 
explicitly be reloaded for each \" +`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13734][SPARKR] Added histogram function

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11569#issuecomment-193585812
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org