[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-214939215
  
@liancheng Thank you very much Lian !!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-214938170
  
@dilipbiswal Great! LGTM now, I'm merging this one to master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-214876293
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-214876307
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57014/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-214876009
  
**[Test build #57014 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57014/consoleFull)**
 for PR 1 at commit 
[`54ff5e4`](https://github.com/apache/spark/commit/54ff5e4a444f93b785a31dc927152461d0c9dfca).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-214847459
  
@liancheng Hi Lian, i just pushed my branch which has the new test case for 
your reference. It also has all your comments addressed :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-214846747
  
**[Test build #57014 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57014/consoleFull)**
 for PR 1 at commit 
[`54ff5e4`](https://github.com/apache/spark/commit/54ff5e4a444f93b785a31dc927152461d0c9dfca).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-214815027
  
@liancheng I had all the comments addressed and had left the tests running 
last night was about to push today :-). Anyways thanks !! I will take a look at 
your PR.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-214803024
  
Hi @dilipbiswal, we did a bunch of major refactoring on the master branch, 
and it's pretty close to 2.0 code freeze, so I took it over based on your 
version and opened PR #12703. Would you mind to have a look at it? Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r61027608
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
 ---
@@ -122,4 +134,105 @@ class HiveCommandSuite extends QueryTest with 
SQLTestUtils with TestHiveSingleto
   checkAnswer(sql("SHOW TBLPROPERTIES parquet_temp"), Nil)
 }
   }
+
+  test("show columns") {
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab3"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN default.parquet_tab3"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab3 FROM default"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab4 IN default"),
+  Row("price") :: Row("qty") :: Row("year") :: Row("month") :: Nil)
+
+val message = intercept[NoSuchTableException] {
+  sql("SHOW COLUMNS IN badtable FROM default")
+}.getMessage
+assert(message.contains("badtable not found in database"))
+  }
+
+  test("show partitions - show everything") {
+checkAnswer(
+  sql("show partitions parquet_tab4"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") ::
+Row("year=2016/month=2") ::
+Row("year=2016/month=3") :: Nil)
--- End diff --

@liancheng Thanks !! You are right !! Having >= 5 partition keys does 
expose the problem. Any advice on how to go about handling this ? Can we change 
TablePartitionSpec to be a LinkedHashMap instead ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r61011875
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
 ---
@@ -424,7 +424,7 @@ private[sql] object PartitioningUtils {
 path.foreach { c =>
   if (needsEscaping(c)) {
 builder.append('%')
-builder.append(f"${c.asInstanceOf[Int]}%02x")
+builder.append(f"${c.asInstanceOf[Int]}%02X")
--- End diff --

Makes sense. Thanks for the explanation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r61006959
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
 ---
@@ -424,7 +424,7 @@ private[sql] object PartitioningUtils {
 path.foreach { c =>
   if (needsEscaping(c)) {
 builder.append('%')
-builder.append(f"${c.asInstanceOf[Int]}%02x")
+builder.append(f"${c.asInstanceOf[Int]}%02X")
--- End diff --

@liancheng So i was comparing the output of our impl against hive. And hive 
reports the escaped name in upper case,  I looked at FileUtils.escapePathName() 
in hive as reference before changing the code here. Please let me know what you 
think.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r61004951
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +426,102 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
--- End diff --

@liancheng I followed other commands and some of them mention the SQL 
syntax. Do you want them removed from all other commands or just this two ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r61004685
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +426,102 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
--- End diff --

ok.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r61003756
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -277,9 +277,18 @@ class InMemoryCatalog extends ExternalCatalog {
 catalog(db).tables(table).partitions(spec)
   }
 
+  /**
+   * List the metadata of all partitions that belong to the specified 
table, assuming it exists.
+   *
+   * A partial partition spec may optionally be provided to filter the 
partitions returned.
+   * For instance, if there exist partitions (a='1', b='2'), (a='1', 
b='3') and (a='2', b='4'),
+   * then a partial spec of (a='1') will return the first two only.
--- End diff --

Sure. I will remove the comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60937575
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
 ---
@@ -36,12 +38,22 @@ class HiveCommandSuite extends QueryTest with 
SQLTestUtils with TestHiveSingleto
 |STORED AS PARQUET
 |TBLPROPERTIES('prop1Key'="prop1Val", '`prop2Key`'="prop2Val")
   """.stripMargin)
+ sql("CREATE TABLE parquet_tab3(col1 int, `col 2` int)")
+ sql("CREATE TABLE parquet_tab4 (price int, qty int) partitioned by 
(year int, month int)")
+ sql("INSERT INTO parquet_tab4 PARTITION(year = 2015, month=1) SELECT 
1,1")
+ sql("INSERT INTO parquet_tab4 PARTITION(year = 2015, month=2) SELECT 
2,2")
+ sql("INSERT INTO parquet_tab4 PARTITION(year = 2016, month=2) SELECT 
3,3")
+ sql("INSERT INTO parquet_tab4 PARTITION(year = 2016, month=3) SELECT 
3,3")
--- End diff --

Nit: Spaces around `=`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60937498
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
 ---
@@ -17,11 +17,13 @@
 
 package org.apache.spark.sql.hive.execution
 
-import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
-import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row, SaveMode}
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException
+import org.apache.spark.sql.hive.test.{TestHive, TestHiveSingleton}
 import org.apache.spark.sql.test.SQLTestUtils
 
 class HiveCommandSuite extends QueryTest with SQLTestUtils with 
TestHiveSingleton {
+  import testImplicits._
protected override def beforeAll(): Unit = {
--- End diff --

Please help fix this indentation, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60934317
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
 ---
@@ -424,7 +424,7 @@ private[sql] object PartitioningUtils {
 path.foreach { c =>
   if (needsEscaping(c)) {
 builder.append('%')
-builder.append(f"${c.asInstanceOf[Int]}%02x")
+builder.append(f"${c.asInstanceOf[Int]}%02X")
--- End diff --

Why this change is necessary?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60934120
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +426,102 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+sqlContext.sessionState.catalog.getTableMetadata(table).schema.map { c 
=>
+  Row(c.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+spec: Option[TablePartitionSpec]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  def getPartName(spec: TablePartitionSpec): String = {
+spec.map {s =>
+  PartitioningUtils.escapePathName(s._1) + "=" + 
PartitioningUtils.escapePathName(s._2)
+}.mkString("/")
+  }
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val catalog = sqlContext.sessionState.catalog
+val db = table.database.getOrElse(catalog.getCurrentDatabase)
+if (catalog.isTemporaryTable(table)) {
+  throw new AnalysisException("SHOW PARTITIONS is not allowed on a 
temporary table: " +
+  s"${table.unquotedString}")
+} else {
+  val tab = catalog.getTableMetadata(table)
+  /**
+   * Validate and throws an [[AnalysisException]] exception under the 
following conditions:
+   * 1. If the table is not partitioned.
+   * 2. If it is a datasource table.
+   * 3. If it is a view or index table.
+   */
+  if (tab.tableType == CatalogTableType.VIRTUAL_VIEW ||
+tab.tableType == CatalogTableType.INDEX_TABLE) {
+throw new AnalysisException("SHOW PARTITIONS is not allowed on a 
view or index table: " +
+  s"${tab.qualifiedName}")
+  }
+  if (!DDLUtils.isTablePartitioned(tab)) {
+throw new AnalysisException("SHOW PARTITIONS is not allowed on a 
table that is not " +
+  s"partitioned: ${tab.qualifiedName}")
+  }
+  if (DDLUtils.isDatasourceTable(tab)) {
+throw new AnalysisException("SHOW PARTITIONS is not allowed on a 
datasource table: " +
+  s"${tab.qualifiedName}")
+  }
+  /**
+   * Validate the partitioning spec by making sure all the referenced 
columns are
+   * defined as partitioning columns in table definition. An 
AnalysisException exception is
+   * thrown if the partitioning spec is invalid.
+   */
+  if (spec.isDefined) {
+val badColumns = 
spec.get.keySet.filterNot(tab.partitionColumns.map(_.name).contains)
+if (badColumns.nonEmpty) {
+  throw new AnalysisException(
+s"Non-partitioned column(s) [${badColumns.mkString(", ")}] are 
" +
--- End diff --

Nit: Non-partitioning


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60931135
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
 ---
@@ -122,4 +134,105 @@ class HiveCommandSuite extends QueryTest with 
SQLTestUtils with TestHiveSingleto
   checkAnswer(sql("SHOW TBLPROPERTIES parquet_temp"), Nil)
 }
   }
+
+  test("show columns") {
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab3"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN default.parquet_tab3"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab3 FROM default"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab4 IN default"),
+  Row("price") :: Row("qty") :: Row("year") :: Row("month") :: Nil)
+
+val message = intercept[NoSuchTableException] {
+  sql("SHOW COLUMNS IN badtable FROM default")
+}.getMessage
+assert(message.contains("badtable not found in database"))
+  }
+
+  test("show partitions - show everything") {
+checkAnswer(
+  sql("show partitions parquet_tab4"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") ::
+Row("year=2016/month=2") ::
+Row("year=2016/month=3") :: Nil)
--- End diff --

As a simple experiment, order is preserved for maps with less than 5 
elements:

```scala
scala> Map("year" -> 1) foreach println
(year,1)

scala> Map("year" -> 1, "month" -> 2) foreach println
(year,1)
(month,2)

scala> Map("year" -> 1, "month" -> 2, "day" -> 3) foreach println
(year,1)
(month,2)
(day,3)

scala> Map("year" -> 1, "month" -> 2, "day" -> 3, "hour" -> 4) foreach 
println
(year,1)
(month,2)
(day,3)
(hour,4)

scala> Map("year" -> 1, "month" -> 2, "day" -> 3, "hour" -> 4, "minute" -> 
5) foreach println
(minute,5)
(year,1)
(hour,4)
(day,3)
(month,2)
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60930580
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
 ---
@@ -122,4 +134,105 @@ class HiveCommandSuite extends QueryTest with 
SQLTestUtils with TestHiveSingleto
   checkAnswer(sql("SHOW TBLPROPERTIES parquet_temp"), Nil)
 }
   }
+
+  test("show columns") {
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab3"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN default.parquet_tab3"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab3 FROM default"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab4 IN default"),
+  Row("price") :: Row("qty") :: Row("year") :: Row("month") :: Nil)
+
+val message = intercept[NoSuchTableException] {
+  sql("SHOW COLUMNS IN badtable FROM default")
+}.getMessage
+assert(message.contains("badtable not found in database"))
+  }
+
+  test("show partitions - show everything") {
+checkAnswer(
+  sql("show partitions parquet_tab4"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") ::
+Row("year=2016/month=2") ::
+Row("year=2016/month=3") :: Nil)
--- End diff --

Could you please add test cases where the testing table containing >= 5 
partition columns? I believe Scala standard library provides specialized 
classes for maps containing <= 4 elements, which covers the out-of-order bug.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60928725
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +426,102 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+sqlContext.sessionState.catalog.getTableMetadata(table).schema.map { c 
=>
+  Row(c.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+spec: Option[TablePartitionSpec]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  def getPartName(spec: TablePartitionSpec): String = {
+spec.map {s =>
+  PartitioningUtils.escapePathName(s._1) + "=" + 
PartitioningUtils.escapePathName(s._2)
+}.mkString("/")
--- End diff --

@rxin I'm thinking it might not be a good idea to use `Map` to represent 
partition spec in `CatalogTablePartition` since it doesn't preserve partition 
column order. What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60927623
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +426,102 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+sqlContext.sessionState.catalog.getTableMetadata(table).schema.map { c 
=>
+  Row(c.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+spec: Option[TablePartitionSpec]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  def getPartName(spec: TablePartitionSpec): String = {
+spec.map {s =>
+  PartitioningUtils.escapePathName(s._1) + "=" + 
PartitioningUtils.escapePathName(s._2)
+}.mkString("/")
+  }
--- End diff --

New line here please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60927757
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +426,102 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+sqlContext.sessionState.catalog.getTableMetadata(table).schema.map { c 
=>
+  Row(c.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+spec: Option[TablePartitionSpec]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  def getPartName(spec: TablePartitionSpec): String = {
+spec.map {s =>
+  PartitioningUtils.escapePathName(s._1) + "=" + 
PartitioningUtils.escapePathName(s._2)
+}.mkString("/")
+  }
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val catalog = sqlContext.sessionState.catalog
+val db = table.database.getOrElse(catalog.getCurrentDatabase)
+if (catalog.isTemporaryTable(table)) {
+  throw new AnalysisException("SHOW PARTITIONS is not allowed on a 
temporary table: " +
+  s"${table.unquotedString}")
--- End diff --

Wrong indentation here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60927505
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +426,102 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+sqlContext.sessionState.catalog.getTableMetadata(table).schema.map { c 
=>
+  Row(c.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+spec: Option[TablePartitionSpec]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  def getPartName(spec: TablePartitionSpec): String = {
--- End diff --

Mark this as private.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60927399
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +426,102 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+sqlContext.sessionState.catalog.getTableMetadata(table).schema.map { c 
=>
+  Row(c.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+spec: Option[TablePartitionSpec]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  def getPartName(spec: TablePartitionSpec): String = {
+spec.map {s =>
+  PartitioningUtils.escapePathName(s._1) + "=" + 
PartitioningUtils.escapePathName(s._2)
+}.mkString("/")
--- End diff --

I don't think we can simply join all parts using here since 
`TablePartitionSpec` is a `Map` rather than a `Seq`. Order of partition columns 
is not preserved.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60926964
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +426,102 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+sqlContext.sessionState.catalog.getTableMetadata(table).schema.map { c 
=>
+  Row(c.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+spec: Option[TablePartitionSpec]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  def getPartName(spec: TablePartitionSpec): String = {
+spec.map {s =>
--- End diff --

Space after `{`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60926196
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +426,102 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
--- End diff --

We don't need to mention the SQL syntax here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60925953
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +426,102 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
--- End diff --

"A command to list column names of a given table."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60923441
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -277,9 +277,18 @@ class InMemoryCatalog extends ExternalCatalog {
 catalog(db).tables(table).partitions(spec)
   }
 
+  /**
+   * List the metadata of all partitions that belong to the specified 
table, assuming it exists.
+   *
+   * A partial partition spec may optionally be provided to filter the 
partitions returned.
+   * For instance, if there exist partitions (a='1', b='2'), (a='1', 
b='3') and (a='2', b='4'),
+   * then a partial spec of (a='1') will return the first two only.
+   * TODO: Currently partialSpec is not used for memory catalog and it 
returns all the partitions.
--- End diff --

Can we throw exception instead of returning all the partitions if a 
partition spec is given to an in-memory catalog? Silently returning wrong 
answer can be dangerous.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60923122
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -427,13 +427,18 @@ class SessionCatalog(
   }
 
   /**
-   * List all partitions in a table, assuming it exists.
-   * If no database is specified, assume the table is in the current 
database.
+   * List the metadata of all partitions that belong to the specified 
table, assuming it exists.
+   *
+   * A partial partition spec may optionally be provided to filter the 
partitions returned.
+   * For instance, if there exist partitions (a='1', b='2'), (a='1', 
b='3') and (a='2', b='4'),
+   * then a partial spec of (a='1') will return the first two only.
*/
-  def listPartitions(tableName: TableIdentifier): 
Seq[CatalogTablePartition] = {
+  def listPartitions(
--- End diff --

Nit: Missing `override`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60923039
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -427,13 +427,18 @@ class SessionCatalog(
   }
 
   /**
-   * List all partitions in a table, assuming it exists.
-   * If no database is specified, assume the table is in the current 
database.
+   * List the metadata of all partitions that belong to the specified 
table, assuming it exists.
+   *
+   * A partial partition spec may optionally be provided to filter the 
partitions returned.
+   * For instance, if there exist partitions (a='1', b='2'), (a='1', 
b='3') and (a='2', b='4'),
+   * then a partial spec of (a='1') will return the first two only.
--- End diff --

Same as above.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r60922965
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -277,9 +277,18 @@ class InMemoryCatalog extends ExternalCatalog {
 catalog(db).tables(table).partitions(spec)
   }
 
+  /**
+   * List the metadata of all partitions that belong to the specified 
table, assuming it exists.
+   *
+   * A partial partition spec may optionally be provided to filter the 
partitions returned.
+   * For instance, if there exist partitions (a='1', b='2'), (a='1', 
b='3') and (a='2', b='4'),
+   * then a partial spec of (a='1') will return the first two only.
--- End diff --

Nit: No need to repeat the comment here since it's automatically inherited 
from the parent class/trait.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-210346927
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55905/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-210346924
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-210346717
  
**[Test build #55905 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55905/consoleFull)**
 for PR 1 at commit 
[`587263b`](https://github.com/apache/spark/commit/587263b2fd6b33d4d59e71417539b094791e431b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-210317592
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-210317598
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55901/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-210317168
  
**[Test build #55901 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55901/consoleFull)**
 for PR 1 at commit 
[`1171a90`](https://github.com/apache/spark/commit/1171a90b8e2df9f8a95bc289e1d813ded9b441c2).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-210304187
  
**[Test build #55905 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55905/consoleFull)**
 for PR 1 at commit 
[`587263b`](https://github.com/apache/spark/commit/587263b2fd6b33d4d59e71417539b094791e431b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-210294481
  
@andrewor14 Thank you for a detailed review. I have implemented your 
comments. A few points/questions.
1) Currently InMemoryCatalog does not make use of partiton spec. If it is 
ok with you, can i work on it as a follow up ?
2) In listPartitions API , the optional partition spec has a default value 
to avoid changing the callers. Is that a ok thing to do ? If its not, i will 
change the callers to pass None where applicable.

Thanks again for your time and help !!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-210292905
  
**[Test build #55901 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55901/consoleFull)**
 for PR 1 at commit 
[`1171a90`](https://github.com/apache/spark/commit/1171a90b8e2df9f8a95bc289e1d813ded9b441c2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59791626
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
--- End diff --

@andrewor14 Thanks Andrew. Let me re-base now :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59790218
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
--- End diff --

it does now


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59781546
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
--- End diff --

@andrewor14 Hi Andrew, actually i remember i had tried this before. The 
table metadata's schema does not include partition columns ? Thats why i had 
used the lookupRelation.. 

Please let me know what you think ..


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59688027
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 
---
@@ -404,6 +404,28 @@ private[hive] class HiveClientImpl(
 Option(hivePartition).map(fromHivePartition)
   }
 
+  /**
+   * Returns the partition names from hive metastore for a given table in 
a database.
+   */
+  override def getPartitionNames(
+  db: String,
+  table: String,
+  range: Short): Seq[String] = withHiveState {
--- End diff --

Sorry, i should be passing the range to hive's API. I had it hard coded to 
-1 initially and later changed , so that caller of this API can pass in a range 
to restrict the number of partitions that are returned to the client. I will 
make the change to pass the supplied range to hive.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59687762
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -409,6 +409,25 @@ class SessionCatalog(
   }
 
   /**
+   * Returns the partition names from catalog for a given table in a 
database.
+   */
+  def getPartitionNames(db: String, table: String, range: Short): 
Seq[String] = {
+externalCatalog.getPartitionNames(db, table, range)
+  }
+
+  /**
+   * Returns the partition names that matche the partition spec for a 
given table in a database.
+   * When no match is found, an empty Sequence is returned.
+   */
+  def getPartitionNames(
--- End diff --

Sure Andrew. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59687495
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
--- End diff --

ok.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59687465
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+partitionSpec: Option[Map[String, String]]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  /**
+   * This function validates the partitioning spec by making sure all the 
referenced columns are
+   * defined as partitioning columns in table definition. An 
AnalysisException exception is
+   * thrown if the partitioning spec is invalid.
+   */
+  private def validatePartitionSpec(table: CatalogTable, spec: Map[String, 
String]): Unit = {
+if (!spec.keySet.forall(table.partitionColumns.map(_.name).contains)) {
+  throw new AnalysisException(s"Partition spec ${spec.mkString("(", ", 
", ")")} contains " +
+s"non-partition columns")
+}
+  }
+
+  /**
+   * Validates and throws an [[AnalysisException]] exception under the 
following conditions:
+   * 1. If the table is not partitioned.
+   * 2. If it is a datasource table.
+   * 3. If it is a view or index table.
+   */
+  private def checkRequirements(table: CatalogTable): Unit = {
+if (table.tableType == CatalogTableType.VIRTUAL_VIEW ||
+  table.tableType == CatalogTableType.INDEX_TABLE) {
+  throw new AnalysisException("Operation not allowed: view or index 
table")
+} else if (!DDLUtils.isTablePartitioned(table)) {
+  throw new AnalysisException(s"Table ${table.qualifiedName} is not a 
partitioned table")
+} else if (DDLUtils.isDatasourceTable(table)) {
+  throw new AnalysisException("Operation not allowed: datasource 
table")
+}
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val catalog = sqlContext.sessionState.catalog
+val db = table.database.getOrElse(catalog.getCurrentDatabase)
+if (catalog.isTemporaryTable(table)) {
+  Seq.empty[Row]
--- End diff --

OK. will make the change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59687345
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+partitionSpec: Option[Map[String, String]]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  /**
+   * This function validates the partitioning spec by making sure all the 
referenced columns are
+   * defined as partitioning columns in table definition. An 
AnalysisException exception is
+   * thrown if the partitioning spec is invalid.
+   */
+  private def validatePartitionSpec(table: CatalogTable, spec: Map[String, 
String]): Unit = {
+if (!spec.keySet.forall(table.partitionColumns.map(_.name).contains)) {
+  throw new AnalysisException(s"Partition spec ${spec.mkString("(", ", 
", ")")} contains " +
+s"non-partition columns")
+}
+  }
+
+  /**
+   * Validates and throws an [[AnalysisException]] exception under the 
following conditions:
+   * 1. If the table is not partitioned.
+   * 2. If it is a datasource table.
+   * 3. If it is a view or index table.
+   */
+  private def checkRequirements(table: CatalogTable): Unit = {
+if (table.tableType == CatalogTableType.VIRTUAL_VIEW ||
+  table.tableType == CatalogTableType.INDEX_TABLE) {
+  throw new AnalysisException("Operation not allowed: view or index 
table")
+} else if (!DDLUtils.isTablePartitioned(table)) {
+  throw new AnalysisException(s"Table ${table.qualifiedName} is not a 
partitioned table")
+} else if (DDLUtils.isDatasourceTable(table)) {
+  throw new AnalysisException("Operation not allowed: datasource 
table")
--- End diff --

Will make the change


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59687188
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+partitionSpec: Option[Map[String, String]]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  /**
+   * This function validates the partitioning spec by making sure all the 
referenced columns are
+   * defined as partitioning columns in table definition. An 
AnalysisException exception is
+   * thrown if the partitioning spec is invalid.
+   */
+  private def validatePartitionSpec(table: CatalogTable, spec: Map[String, 
String]): Unit = {
+if (!spec.keySet.forall(table.partitionColumns.map(_.name).contains)) {
+  throw new AnalysisException(s"Partition spec ${spec.mkString("(", ", 
", ")")} contains " +
--- End diff --

Sure. Will make the change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59687220
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+partitionSpec: Option[Map[String, String]]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  /**
+   * This function validates the partitioning spec by making sure all the 
referenced columns are
+   * defined as partitioning columns in table definition. An 
AnalysisException exception is
+   * thrown if the partitioning spec is invalid.
+   */
+  private def validatePartitionSpec(table: CatalogTable, spec: Map[String, 
String]): Unit = {
+if (!spec.keySet.forall(table.partitionColumns.map(_.name).contains)) {
+  throw new AnalysisException(s"Partition spec ${spec.mkString("(", ", 
", ")")} contains " +
+s"non-partition columns")
+}
+  }
+
+  /**
+   * Validates and throws an [[AnalysisException]] exception under the 
following conditions:
+   * 1. If the table is not partitioned.
+   * 2. If it is a datasource table.
+   * 3. If it is a view or index table.
+   */
+  private def checkRequirements(table: CatalogTable): Unit = {
--- End diff --

ok.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59686964
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
 ---
@@ -122,4 +133,103 @@ class HiveCommandSuite extends QueryTest with 
SQLTestUtils with TestHiveSingleto
   checkAnswer(sql("SHOW TBLPROPERTIES parquet_temp"), Nil)
 }
   }
+
+  test("show columns") {
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab3"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN default.parquet_tab3"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab3 FROM default"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab4 IN default"),
+  Row("price") :: Row("qty") :: Row("year") :: Row("month") :: Nil)
+
+val message = intercept[NoSuchTableException] {
+  sql("SHOW COLUMNS IN badtable FROM default")
+}.getMessage
+assert(message.contains("Table badtable not found in database"))
+  }
+
+  test("show partitions - show everything") {
+checkAnswer(
+  sql("show partitions parquet_tab4"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") ::
+Row("year=2016/month=2") ::
+Row("year=2016/month=3") :: Nil)
+
+checkAnswer(
+  sql("show partitions default.parquet_tab4"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") ::
+Row("year=2016/month=2") ::
+Row("year=2016/month=3") :: Nil)
+  }
+
+  test("show partitions - filter") {
+checkAnswer(
+  sql("show partitions default.parquet_tab4 PARTITION(year=2015)"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") :: Nil)
+
+checkAnswer(
+  sql("show partitions default.parquet_tab4 PARTITION(year=2015, 
month=1)"),
+  Row("year=2015/month=1") :: Nil)
+
+checkAnswer(
+  sql("show partitions default.parquet_tab4 PARTITION(month=2)"),
+  Row("year=2015/month=2") ::
+Row("year=2016/month=2") :: Nil)
+  }
+
+  test("show partitions - empty row") {
+withTempTable("parquet_temp") {
+  sql(
+"""
+  |CREATE TEMPORARY TABLE parquet_temp (c1 INT, c2 STRING)
+  |USING org.apache.spark.sql.parquet.DefaultSource
+""".stripMargin)
+  // An empty sequence of row is returned for session temporary table.
+  checkAnswer(sql("SHOW PARTITIONS parquet_temp"), Nil)
+  val message1 = intercept[AnalysisException] {
+sql("SHOW PARTITIONS parquet_tab3")
+  }.getMessage
+  assert(message1.contains("is not a partitioned table"))
+
+  val message2 = intercept[AnalysisException] {
+sql("SHOW PARTITIONS parquet_tab4 PARTITION(abcd=2015, xyz=1)")
+  }.getMessage
+  assert(message2.contains("Partition spec (abcd -> 2015, xyz -> 1) 
contains " +
+"non-partition columns"))
+
+  val message3 = intercept[AnalysisException] {
+sql("SHOW PARTITIONS parquet_view1")
+  }.getMessage
+  assert(message3.contains("Operation not allowed: view or index 
table"))
+}
+  }
+
+  test("show partitions - datasource") {
+import sqlContext.implicits._
+withTable("part_datasrc") {
+  val df = (1 to 3).map(i => (i, s"val_$i", i * 2)).toDF("a", "b", "c")
+  df.write
+.partitionBy("a")
+.format("parquet")
+.mode(SaveMode.Overwrite)
+.saveAsTable("part_datasrc")
+
+  val message1 = intercept[AnalysisException] {
+sql("SHOW PARTITIONS part_datasrc")
+  }.getMessage
+  assert(message1.contains("Operation not allowed: datasource table"))
--- End diff --

ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59686977
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
 ---
@@ -122,4 +133,103 @@ class HiveCommandSuite extends QueryTest with 
SQLTestUtils with TestHiveSingleto
   checkAnswer(sql("SHOW TBLPROPERTIES parquet_temp"), Nil)
 }
   }
+
+  test("show columns") {
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab3"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN default.parquet_tab3"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab3 FROM default"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab4 IN default"),
+  Row("price") :: Row("qty") :: Row("year") :: Row("month") :: Nil)
+
+val message = intercept[NoSuchTableException] {
+  sql("SHOW COLUMNS IN badtable FROM default")
+}.getMessage
+assert(message.contains("Table badtable not found in database"))
+  }
+
+  test("show partitions - show everything") {
+checkAnswer(
+  sql("show partitions parquet_tab4"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") ::
+Row("year=2016/month=2") ::
+Row("year=2016/month=3") :: Nil)
+
+checkAnswer(
+  sql("show partitions default.parquet_tab4"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") ::
+Row("year=2016/month=2") ::
+Row("year=2016/month=3") :: Nil)
+  }
+
+  test("show partitions - filter") {
+checkAnswer(
+  sql("show partitions default.parquet_tab4 PARTITION(year=2015)"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") :: Nil)
+
+checkAnswer(
+  sql("show partitions default.parquet_tab4 PARTITION(year=2015, 
month=1)"),
+  Row("year=2015/month=1") :: Nil)
+
+checkAnswer(
+  sql("show partitions default.parquet_tab4 PARTITION(month=2)"),
+  Row("year=2015/month=2") ::
+Row("year=2016/month=2") :: Nil)
+  }
+
+  test("show partitions - empty row") {
+withTempTable("parquet_temp") {
+  sql(
+"""
+  |CREATE TEMPORARY TABLE parquet_temp (c1 INT, c2 STRING)
+  |USING org.apache.spark.sql.parquet.DefaultSource
+""".stripMargin)
+  // An empty sequence of row is returned for session temporary table.
+  checkAnswer(sql("SHOW PARTITIONS parquet_temp"), Nil)
+  val message1 = intercept[AnalysisException] {
+sql("SHOW PARTITIONS parquet_tab3")
+  }.getMessage
+  assert(message1.contains("is not a partitioned table"))
+
+  val message2 = intercept[AnalysisException] {
+sql("SHOW PARTITIONS parquet_tab4 PARTITION(abcd=2015, xyz=1)")
+  }.getMessage
+  assert(message2.contains("Partition spec (abcd -> 2015, xyz -> 1) 
contains " +
+"non-partition columns"))
+
+  val message3 = intercept[AnalysisException] {
+sql("SHOW PARTITIONS parquet_view1")
+  }.getMessage
+  assert(message3.contains("Operation not allowed: view or index 
table"))
+}
+  }
+
+  test("show partitions - datasource") {
+import sqlContext.implicits._
--- End diff --

ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59675036
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -409,6 +409,25 @@ class SessionCatalog(
   }
 
   /**
+   * Returns the partition names from catalog for a given table in a 
database.
+   */
+  def getPartitionNames(db: String, table: String, range: Short): 
Seq[String] = {
+externalCatalog.getPartitionNames(db, table, range)
+  }
+
+  /**
+   * Returns the partition names that matche the partition spec for a 
given table in a database.
+   * When no match is found, an empty Sequence is returned.
+   */
+  def getPartitionNames(
--- End diff --

+1 we should follow current catalog API especially we already have 
`listPartitions` method.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-209673821
  
@dilipbiswal Thanks for working on this. I think it looks pretty good 
overall. The existing API for show partitions is a little too tied to Hive so I 
suggested an alternative to make it more consistent with the rest of the 
catalog. Other than that there weren't really any major issues.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59636620
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
 ---
@@ -122,4 +133,103 @@ class HiveCommandSuite extends QueryTest with 
SQLTestUtils with TestHiveSingleto
   checkAnswer(sql("SHOW TBLPROPERTIES parquet_temp"), Nil)
 }
   }
+
+  test("show columns") {
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab3"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN default.parquet_tab3"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab3 FROM default"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab4 IN default"),
+  Row("price") :: Row("qty") :: Row("year") :: Row("month") :: Nil)
+
+val message = intercept[NoSuchTableException] {
+  sql("SHOW COLUMNS IN badtable FROM default")
+}.getMessage
+assert(message.contains("Table badtable not found in database"))
+  }
+
+  test("show partitions - show everything") {
+checkAnswer(
+  sql("show partitions parquet_tab4"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") ::
+Row("year=2016/month=2") ::
+Row("year=2016/month=3") :: Nil)
+
+checkAnswer(
+  sql("show partitions default.parquet_tab4"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") ::
+Row("year=2016/month=2") ::
+Row("year=2016/month=3") :: Nil)
+  }
+
+  test("show partitions - filter") {
+checkAnswer(
+  sql("show partitions default.parquet_tab4 PARTITION(year=2015)"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") :: Nil)
+
+checkAnswer(
+  sql("show partitions default.parquet_tab4 PARTITION(year=2015, 
month=1)"),
+  Row("year=2015/month=1") :: Nil)
+
+checkAnswer(
+  sql("show partitions default.parquet_tab4 PARTITION(month=2)"),
+  Row("year=2015/month=2") ::
+Row("year=2016/month=2") :: Nil)
+  }
+
+  test("show partitions - empty row") {
+withTempTable("parquet_temp") {
+  sql(
+"""
+  |CREATE TEMPORARY TABLE parquet_temp (c1 INT, c2 STRING)
+  |USING org.apache.spark.sql.parquet.DefaultSource
+""".stripMargin)
+  // An empty sequence of row is returned for session temporary table.
+  checkAnswer(sql("SHOW PARTITIONS parquet_temp"), Nil)
+  val message1 = intercept[AnalysisException] {
+sql("SHOW PARTITIONS parquet_tab3")
+  }.getMessage
+  assert(message1.contains("is not a partitioned table"))
+
+  val message2 = intercept[AnalysisException] {
+sql("SHOW PARTITIONS parquet_tab4 PARTITION(abcd=2015, xyz=1)")
+  }.getMessage
+  assert(message2.contains("Partition spec (abcd -> 2015, xyz -> 1) 
contains " +
+"non-partition columns"))
+
+  val message3 = intercept[AnalysisException] {
+sql("SHOW PARTITIONS parquet_view1")
+  }.getMessage
+  assert(message3.contains("Operation not allowed: view or index 
table"))
+}
+  }
+
+  test("show partitions - datasource") {
+import sqlContext.implicits._
+withTable("part_datasrc") {
+  val df = (1 to 3).map(i => (i, s"val_$i", i * 2)).toDF("a", "b", "c")
+  df.write
+.partitionBy("a")
+.format("parquet")
+.mode(SaveMode.Overwrite)
+.saveAsTable("part_datasrc")
+
+  val message1 = intercept[AnalysisException] {
+sql("SHOW PARTITIONS part_datasrc")
+  }.getMessage
+  assert(message1.contains("Operation not allowed: datasource table"))
--- End diff --

this assert is way too specific, just make it lower case and grep for 
"operation not allowed"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59636516
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
 ---
@@ -122,4 +133,103 @@ class HiveCommandSuite extends QueryTest with 
SQLTestUtils with TestHiveSingleto
   checkAnswer(sql("SHOW TBLPROPERTIES parquet_temp"), Nil)
 }
   }
+
+  test("show columns") {
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab3"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN default.parquet_tab3"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab3 FROM default"),
+  Row("col1") :: Row("col 2") :: Nil)
+
+checkAnswer(
+  sql("SHOW COLUMNS IN parquet_tab4 IN default"),
+  Row("price") :: Row("qty") :: Row("year") :: Row("month") :: Nil)
+
+val message = intercept[NoSuchTableException] {
+  sql("SHOW COLUMNS IN badtable FROM default")
+}.getMessage
+assert(message.contains("Table badtable not found in database"))
+  }
+
+  test("show partitions - show everything") {
+checkAnswer(
+  sql("show partitions parquet_tab4"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") ::
+Row("year=2016/month=2") ::
+Row("year=2016/month=3") :: Nil)
+
+checkAnswer(
+  sql("show partitions default.parquet_tab4"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") ::
+Row("year=2016/month=2") ::
+Row("year=2016/month=3") :: Nil)
+  }
+
+  test("show partitions - filter") {
+checkAnswer(
+  sql("show partitions default.parquet_tab4 PARTITION(year=2015)"),
+  Row("year=2015/month=1") ::
+Row("year=2015/month=2") :: Nil)
+
+checkAnswer(
+  sql("show partitions default.parquet_tab4 PARTITION(year=2015, 
month=1)"),
+  Row("year=2015/month=1") :: Nil)
+
+checkAnswer(
+  sql("show partitions default.parquet_tab4 PARTITION(month=2)"),
+  Row("year=2015/month=2") ::
+Row("year=2016/month=2") :: Nil)
+  }
+
+  test("show partitions - empty row") {
+withTempTable("parquet_temp") {
+  sql(
+"""
+  |CREATE TEMPORARY TABLE parquet_temp (c1 INT, c2 STRING)
+  |USING org.apache.spark.sql.parquet.DefaultSource
+""".stripMargin)
+  // An empty sequence of row is returned for session temporary table.
+  checkAnswer(sql("SHOW PARTITIONS parquet_temp"), Nil)
+  val message1 = intercept[AnalysisException] {
+sql("SHOW PARTITIONS parquet_tab3")
+  }.getMessage
+  assert(message1.contains("is not a partitioned table"))
+
+  val message2 = intercept[AnalysisException] {
+sql("SHOW PARTITIONS parquet_tab4 PARTITION(abcd=2015, xyz=1)")
+  }.getMessage
+  assert(message2.contains("Partition spec (abcd -> 2015, xyz -> 1) 
contains " +
+"non-partition columns"))
+
+  val message3 = intercept[AnalysisException] {
+sql("SHOW PARTITIONS parquet_view1")
+  }.getMessage
+  assert(message3.contains("Operation not allowed: view or index 
table"))
+}
+  }
+
+  test("show partitions - datasource") {
+import sqlContext.implicits._
--- End diff --

just `import testImplicits._` at the top


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59636373
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+partitionSpec: Option[Map[String, String]]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  /**
+   * This function validates the partitioning spec by making sure all the 
referenced columns are
+   * defined as partitioning columns in table definition. An 
AnalysisException exception is
+   * thrown if the partitioning spec is invalid.
+   */
+  private def validatePartitionSpec(table: CatalogTable, spec: Map[String, 
String]): Unit = {
+if (!spec.keySet.forall(table.partitionColumns.map(_.name).contains)) {
+  throw new AnalysisException(s"Partition spec ${spec.mkString("(", ", 
", ")")} contains " +
+s"non-partition columns")
+}
+  }
+
+  /**
+   * Validates and throws an [[AnalysisException]] exception under the 
following conditions:
+   * 1. If the table is not partitioned.
+   * 2. If it is a datasource table.
+   * 3. If it is a view or index table.
+   */
+  private def checkRequirements(table: CatalogTable): Unit = {
+if (table.tableType == CatalogTableType.VIRTUAL_VIEW ||
+  table.tableType == CatalogTableType.INDEX_TABLE) {
+  throw new AnalysisException("Operation not allowed: view or index 
table")
+} else if (!DDLUtils.isTablePartitioned(table)) {
+  throw new AnalysisException(s"Table ${table.qualifiedName} is not a 
partitioned table")
+} else if (DDLUtils.isDatasourceTable(table)) {
+  throw new AnalysisException("Operation not allowed: datasource 
table")
+}
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val catalog = sqlContext.sessionState.catalog
+val db = table.database.getOrElse(catalog.getCurrentDatabase)
+if (catalog.isTemporaryTable(table)) {
+  Seq.empty[Row]
+} else {
+  val tab = catalog.getTable(table)
+  checkRequirements(tab)
+  val partNames = partitionSpec match {
+case None => catalog.getPartitionNames(db, table.identifier, 
-1.asInstanceOf[Short])
+case Some(spec) =>
+  validatePartitionSpec(tab, spec)
+  catalog.getPartitionNames(db, table.identifier, spec, 
-1.asInstanceOf[Short])
+  }
--- End diff --

if you still want to validate the partition spec just do it in an option 
foreach or something


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: 

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59636345
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+partitionSpec: Option[Map[String, String]]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  /**
+   * This function validates the partitioning spec by making sure all the 
referenced columns are
+   * defined as partitioning columns in table definition. An 
AnalysisException exception is
+   * thrown if the partitioning spec is invalid.
+   */
+  private def validatePartitionSpec(table: CatalogTable, spec: Map[String, 
String]): Unit = {
+if (!spec.keySet.forall(table.partitionColumns.map(_.name).contains)) {
+  throw new AnalysisException(s"Partition spec ${spec.mkString("(", ", 
", ")")} contains " +
+s"non-partition columns")
+}
+  }
+
+  /**
+   * Validates and throws an [[AnalysisException]] exception under the 
following conditions:
+   * 1. If the table is not partitioned.
+   * 2. If it is a datasource table.
+   * 3. If it is a view or index table.
+   */
+  private def checkRequirements(table: CatalogTable): Unit = {
+if (table.tableType == CatalogTableType.VIRTUAL_VIEW ||
+  table.tableType == CatalogTableType.INDEX_TABLE) {
+  throw new AnalysisException("Operation not allowed: view or index 
table")
+} else if (!DDLUtils.isTablePartitioned(table)) {
+  throw new AnalysisException(s"Table ${table.qualifiedName} is not a 
partitioned table")
+} else if (DDLUtils.isDatasourceTable(table)) {
+  throw new AnalysisException("Operation not allowed: datasource 
table")
+}
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val catalog = sqlContext.sessionState.catalog
+val db = table.database.getOrElse(catalog.getCurrentDatabase)
+if (catalog.isTemporaryTable(table)) {
+  Seq.empty[Row]
+} else {
+  val tab = catalog.getTable(table)
+  checkRequirements(tab)
+  val partNames = partitionSpec match {
+case None => catalog.getPartitionNames(db, table.identifier, 
-1.asInstanceOf[Short])
+case Some(spec) =>
+  validatePartitionSpec(tab, spec)
+  catalog.getPartitionNames(db, table.identifier, spec, 
-1.asInstanceOf[Short])
+  }
--- End diff --

if you take my suggestion elsewhere then you don't need to do this match. 
Just pass the option into catalog.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional 

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59636244
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+partitionSpec: Option[Map[String, String]]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  /**
+   * This function validates the partitioning spec by making sure all the 
referenced columns are
+   * defined as partitioning columns in table definition. An 
AnalysisException exception is
+   * thrown if the partitioning spec is invalid.
+   */
+  private def validatePartitionSpec(table: CatalogTable, spec: Map[String, 
String]): Unit = {
+if (!spec.keySet.forall(table.partitionColumns.map(_.name).contains)) {
+  throw new AnalysisException(s"Partition spec ${spec.mkString("(", ", 
", ")")} contains " +
--- End diff --

to provide a better error message I would do:
```
val badColumns = spec.keySet.filterNot(table.partitionColumnNames.contains)
throw new AnalysisException("Non-partitioned column(s) 
${badColumns.mkString(", ")} are specified for SHOW PARTITIONS")
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59635926
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+partitionSpec: Option[Map[String, String]]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  /**
+   * This function validates the partitioning spec by making sure all the 
referenced columns are
+   * defined as partitioning columns in table definition. An 
AnalysisException exception is
+   * thrown if the partitioning spec is invalid.
+   */
+  private def validatePartitionSpec(table: CatalogTable, spec: Map[String, 
String]): Unit = {
+if (!spec.keySet.forall(table.partitionColumns.map(_.name).contains)) {
+  throw new AnalysisException(s"Partition spec ${spec.mkString("(", ", 
", ")")} contains " +
+s"non-partition columns")
+}
+  }
+
+  /**
+   * Validates and throws an [[AnalysisException]] exception under the 
following conditions:
+   * 1. If the table is not partitioned.
+   * 2. If it is a datasource table.
+   * 3. If it is a view or index table.
+   */
+  private def checkRequirements(table: CatalogTable): Unit = {
+if (table.tableType == CatalogTableType.VIRTUAL_VIEW ||
+  table.tableType == CatalogTableType.INDEX_TABLE) {
+  throw new AnalysisException("Operation not allowed: view or index 
table")
+} else if (!DDLUtils.isTablePartitioned(table)) {
+  throw new AnalysisException(s"Table ${table.qualifiedName} is not a 
partitioned table")
+} else if (DDLUtils.isDatasourceTable(table)) {
+  throw new AnalysisException("Operation not allowed: datasource 
table")
--- End diff --

also since these all throw exceptions anyway I would just do `if`s instead 
of `else if`s


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59635881
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+partitionSpec: Option[Map[String, String]]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  /**
+   * This function validates the partitioning spec by making sure all the 
referenced columns are
+   * defined as partitioning columns in table definition. An 
AnalysisException exception is
+   * thrown if the partitioning spec is invalid.
+   */
+  private def validatePartitionSpec(table: CatalogTable, spec: Map[String, 
String]): Unit = {
+if (!spec.keySet.forall(table.partitionColumns.map(_.name).contains)) {
+  throw new AnalysisException(s"Partition spec ${spec.mkString("(", ", 
", ")")} contains " +
+s"non-partition columns")
+}
+  }
+
+  /**
+   * Validates and throws an [[AnalysisException]] exception under the 
following conditions:
+   * 1. If the table is not partitioned.
+   * 2. If it is a datasource table.
+   * 3. If it is a view or index table.
+   */
+  private def checkRequirements(table: CatalogTable): Unit = {
--- End diff --

this is only called in 1 place. I would just inline it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59635840
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+partitionSpec: Option[Map[String, String]]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  /**
+   * This function validates the partitioning spec by making sure all the 
referenced columns are
+   * defined as partitioning columns in table definition. An 
AnalysisException exception is
+   * thrown if the partitioning spec is invalid.
+   */
+  private def validatePartitionSpec(table: CatalogTable, spec: Map[String, 
String]): Unit = {
+if (!spec.keySet.forall(table.partitionColumns.map(_.name).contains)) {
+  throw new AnalysisException(s"Partition spec ${spec.mkString("(", ", 
", ")")} contains " +
+s"non-partition columns")
+}
+  }
+
+  /**
+   * Validates and throws an [[AnalysisException]] exception under the 
following conditions:
+   * 1. If the table is not partitioned.
+   * 2. If it is a datasource table.
+   * 3. If it is a view or index table.
+   */
+  private def checkRequirements(table: CatalogTable): Unit = {
+if (table.tableType == CatalogTableType.VIRTUAL_VIEW ||
+  table.tableType == CatalogTableType.INDEX_TABLE) {
+  throw new AnalysisException("Operation not allowed: view or index 
table")
+} else if (!DDLUtils.isTablePartitioned(table)) {
+  throw new AnalysisException(s"Table ${table.qualifiedName} is not a 
partitioned table")
+} else if (DDLUtils.isDatasourceTable(table)) {
+  throw new AnalysisException("Operation not allowed: datasource 
table")
+}
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val catalog = sqlContext.sessionState.catalog
+val db = table.database.getOrElse(catalog.getCurrentDatabase)
+if (catalog.isTemporaryTable(table)) {
+  Seq.empty[Row]
--- End diff --

I think we should throw an exception instead if it's a temporary table


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59635781
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+partitionSpec: Option[Map[String, String]]) extends RunnableCommand {
+  // The result of SHOW PARTITIONS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  /**
+   * This function validates the partitioning spec by making sure all the 
referenced columns are
+   * defined as partitioning columns in table definition. An 
AnalysisException exception is
+   * thrown if the partitioning spec is invalid.
+   */
+  private def validatePartitionSpec(table: CatalogTable, spec: Map[String, 
String]): Unit = {
+if (!spec.keySet.forall(table.partitionColumns.map(_.name).contains)) {
+  throw new AnalysisException(s"Partition spec ${spec.mkString("(", ", 
", ")")} contains " +
+s"non-partition columns")
+}
+  }
+
+  /**
+   * Validates and throws an [[AnalysisException]] exception under the 
following conditions:
+   * 1. If the table is not partitioned.
+   * 2. If it is a datasource table.
+   * 3. If it is a view or index table.
+   */
+  private def checkRequirements(table: CatalogTable): Unit = {
+if (table.tableType == CatalogTableType.VIRTUAL_VIEW ||
+  table.tableType == CatalogTableType.INDEX_TABLE) {
+  throw new AnalysisException("Operation not allowed: view or index 
table")
+} else if (!DDLUtils.isTablePartitioned(table)) {
+  throw new AnalysisException(s"Table ${table.qualifiedName} is not a 
partitioned table")
+} else if (DDLUtils.isDatasourceTable(table)) {
+  throw new AnalysisException("Operation not allowed: datasource 
table")
--- End diff --

these error messages are too terse; "datasource table" is not an operation. 
I think it should sound more like:
```
SHOW PARTITIONS is not allowed on a view or index table: $name
SHOW PARTITIONS is not allowed on a table that is not partitioned: $name
SHOW PARTITIONS is not allowed on a datasource table: $name
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59635435
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
--- End diff --

It's better to get this information from the table metadata. 1 line:
```
sqlContext.sessionState.catalog.getTableMetadata(table).schema.map(_.name)
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59634825
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -409,6 +409,25 @@ class SessionCatalog(
   }
 
   /**
+   * Returns the partition names from catalog for a given table in a 
database.
+   */
+  def getPartitionNames(db: String, table: String, range: Short): 
Seq[String] = {
+externalCatalog.getPartitionNames(db, table, range)
+  }
+
+  /**
+   * Returns the partition names that matche the partition spec for a 
given table in a database.
+   * When no match is found, an empty Sequence is returned.
+   */
+  def getPartitionNames(
--- End diff --

I think a better API is:
```
def listPartitions(
db: String,
table: String,
partialSpec: Option[TablePartitionSpec]): Seq[CatalogTablePartition]
```

Note that we already have a `listPartitions`, but it doesn't allow us to 
match anything. Once we have this we can do the formatting ourselves in the 
`ShowPartitions` DDL. This leads to a more consistent catalog API and allows us 
to provide a less awkward implementation in `InMemoryCatalog`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59633351
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 
---
@@ -404,6 +404,28 @@ private[hive] class HiveClientImpl(
 Option(hivePartition).map(fromHivePartition)
   }
 
+  /**
+   * Returns the partition names from hive metastore for a given table in 
a database.
+   */
+  override def getPartitionNames(
+  db: String,
+  table: String,
+  range: Short): Seq[String] = withHiveState {
--- End diff --

what is this `range` thing? It seems that it's never used?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/1#discussion_r59631726
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -423,6 +424,100 @@ case class ShowTablePropertiesCommand(
 }
 
 /**
+ * A command for users to list the column names for a table. This function 
creates a
+ * [[ShowColumnsCommand]] logical plan.
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
+ * }}}
+ */
+case class ShowColumnsCommand(table: TableIdentifier) extends 
RunnableCommand {
+  // The result of SHOW COLUMNS has one column called 'result'
+  override val output: Seq[Attribute] = {
+AttributeReference("result", StringType, nullable = false)() :: Nil
+  }
+
+  override def run(sqlContext: SQLContext): Seq[Row] = {
+val relation = sqlContext.sessionState.catalog.lookupRelation(table, 
None)
+relation.schema.fields.map { field =>
+  Row(field.name)
+}
+  }
+}
+
+/**
+ * A command for users to list the partition names of a table. If the 
partition spec is specified,
+ * partitions that match the spec are returned. [[AnalysisException]] 
exception is thrown under
+ * the following conditions:
+ *
+ * 1. If the command is called for a non partitioned table.
+ * 2. If the partition spec refers to the columns that are not defined as 
partitioning columns.
+ *
+ * This function creates a [[ShowPartitionsCommand]] logical plan
+ *
+ * The syntax of using this command in SQL is:
+ * {{{
+ *   SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
+ * }}}
+ */
+case class ShowPartitionsCommand(
+table: TableIdentifier,
+partitionSpec: Option[Map[String, String]]) extends RunnableCommand {
--- End diff --

please use `ExternalCatalog.TablePartitionSpec` here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[gatorsmile]

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-207148007
  
cc @yhuai @andrewor14 @hvanhovell Could you review this PR too? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-206658923
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55169/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-206658922
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-206658766
  
**[Test build #55169 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55169/consoleFull)**
 for PR 1 at commit 
[`7e93a9a`](https://github.com/apache/spark/commit/7e93a9a9d23b130db5354b8e73971ec17cf952a0).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-206636777
  
**[Test build #55169 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55169/consoleFull)**
 for PR 1 at commit 
[`7e93a9a`](https://github.com/apache/spark/commit/7e93a9a9d23b130db5354b8e73971ec17cf952a0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-206636426
  
 retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-206628150
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-206628154
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55167/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-206625736
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55165/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-206625734
  
Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

Github user dilipbiswal commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-206625534
  
Currently i throw an exception for data source tables as in my 
understanding the
partition spec metadata is not available in the hive meta store. I saw a 
[PR-12204] (https://github.com/apache/spark/pull/12204) from @viirya. If with 
this we can have the
partitioning metadata from meta store, then we can remove this restriction. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14445][SQL] Support native execution of...

[dilipbiswal]

GitHub user dilipbiswal opened a pull request:

https://github.com/apache/spark/pull/1

[SPARK-14445][SQL] Support native execution of SHOW COLUMNS and SHOW 
PARTITIONS

## What changes were proposed in this pull request?
This PR adds Native execution of SHOW COLUMNS and SHOW PARTITION commands.

Command Syntax:
``` SQL
SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database]
```
``` SQL
SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
```

## How was this patch tested?

Added test cases in HiveCommandSuite to verify execution and DDLCommandSuite
to verify plans.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dilipbiswal/spark dkb_show_columns

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/1.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1


commit 1ef6b8b3949abad1949e06454dd68a7f8d3a4df1
Author: Dilip Biswal 
Date:   2016-04-01T17:25:41Z

[SPARK-14445] Support native execution of SHOW COLUMNS and SHOW PARTITIONS




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org