[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49813634
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala ---
@@ -372,6 +372,14 @@ private[spark] object SQLConf {
"possible, or you may get wrong result.",
isPublic = false)
+ val CANONICAL_NATIVE_VIEW = booleanConf("spark.sql.nativeView.canonical",
+defaultValue = Some(false),
--- End diff --
Should we enable it by default? So, we can get it tested more.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49813697
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala ---
@@ -372,6 +372,14 @@ private[spark] object SQLConf {
"possible, or you may get wrong result.",
isPublic = false)
+ val CANONICAL_NATIVE_VIEW = booleanConf("spark.sql.nativeView.canonical",
+defaultValue = Some(false),
--- End diff --
Also, should we enable `nativeView` as well?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49813930
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateViewAsSelect.scala
---
@@ -44,26 +47,36 @@ private[hive] case class CreateViewAsSelect(
override def run(sqlContext: SQLContext): Seq[Row] = {
val hiveContext = sqlContext.asInstanceOf[HiveContext]
-if (hiveContext.catalog.tableExists(tableIdentifier)) {
- if (allowExisting) {
+hiveContext.catalog.tableExists(tableIdentifier) match {
+ case true if allowExisting =>
--- End diff --
While you are changing this part, how about we add a comment to show the
sql command that will trigger this case? (also add the comment for `orReplace`
branch?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49819690
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateViewAsSelect.scala
---
@@ -44,26 +47,36 @@ private[hive] case class CreateViewAsSelect(
override def run(sqlContext: SQLContext): Seq[Row] = {
val hiveContext = sqlContext.asInstanceOf[HiveContext]
-if (hiveContext.catalog.tableExists(tableIdentifier)) {
- if (allowExisting) {
+hiveContext.catalog.tableExists(tableIdentifier) match {
+ case true if allowExisting =>
// view already exists, will do nothing, to keep consistent with
Hive
- } else if (orReplace) {
-hiveContext.catalog.client.alertView(prepareTable())
- } else {
+
+ case true if orReplace =>
+hiveContext.catalog.client.alertView(prepareTable(sqlContext))
+
+ case true =>
throw new AnalysisException(s"View $tableIdentifier already
exists. " +
"If you want to update the view definition, please use ALTER
VIEW AS or " +
"CREATE OR REPLACE VIEW AS")
- }
-} else {
- hiveContext.catalog.client.createView(prepareTable())
+
+ case false =>
+hiveContext.catalog.client.createView(prepareTable(sqlContext))
}
Seq.empty[Row]
}
- private def prepareTable(): HiveTable = {
-// setup column types according to the schema of child.
-val schema = if (tableDesc.schema == Nil) {
+ private def prepareTable(sqlContext: SQLContext): HiveTable = {
+val expandedText = if (sqlContext.conf.canonicalView) {
+ rebuildViewQueryString(sqlContext).getOrElse(expandViewText)
+} else {
+ expandViewText
+}
+tableDesc.copy(schema = viewSchema, viewText = Some(expandedText))
+ }
+
+ private def viewSchema: Seq[HiveColumn] = {
--- End diff --
Should we inline this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49819806
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateViewAsSelect.scala
---
@@ -72,27 +85,40 @@ private[hive] case class CreateViewAsSelect(
HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
}
}
+ }
-val columnNames = childSchema.map(f => verbose(f.name))
-
+ private def expandViewText: String = {
--- End diff --
I feel this name (`expandViewText`) is not really suitable. This is the
version that does not canonicalize the view definition, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49820033
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -1356,68 +1356,84 @@ class SQLQuerySuite extends QueryTest with
SQLTestUtils with TestHiveSingleton {
}
}
- test("correctly handle CREATE OR REPLACE VIEW") {
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true") {
- withTable("jt", "jt2") {
-sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
-sql("CREATE OR REPLACE VIEW testView AS SELECT id FROM jt")
-checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+ Seq(true, false).foreach { enabled =>
+val prefix = (if (enabled) "With" else "Without") + " canonical native
view: "
+test(s"$prefix correctly handle CREATE OR REPLACE VIEW") {
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt", "jt2") {
+ sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+ sql("CREATE OR REPLACE VIEW testView AS SELECT id FROM jt")
+ checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+
+ val df = (1 until 10).map(i => i -> i).toDF("i", "j")
+ df.write.format("json").saveAsTable("jt2")
+ sql("CREATE OR REPLACE VIEW testView AS SELECT * FROM jt2")
+ // make sure the view has been changed.
+ checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
+
+ sql("DROP VIEW testView")
+
+ val e = intercept[AnalysisException] {
+sql("CREATE OR REPLACE VIEW IF NOT EXISTS testView AS SELECT
id FROM jt")
+ }
+ assert(e.message.contains("not allowed to define a view"))
+}
+ }
+}
-val df = (1 until 10).map(i => i -> i).toDF("i", "j")
-df.write.format("json").saveAsTable("jt2")
-sql("CREATE OR REPLACE VIEW testView AS SELECT * FROM jt2")
-// make sure the view has been changed.
-checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
+test(s"$prefix correctly handle ALTER VIEW") {
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt", "jt2") {
+ sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+ sql("CREATE VIEW testView AS SELECT id FROM jt")
-sql("DROP VIEW testView")
+ val df = (1 until 10).map(i => i -> i).toDF("i", "j")
+ df.write.format("json").saveAsTable("jt2")
+ sql("ALTER VIEW testView AS SELECT * FROM jt2")
+ // make sure the view has been changed.
+ checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
-val e = intercept[AnalysisException] {
- sql("CREATE OR REPLACE VIEW IF NOT EXISTS testView AS SELECT id
FROM jt")
+ sql("DROP VIEW testView")
}
-assert(e.message.contains("not allowed to define a view"))
}
}
- }
-
- test("correctly handle ALTER VIEW") {
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true") {
- withTable("jt", "jt2") {
-sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
-sql("CREATE VIEW testView AS SELECT id FROM jt")
-val df = (1 until 10).map(i => i -> i).toDF("i", "j")
-df.write.format("json").saveAsTable("jt2")
-sql("ALTER VIEW testView AS SELECT * FROM jt2")
-// make sure the view has been changed.
-checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
-
-sql("DROP VIEW testView")
+test(s"$prefix create hive view for json table") {
+ // json table is not hive-compatible, make sure the new flag fix it.
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt") {
+ sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+ sql("CREATE VIEW testView AS SELECT id FROM jt")
+ checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+ sql("DROP VIEW testView")
+}
}
}
- }
- test("create hive view for json table") {
-// json table is not hive-compatible, make sure the new flag fix it.
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true") {
- with
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49820055
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -1356,68 +1356,84 @@ class SQLQuerySuite extends QueryTest with
SQLTestUtils with TestHiveSingleton {
}
}
- test("correctly handle CREATE OR REPLACE VIEW") {
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true") {
- withTable("jt", "jt2") {
-sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
-sql("CREATE OR REPLACE VIEW testView AS SELECT id FROM jt")
-checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+ Seq(true, false).foreach { enabled =>
+val prefix = (if (enabled) "With" else "Without") + " canonical native
view: "
+test(s"$prefix correctly handle CREATE OR REPLACE VIEW") {
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt", "jt2") {
+ sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+ sql("CREATE OR REPLACE VIEW testView AS SELECT id FROM jt")
+ checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+
+ val df = (1 until 10).map(i => i -> i).toDF("i", "j")
+ df.write.format("json").saveAsTable("jt2")
+ sql("CREATE OR REPLACE VIEW testView AS SELECT * FROM jt2")
+ // make sure the view has been changed.
+ checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
+
+ sql("DROP VIEW testView")
+
+ val e = intercept[AnalysisException] {
+sql("CREATE OR REPLACE VIEW IF NOT EXISTS testView AS SELECT
id FROM jt")
+ }
+ assert(e.message.contains("not allowed to define a view"))
+}
+ }
+}
-val df = (1 until 10).map(i => i -> i).toDF("i", "j")
-df.write.format("json").saveAsTable("jt2")
-sql("CREATE OR REPLACE VIEW testView AS SELECT * FROM jt2")
-// make sure the view has been changed.
-checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
+test(s"$prefix correctly handle ALTER VIEW") {
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt", "jt2") {
+ sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+ sql("CREATE VIEW testView AS SELECT id FROM jt")
-sql("DROP VIEW testView")
+ val df = (1 until 10).map(i => i -> i).toDF("i", "j")
+ df.write.format("json").saveAsTable("jt2")
+ sql("ALTER VIEW testView AS SELECT * FROM jt2")
+ // make sure the view has been changed.
+ checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
-val e = intercept[AnalysisException] {
- sql("CREATE OR REPLACE VIEW IF NOT EXISTS testView AS SELECT id
FROM jt")
+ sql("DROP VIEW testView")
}
-assert(e.message.contains("not allowed to define a view"))
}
}
- }
-
- test("correctly handle ALTER VIEW") {
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true") {
- withTable("jt", "jt2") {
-sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
-sql("CREATE VIEW testView AS SELECT id FROM jt")
-val df = (1 until 10).map(i => i -> i).toDF("i", "j")
-df.write.format("json").saveAsTable("jt2")
-sql("ALTER VIEW testView AS SELECT * FROM jt2")
-// make sure the view has been changed.
-checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
-
-sql("DROP VIEW testView")
+test(s"$prefix create hive view for json table") {
+ // json table is not hive-compatible, make sure the new flag fix it.
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt") {
+ sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+ sql("CREATE VIEW testView AS SELECT id FROM jt")
+ checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+ sql("DROP VIEW testView")
+}
}
}
- }
- test("create hive view for json table") {
-// json table is not hive-compatible, make sure the new flag fix it.
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true") {
- with
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49820229
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -1356,68 +1356,84 @@ class SQLQuerySuite extends QueryTest with
SQLTestUtils with TestHiveSingleton {
}
}
- test("correctly handle CREATE OR REPLACE VIEW") {
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true") {
- withTable("jt", "jt2") {
-sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
-sql("CREATE OR REPLACE VIEW testView AS SELECT id FROM jt")
-checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+ Seq(true, false).foreach { enabled =>
--- End diff --
I am wondering if we should add more tests to really test cases that will
break if we do not canonicalize view, like
* After creating the view, we switch to another database having a table
with the same name as the one used in the view.
* After creating the view, we add more columns to the table.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49957679
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala ---
@@ -372,6 +372,14 @@ private[spark] object SQLConf {
"possible, or you may get wrong result.",
isPublic = false)
+ val CANONICAL_NATIVE_VIEW = booleanConf("spark.sql.nativeView.canonical",
+defaultValue = Some(false),
--- End diff --
I'm for enabling canonical native view by default, but not sure whether
native view should be enabled by default. After all, native view is still
feature incomplete, no matter canonicalized or not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49957685
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateViewAsSelect.scala
---
@@ -44,26 +47,36 @@ private[hive] case class CreateViewAsSelect(
override def run(sqlContext: SQLContext): Seq[Row] = {
val hiveContext = sqlContext.asInstanceOf[HiveContext]
-if (hiveContext.catalog.tableExists(tableIdentifier)) {
- if (allowExisting) {
+hiveContext.catalog.tableExists(tableIdentifier) match {
+ case true if allowExisting =>
// view already exists, will do nothing, to keep consistent with
Hive
- } else if (orReplace) {
-hiveContext.catalog.client.alertView(prepareTable())
- } else {
+
+ case true if orReplace =>
+hiveContext.catalog.client.alertView(prepareTable(sqlContext))
+
+ case true =>
throw new AnalysisException(s"View $tableIdentifier already
exists. " +
"If you want to update the view definition, please use ALTER
VIEW AS or " +
"CREATE OR REPLACE VIEW AS")
- }
-} else {
- hiveContext.catalog.client.createView(prepareTable())
+
+ case false =>
+hiveContext.catalog.client.createView(prepareTable(sqlContext))
}
Seq.empty[Row]
}
- private def prepareTable(): HiveTable = {
-// setup column types according to the schema of child.
-val schema = if (tableDesc.schema == Nil) {
+ private def prepareTable(sqlContext: SQLContext): HiveTable = {
+val expandedText = if (sqlContext.conf.canonicalView) {
+ rebuildViewQueryString(sqlContext).getOrElse(expandViewText)
+} else {
+ expandViewText
+}
+tableDesc.copy(schema = viewSchema, viewText = Some(expandedText))
+ }
+
+ private def viewSchema: Seq[HiveColumn] = {
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49957696
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateViewAsSelect.scala
---
@@ -72,27 +85,40 @@ private[hive] case class CreateViewAsSelect(
HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
}
}
+ }
-val columnNames = childSchema.map(f => verbose(f.name))
-
+ private def expandViewText: String = {
--- End diff --
Renamed to `wrapViewTextWithSelect`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49957698
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -1356,68 +1356,84 @@ class SQLQuerySuite extends QueryTest with
SQLTestUtils with TestHiveSingleton {
}
}
- test("correctly handle CREATE OR REPLACE VIEW") {
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true") {
- withTable("jt", "jt2") {
-sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
-sql("CREATE OR REPLACE VIEW testView AS SELECT id FROM jt")
-checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+ Seq(true, false).foreach { enabled =>
--- End diff --
Good point. Added more test cases as suggested.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49957776
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -1356,68 +1356,84 @@ class SQLQuerySuite extends QueryTest with
SQLTestUtils with TestHiveSingleton {
}
}
- test("correctly handle CREATE OR REPLACE VIEW") {
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true") {
- withTable("jt", "jt2") {
-sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
-sql("CREATE OR REPLACE VIEW testView AS SELECT id FROM jt")
-checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+ Seq(true, false).foreach { enabled =>
+val prefix = (if (enabled) "With" else "Without") + " canonical native
view: "
+test(s"$prefix correctly handle CREATE OR REPLACE VIEW") {
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt", "jt2") {
+ sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+ sql("CREATE OR REPLACE VIEW testView AS SELECT id FROM jt")
+ checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+
+ val df = (1 until 10).map(i => i -> i).toDF("i", "j")
+ df.write.format("json").saveAsTable("jt2")
+ sql("CREATE OR REPLACE VIEW testView AS SELECT * FROM jt2")
+ // make sure the view has been changed.
+ checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
+
+ sql("DROP VIEW testView")
+
+ val e = intercept[AnalysisException] {
+sql("CREATE OR REPLACE VIEW IF NOT EXISTS testView AS SELECT
id FROM jt")
+ }
+ assert(e.message.contains("not allowed to define a view"))
+}
+ }
+}
-val df = (1 until 10).map(i => i -> i).toDF("i", "j")
-df.write.format("json").saveAsTable("jt2")
-sql("CREATE OR REPLACE VIEW testView AS SELECT * FROM jt2")
-// make sure the view has been changed.
-checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
+test(s"$prefix correctly handle ALTER VIEW") {
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt", "jt2") {
+ sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+ sql("CREATE VIEW testView AS SELECT id FROM jt")
-sql("DROP VIEW testView")
+ val df = (1 until 10).map(i => i -> i).toDF("i", "j")
+ df.write.format("json").saveAsTable("jt2")
+ sql("ALTER VIEW testView AS SELECT * FROM jt2")
+ // make sure the view has been changed.
+ checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
-val e = intercept[AnalysisException] {
- sql("CREATE OR REPLACE VIEW IF NOT EXISTS testView AS SELECT id
FROM jt")
+ sql("DROP VIEW testView")
}
-assert(e.message.contains("not allowed to define a view"))
}
}
- }
-
- test("correctly handle ALTER VIEW") {
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true") {
- withTable("jt", "jt2") {
-sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
-sql("CREATE VIEW testView AS SELECT id FROM jt")
-val df = (1 until 10).map(i => i -> i).toDF("i", "j")
-df.write.format("json").saveAsTable("jt2")
-sql("ALTER VIEW testView AS SELECT * FROM jt2")
-// make sure the view has been changed.
-checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
-
-sql("DROP VIEW testView")
+test(s"$prefix create hive view for json table") {
+ // json table is not hive-compatible, make sure the new flag fix it.
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt") {
+ sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+ sql("CREATE VIEW testView AS SELECT id FROM jt")
+ checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+ sql("DROP VIEW testView")
+}
}
}
- }
- test("create hive view for json table") {
-// json table is not hive-compatible, make sure the new flag fix it.
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true") {
-
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-172410088 **[Test build #49567 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49567/consoleFull)** for PR 10733 at commit [`c9e9c1b`](https://github.com/apache/spark/commit/c9e9c1b7e28142e3e2df4c613c8f0a64da607b3b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-172423544 **[Test build #49567 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49567/consoleFull)** for PR 10733 at commit [`c9e9c1b`](https://github.com/apache/spark/commit/c9e9c1b7e28142e3e2df4c613c8f0a64da607b3b). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-172424063 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49567/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-172424061 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-172482061 **[Test build #49589 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49589/consoleFull)** for PR 10733 at commit [`3c50fd6`](https://github.com/apache/spark/commit/3c50fd6c6dae0f9a2d09b54d00823b41a6c1b4df). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-172503936 **[Test build #49589 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49589/consoleFull)** for PR 10733 at commit [`3c50fd6`](https://github.com/apache/spark/commit/3c50fd6c6dae0f9a2d09b54d00823b41a6c1b4df). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-172504123 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-172504124 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49589/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175075600 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175081901 **[Test build #50108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50108/consoleFull)** for PR 10733 at commit [`3c50fd6`](https://github.com/apache/spark/commit/3c50fd6c6dae0f9a2d09b54d00823b41a6c1b4df). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175125170 **[Test build #50108 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50108/consoleFull)** for PR 10733 at commit [`3c50fd6`](https://github.com/apache/spark/commit/3c50fd6c6dae0f9a2d09b54d00823b41a6c1b4df). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175125589 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175125591 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50108/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r50879059
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateViewAsSelect.scala
---
@@ -44,55 +47,83 @@ private[hive] case class CreateViewAsSelect(
override def run(sqlContext: SQLContext): Seq[Row] = {
val hiveContext = sqlContext.asInstanceOf[HiveContext]
-if (hiveContext.catalog.tableExists(tableIdentifier)) {
- if (allowExisting) {
-// view already exists, will do nothing, to keep consistent with
Hive
- } else if (orReplace) {
-hiveContext.catalog.client.alertView(prepareTable())
- } else {
+hiveContext.catalog.tableExists(tableIdentifier) match {
+ case true if allowExisting =>
+// Handles `CREATE VIEW IF NOT EXISTS v0 AS SELECT ...`. Does
nothing when the target view
+// already exists.
+
+ case true if orReplace =>
+// Handles `CREATE OR REPLACE VIEW v0 AS SELECT ...`
+hiveContext.catalog.client.alertView(prepareTable(sqlContext))
+
+ case true =>
+// Handles `CREATE VIEW v0 AS SELECT ...`. Throws exception when
the target view already
+// exists.
throw new AnalysisException(s"View $tableIdentifier already
exists. " +
"If you want to update the view definition, please use ALTER
VIEW AS or " +
"CREATE OR REPLACE VIEW AS")
- }
-} else {
- hiveContext.catalog.client.createView(prepareTable())
+
+ case false =>
+hiveContext.catalog.client.createView(prepareTable(sqlContext))
}
Seq.empty[Row]
}
- private def prepareTable(): HiveTable = {
-// setup column types according to the schema of child.
-val schema = if (tableDesc.schema == Nil) {
- childSchema.map { attr =>
-HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
- }
+ private def prepareTable(sqlContext: SQLContext): HiveTable = {
+val expandedText = if (sqlContext.conf.canonicalView) {
+ rebuildViewQueryString(sqlContext).getOrElse(wrapViewTextWithSelect)
} else {
- childSchema.zip(tableDesc.schema).map { case (attr, col) =>
-HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
+ wrapViewTextWithSelect
+}
+
+val viewSchema = {
+ if (tableDesc.schema.isEmpty) {
+childSchema.map { attr =>
+ HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
+}
+ } else {
+childSchema.zip(tableDesc.schema).map { case (attr, col) =>
+ HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
+}
}
}
-val columnNames = childSchema.map(f => verbose(f.name))
+tableDesc.copy(schema = viewSchema, viewText = Some(expandedText))
+ }
+ private def wrapViewTextWithSelect: String = {
// When user specified column names for view, we should create a
project to do the renaming.
// When no column name specified, we still need to create a project to
declare the columns
// we need, to make us more robust to top level `*`s.
-val projectList = if (tableDesc.schema == Nil) {
- columnNames.mkString(", ")
-} else {
- columnNames.zip(tableDesc.schema.map(f => verbose(f.name))).map {
-case (name, alias) => s"$name AS $alias"
- }.mkString(", ")
+val viewOutput = {
+ val columnNames = childSchema.map(f => quote(f.name))
+ if (tableDesc.schema.isEmpty) {
+columnNames.mkString(", ")
+ } else {
+columnNames.zip(tableDesc.schema.map(f => quote(f.name))).map {
+ case (name, alias) => s"$name AS $alias"
+}.mkString(", ")
+ }
}
-val viewName = verbose(tableDesc.name)
-
-val expandedText = s"SELECT $projectList FROM
(${tableDesc.viewText.get}) $viewName"
+val viewText = tableDesc.viewText.get
+val viewName = quote(tableDesc.name)
+s"SELECT $viewOutput FROM ($viewText) $viewName"
+ }
-tableDesc.copy(schema = schema, viewText = Some(expandedText))
+ private def rebuildViewQueryString(sqlContext: SQLContext):
Option[String] = {
+val logicalPlan = if (tableDesc.schema.isEmpty) {
+ child
+} else {
+ val projectList = childSchema.zip(tableDesc.schema).map {
--- End diff --
Is it safe to call zip? We need to check the number of fields, right?
---
If your project is set up for it, you can reply to this email and have your
reply appea
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r50879131
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateViewAsSelect.scala
---
@@ -44,55 +47,83 @@ private[hive] case class CreateViewAsSelect(
override def run(sqlContext: SQLContext): Seq[Row] = {
val hiveContext = sqlContext.asInstanceOf[HiveContext]
-if (hiveContext.catalog.tableExists(tableIdentifier)) {
- if (allowExisting) {
-// view already exists, will do nothing, to keep consistent with
Hive
- } else if (orReplace) {
-hiveContext.catalog.client.alertView(prepareTable())
- } else {
+hiveContext.catalog.tableExists(tableIdentifier) match {
+ case true if allowExisting =>
+// Handles `CREATE VIEW IF NOT EXISTS v0 AS SELECT ...`. Does
nothing when the target view
+// already exists.
+
+ case true if orReplace =>
+// Handles `CREATE OR REPLACE VIEW v0 AS SELECT ...`
+hiveContext.catalog.client.alertView(prepareTable(sqlContext))
+
+ case true =>
+// Handles `CREATE VIEW v0 AS SELECT ...`. Throws exception when
the target view already
+// exists.
throw new AnalysisException(s"View $tableIdentifier already
exists. " +
"If you want to update the view definition, please use ALTER
VIEW AS or " +
"CREATE OR REPLACE VIEW AS")
- }
-} else {
- hiveContext.catalog.client.createView(prepareTable())
+
+ case false =>
+hiveContext.catalog.client.createView(prepareTable(sqlContext))
}
Seq.empty[Row]
}
- private def prepareTable(): HiveTable = {
-// setup column types according to the schema of child.
-val schema = if (tableDesc.schema == Nil) {
- childSchema.map { attr =>
-HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
- }
+ private def prepareTable(sqlContext: SQLContext): HiveTable = {
+val expandedText = if (sqlContext.conf.canonicalView) {
+ rebuildViewQueryString(sqlContext).getOrElse(wrapViewTextWithSelect)
} else {
- childSchema.zip(tableDesc.schema).map { case (attr, col) =>
-HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
+ wrapViewTextWithSelect
+}
+
+val viewSchema = {
+ if (tableDesc.schema.isEmpty) {
+childSchema.map { attr =>
+ HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
+}
+ } else {
+childSchema.zip(tableDesc.schema).map { case (attr, col) =>
+ HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
+}
}
}
-val columnNames = childSchema.map(f => verbose(f.name))
+tableDesc.copy(schema = viewSchema, viewText = Some(expandedText))
+ }
+ private def wrapViewTextWithSelect: String = {
// When user specified column names for view, we should create a
project to do the renaming.
// When no column name specified, we still need to create a project to
declare the columns
// we need, to make us more robust to top level `*`s.
-val projectList = if (tableDesc.schema == Nil) {
- columnNames.mkString(", ")
-} else {
- columnNames.zip(tableDesc.schema.map(f => verbose(f.name))).map {
-case (name, alias) => s"$name AS $alias"
- }.mkString(", ")
+val viewOutput = {
+ val columnNames = childSchema.map(f => quote(f.name))
+ if (tableDesc.schema.isEmpty) {
+columnNames.mkString(", ")
+ } else {
+columnNames.zip(tableDesc.schema.map(f => quote(f.name))).map {
+ case (name, alias) => s"$name AS $alias"
+}.mkString(", ")
+ }
}
-val viewName = verbose(tableDesc.name)
-
-val expandedText = s"SELECT $projectList FROM
(${tableDesc.viewText.get}) $viewName"
+val viewText = tableDesc.viewText.get
+val viewName = quote(tableDesc.name)
+s"SELECT $viewOutput FROM ($viewText) $viewName"
+ }
-tableDesc.copy(schema = schema, viewText = Some(expandedText))
+ private def rebuildViewQueryString(sqlContext: SQLContext):
Option[String] = {
+val logicalPlan = if (tableDesc.schema.isEmpty) {
+ child
+} else {
+ val projectList = childSchema.zip(tableDesc.schema).map {
--- End diff --
Let's also have a test for this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r50882509
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -1358,67 +1358,112 @@ class SQLQuerySuite extends QueryTest with
SQLTestUtils with TestHiveSingleton {
}
}
- test("correctly handle CREATE OR REPLACE VIEW") {
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true") {
- withTable("jt", "jt2") {
-sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
-sql("CREATE OR REPLACE VIEW testView AS SELECT id FROM jt")
-checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+ Seq(true, false).foreach { enabled =>
+val prefix = (if (enabled) "With" else "Without") + " canonical native
view: "
+test(s"$prefix correctly handle CREATE OR REPLACE VIEW") {
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt", "jt2") {
+ sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+ sql("CREATE OR REPLACE VIEW testView AS SELECT id FROM jt")
+ checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+
+ val df = (1 until 10).map(i => i -> i).toDF("i", "j")
+ df.write.format("json").saveAsTable("jt2")
+ sql("CREATE OR REPLACE VIEW testView AS SELECT * FROM jt2")
+ // make sure the view has been changed.
+ checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
+
+ sql("DROP VIEW testView")
+
+ val e = intercept[AnalysisException] {
+sql("CREATE OR REPLACE VIEW IF NOT EXISTS testView AS SELECT
id FROM jt")
+ }
+ assert(e.message.contains("not allowed to define a view"))
+}
+ }
+}
-val df = (1 until 10).map(i => i -> i).toDF("i", "j")
-df.write.format("json").saveAsTable("jt2")
-sql("CREATE OR REPLACE VIEW testView AS SELECT * FROM jt2")
-// make sure the view has been changed.
-checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
+test(s"$prefix correctly handle ALTER VIEW") {
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt", "jt2") {
+ withView("testView") {
+sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+sql("CREATE VIEW testView AS SELECT id FROM jt")
+
+val df = (1 until 10).map(i => i -> i).toDF("i", "j")
+df.write.format("json").saveAsTable("jt2")
+sql("ALTER VIEW testView AS SELECT * FROM jt2")
+// make sure the view has been changed.
+checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
+ }
+}
+ }
+}
-sql("DROP VIEW testView")
+test(s"$prefix create hive view for json table") {
+ // json table is not hive-compatible, make sure the new flag fix it.
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt") {
+ withView("testView") {
+sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+sql("CREATE VIEW testView AS SELECT id FROM jt")
+checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+ }
+}
+ }
+}
-val e = intercept[AnalysisException] {
- sql("CREATE OR REPLACE VIEW IF NOT EXISTS testView AS SELECT id
FROM jt")
+test(s"$prefix create hive view for partitioned parquet table") {
+ // partitioned parquet table is not hive-compatible, make sure the
new flag fix it.
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("parTable") {
+ withView("testView") {
+val df = Seq(1 -> "a").toDF("i", "j")
+
df.write.format("parquet").partitionBy("i").saveAsTable("parTable")
+sql("CREATE VIEW testView AS SELECT i, j FROM parTable")
+checkAnswer(sql("SELECT * FROM testView"), Row(1, "a"))
+ }
}
-assert(e.message.contains("not allowed to define a view"))
}
}
}
- test("correctly handle ALTER VIEW") {
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true") {
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175181529 Overall looks good. Left two comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/10733#discussion_r50886520 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala --- @@ -155,6 +155,17 @@ private[sql] trait SQLTestUtils } /** + * Drops table `tableName` after calling `f`. --- End diff -- `Drops view vewName...`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r50903539
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateViewAsSelect.scala
---
@@ -44,55 +47,83 @@ private[hive] case class CreateViewAsSelect(
override def run(sqlContext: SQLContext): Seq[Row] = {
val hiveContext = sqlContext.asInstanceOf[HiveContext]
-if (hiveContext.catalog.tableExists(tableIdentifier)) {
- if (allowExisting) {
-// view already exists, will do nothing, to keep consistent with
Hive
- } else if (orReplace) {
-hiveContext.catalog.client.alertView(prepareTable())
- } else {
+hiveContext.catalog.tableExists(tableIdentifier) match {
+ case true if allowExisting =>
+// Handles `CREATE VIEW IF NOT EXISTS v0 AS SELECT ...`. Does
nothing when the target view
+// already exists.
+
+ case true if orReplace =>
+// Handles `CREATE OR REPLACE VIEW v0 AS SELECT ...`
+hiveContext.catalog.client.alertView(prepareTable(sqlContext))
+
+ case true =>
+// Handles `CREATE VIEW v0 AS SELECT ...`. Throws exception when
the target view already
+// exists.
throw new AnalysisException(s"View $tableIdentifier already
exists. " +
"If you want to update the view definition, please use ALTER
VIEW AS or " +
"CREATE OR REPLACE VIEW AS")
- }
-} else {
- hiveContext.catalog.client.createView(prepareTable())
+
+ case false =>
+hiveContext.catalog.client.createView(prepareTable(sqlContext))
}
Seq.empty[Row]
}
- private def prepareTable(): HiveTable = {
-// setup column types according to the schema of child.
-val schema = if (tableDesc.schema == Nil) {
- childSchema.map { attr =>
-HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
- }
+ private def prepareTable(sqlContext: SQLContext): HiveTable = {
+val expandedText = if (sqlContext.conf.canonicalView) {
+ rebuildViewQueryString(sqlContext).getOrElse(wrapViewTextWithSelect)
} else {
- childSchema.zip(tableDesc.schema).map { case (attr, col) =>
-HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
+ wrapViewTextWithSelect
+}
+
+val viewSchema = {
+ if (tableDesc.schema.isEmpty) {
+childSchema.map { attr =>
+ HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
+}
+ } else {
+childSchema.zip(tableDesc.schema).map { case (attr, col) =>
+ HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
+}
}
}
-val columnNames = childSchema.map(f => verbose(f.name))
+tableDesc.copy(schema = viewSchema, viewText = Some(expandedText))
+ }
+ private def wrapViewTextWithSelect: String = {
// When user specified column names for view, we should create a
project to do the renaming.
// When no column name specified, we still need to create a project to
declare the columns
// we need, to make us more robust to top level `*`s.
-val projectList = if (tableDesc.schema == Nil) {
- columnNames.mkString(", ")
-} else {
- columnNames.zip(tableDesc.schema.map(f => verbose(f.name))).map {
-case (name, alias) => s"$name AS $alias"
- }.mkString(", ")
+val viewOutput = {
+ val columnNames = childSchema.map(f => quote(f.name))
+ if (tableDesc.schema.isEmpty) {
+columnNames.mkString(", ")
+ } else {
+columnNames.zip(tableDesc.schema.map(f => quote(f.name))).map {
+ case (name, alias) => s"$name AS $alias"
+}.mkString(", ")
+ }
}
-val viewName = verbose(tableDesc.name)
-
-val expandedText = s"SELECT $projectList FROM
(${tableDesc.viewText.get}) $viewName"
+val viewText = tableDesc.viewText.get
+val viewName = quote(tableDesc.name)
+s"SELECT $viewOutput FROM ($viewText) $viewName"
+ }
-tableDesc.copy(schema = schema, viewText = Some(expandedText))
+ private def rebuildViewQueryString(sqlContext: SQLContext):
Option[String] = {
+val logicalPlan = if (tableDesc.schema.isEmpty) {
+ child
+} else {
+ val projectList = childSchema.zip(tableDesc.schema).map {
--- End diff --
It's an invariant condition of `CreateViewAsSelect` that these two have the
same size. Users can't construct a case that violates this condition, thus
adding
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/10733#discussion_r50903806 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala --- @@ -155,6 +155,17 @@ private[sql] trait SQLTestUtils } /** + * Drops table `tableName` after calling `f`. --- End diff -- Oh, nice catch, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r50904655
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -1358,67 +1358,112 @@ class SQLQuerySuite extends QueryTest with
SQLTestUtils with TestHiveSingleton {
}
}
- test("correctly handle CREATE OR REPLACE VIEW") {
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true") {
- withTable("jt", "jt2") {
-sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
-sql("CREATE OR REPLACE VIEW testView AS SELECT id FROM jt")
-checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+ Seq(true, false).foreach { enabled =>
+val prefix = (if (enabled) "With" else "Without") + " canonical native
view: "
+test(s"$prefix correctly handle CREATE OR REPLACE VIEW") {
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt", "jt2") {
+ sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+ sql("CREATE OR REPLACE VIEW testView AS SELECT id FROM jt")
+ checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+
+ val df = (1 until 10).map(i => i -> i).toDF("i", "j")
+ df.write.format("json").saveAsTable("jt2")
+ sql("CREATE OR REPLACE VIEW testView AS SELECT * FROM jt2")
+ // make sure the view has been changed.
+ checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
+
+ sql("DROP VIEW testView")
+
+ val e = intercept[AnalysisException] {
+sql("CREATE OR REPLACE VIEW IF NOT EXISTS testView AS SELECT
id FROM jt")
+ }
+ assert(e.message.contains("not allowed to define a view"))
+}
+ }
+}
-val df = (1 until 10).map(i => i -> i).toDF("i", "j")
-df.write.format("json").saveAsTable("jt2")
-sql("CREATE OR REPLACE VIEW testView AS SELECT * FROM jt2")
-// make sure the view has been changed.
-checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
+test(s"$prefix correctly handle ALTER VIEW") {
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt", "jt2") {
+ withView("testView") {
+sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+sql("CREATE VIEW testView AS SELECT id FROM jt")
+
+val df = (1 until 10).map(i => i -> i).toDF("i", "j")
+df.write.format("json").saveAsTable("jt2")
+sql("ALTER VIEW testView AS SELECT * FROM jt2")
+// make sure the view has been changed.
+checkAnswer(sql("SELECT * FROM testView ORDER BY i"), (1 to
9).map(i => Row(i, i)))
+ }
+}
+ }
+}
-sql("DROP VIEW testView")
+test(s"$prefix create hive view for json table") {
+ // json table is not hive-compatible, make sure the new flag fix it.
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("jt") {
+ withView("testView") {
+sqlContext.range(1, 10).write.format("json").saveAsTable("jt")
+sql("CREATE VIEW testView AS SELECT id FROM jt")
+checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to
9).map(i => Row(i)))
+ }
+}
+ }
+}
-val e = intercept[AnalysisException] {
- sql("CREATE OR REPLACE VIEW IF NOT EXISTS testView AS SELECT id
FROM jt")
+test(s"$prefix create hive view for partitioned parquet table") {
+ // partitioned parquet table is not hive-compatible, make sure the
new flag fix it.
+ withSQLConf(
+SQLConf.NATIVE_VIEW.key -> "true",
SQLConf.CANONICAL_NATIVE_VIEW.key -> enabled.toString) {
+withTable("parTable") {
+ withView("testView") {
+val df = Seq(1 -> "a").toDF("i", "j")
+
df.write.format("parquet").partitionBy("i").saveAsTable("parTable")
+sql("CREATE VIEW testView AS SELECT i, j FROM parTable")
+checkAnswer(sql("SELECT * FROM testView"), Row(1, "a"))
+ }
}
-assert(e.message.contains("not allowed to define a view"))
}
}
}
- test("correctly handle ALTER VIEW") {
-withSQLConf(SQLConf.NATIVE_VIEW.key -> "true"
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175249728 **[Test build #50125 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50125/consoleFull)** for PR 10733 at commit [`51b9db2`](https://github.com/apache/spark/commit/51b9db230a2b9f13e13fe8c75f9f9057151028ea). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175250077 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175250082 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50125/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175250066 **[Test build #50125 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50125/consoleFull)** for PR 10733 at commit [`51b9db2`](https://github.com/apache/spark/commit/51b9db230a2b9f13e13fe8c75f9f9057151028ea). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175258865 **[Test build #50128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50128/consoleFull)** for PR 10733 at commit [`737a7d0`](https://github.com/apache/spark/commit/737a7d0765aa20017822c436bd4d22d58b0bd28c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175292737 **[Test build #50128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50128/consoleFull)** for PR 10733 at commit [`737a7d0`](https://github.com/apache/spark/commit/737a7d0765aa20017822c436bd4d22d58b0bd28c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175292928 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50128/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175292923 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175306181 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175310513 **[Test build #50140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50140/consoleFull)** for PR 10733 at commit [`737a7d0`](https://github.com/apache/spark/commit/737a7d0765aa20017822c436bd4d22d58b0bd28c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175343151 **[Test build #50140 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50140/consoleFull)** for PR 10733 at commit [`737a7d0`](https://github.com/apache/spark/commit/737a7d0765aa20017822c436bd4d22d58b0bd28c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r50932890
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateViewAsSelect.scala
---
@@ -44,55 +48,96 @@ private[hive] case class CreateViewAsSelect(
override def run(sqlContext: SQLContext): Seq[Row] = {
val hiveContext = sqlContext.asInstanceOf[HiveContext]
-if (hiveContext.catalog.tableExists(tableIdentifier)) {
- if (allowExisting) {
-// view already exists, will do nothing, to keep consistent with
Hive
- } else if (orReplace) {
-hiveContext.catalog.client.alertView(prepareTable())
- } else {
+hiveContext.catalog.tableExists(tableIdentifier) match {
+ case true if allowExisting =>
+// Handles `CREATE VIEW IF NOT EXISTS v0 AS SELECT ...`. Does
nothing when the target view
+// already exists.
+
+ case true if orReplace =>
+// Handles `CREATE OR REPLACE VIEW v0 AS SELECT ...`
+hiveContext.catalog.client.alertView(prepareTable(sqlContext))
+
+ case true =>
+// Handles `CREATE VIEW v0 AS SELECT ...`. Throws exception when
the target view already
+// exists.
throw new AnalysisException(s"View $tableIdentifier already
exists. " +
"If you want to update the view definition, please use ALTER
VIEW AS or " +
"CREATE OR REPLACE VIEW AS")
- }
-} else {
- hiveContext.catalog.client.createView(prepareTable())
+
+ case false =>
+hiveContext.catalog.client.createView(prepareTable(sqlContext))
}
Seq.empty[Row]
}
- private def prepareTable(): HiveTable = {
-// setup column types according to the schema of child.
-val schema = if (tableDesc.schema == Nil) {
- childSchema.map { attr =>
-HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
- }
+ private def prepareTable(sqlContext: SQLContext): HiveTable = {
+val expandedText = if (sqlContext.conf.canonicalView) {
+ rebuildViewQueryString(sqlContext).getOrElse(wrapViewTextWithSelect)
} else {
- childSchema.zip(tableDesc.schema).map { case (attr, col) =>
-HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
+ wrapViewTextWithSelect
+}
+
+val viewSchema = {
+ if (tableDesc.schema.isEmpty) {
+childSchema.map { attr =>
+ HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
+}
+ } else {
+childSchema.zip(tableDesc.schema).map { case (attr, col) =>
--- End diff --
Sorry. I missed this. How about we also have a check for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r50932896
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateViewAsSelect.scala
---
@@ -44,55 +48,96 @@ private[hive] case class CreateViewAsSelect(
override def run(sqlContext: SQLContext): Seq[Row] = {
val hiveContext = sqlContext.asInstanceOf[HiveContext]
-if (hiveContext.catalog.tableExists(tableIdentifier)) {
- if (allowExisting) {
-// view already exists, will do nothing, to keep consistent with
Hive
- } else if (orReplace) {
-hiveContext.catalog.client.alertView(prepareTable())
- } else {
+hiveContext.catalog.tableExists(tableIdentifier) match {
+ case true if allowExisting =>
+// Handles `CREATE VIEW IF NOT EXISTS v0 AS SELECT ...`. Does
nothing when the target view
+// already exists.
+
+ case true if orReplace =>
+// Handles `CREATE OR REPLACE VIEW v0 AS SELECT ...`
+hiveContext.catalog.client.alertView(prepareTable(sqlContext))
+
+ case true =>
+// Handles `CREATE VIEW v0 AS SELECT ...`. Throws exception when
the target view already
+// exists.
throw new AnalysisException(s"View $tableIdentifier already
exists. " +
"If you want to update the view definition, please use ALTER
VIEW AS or " +
"CREATE OR REPLACE VIEW AS")
- }
-} else {
- hiveContext.catalog.client.createView(prepareTable())
+
+ case false =>
+hiveContext.catalog.client.createView(prepareTable(sqlContext))
}
Seq.empty[Row]
}
- private def prepareTable(): HiveTable = {
-// setup column types according to the schema of child.
-val schema = if (tableDesc.schema == Nil) {
- childSchema.map { attr =>
-HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
- }
+ private def prepareTable(sqlContext: SQLContext): HiveTable = {
+val expandedText = if (sqlContext.conf.canonicalView) {
+ rebuildViewQueryString(sqlContext).getOrElse(wrapViewTextWithSelect)
} else {
- childSchema.zip(tableDesc.schema).map { case (attr, col) =>
-HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
+ wrapViewTextWithSelect
+}
+
+val viewSchema = {
+ if (tableDesc.schema.isEmpty) {
+childSchema.map { attr =>
+ HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
+}
+ } else {
+childSchema.zip(tableDesc.schema).map { case (attr, col) =>
+ HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
+}
}
}
-val columnNames = childSchema.map(f => verbose(f.name))
+tableDesc.copy(schema = viewSchema, viewText = Some(expandedText))
+ }
+ private def wrapViewTextWithSelect: String = {
// When user specified column names for view, we should create a
project to do the renaming.
// When no column name specified, we still need to create a project to
declare the columns
// we need, to make us more robust to top level `*`s.
-val projectList = if (tableDesc.schema == Nil) {
- columnNames.mkString(", ")
-} else {
- columnNames.zip(tableDesc.schema.map(f => verbose(f.name))).map {
-case (name, alias) => s"$name AS $alias"
- }.mkString(", ")
+val viewOutput = {
+ val columnNames = childSchema.map(f => quote(f.name))
+ if (tableDesc.schema.isEmpty) {
+columnNames.mkString(", ")
+ } else {
+columnNames.zip(tableDesc.schema.map(f => quote(f.name))).map {
--- End diff --
and this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r50932979
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateViewAsSelect.scala
---
@@ -44,55 +48,96 @@ private[hive] case class CreateViewAsSelect(
override def run(sqlContext: SQLContext): Seq[Row] = {
val hiveContext = sqlContext.asInstanceOf[HiveContext]
-if (hiveContext.catalog.tableExists(tableIdentifier)) {
- if (allowExisting) {
-// view already exists, will do nothing, to keep consistent with
Hive
- } else if (orReplace) {
-hiveContext.catalog.client.alertView(prepareTable())
- } else {
+hiveContext.catalog.tableExists(tableIdentifier) match {
+ case true if allowExisting =>
+// Handles `CREATE VIEW IF NOT EXISTS v0 AS SELECT ...`. Does
nothing when the target view
+// already exists.
+
+ case true if orReplace =>
+// Handles `CREATE OR REPLACE VIEW v0 AS SELECT ...`
+hiveContext.catalog.client.alertView(prepareTable(sqlContext))
+
+ case true =>
+// Handles `CREATE VIEW v0 AS SELECT ...`. Throws exception when
the target view already
+// exists.
throw new AnalysisException(s"View $tableIdentifier already
exists. " +
"If you want to update the view definition, please use ALTER
VIEW AS or " +
"CREATE OR REPLACE VIEW AS")
- }
-} else {
- hiveContext.catalog.client.createView(prepareTable())
+
+ case false =>
+hiveContext.catalog.client.createView(prepareTable(sqlContext))
}
Seq.empty[Row]
}
- private def prepareTable(): HiveTable = {
-// setup column types according to the schema of child.
-val schema = if (tableDesc.schema == Nil) {
- childSchema.map { attr =>
-HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
- }
+ private def prepareTable(sqlContext: SQLContext): HiveTable = {
+val expandedText = if (sqlContext.conf.canonicalView) {
+ rebuildViewQueryString(sqlContext).getOrElse(wrapViewTextWithSelect)
} else {
- childSchema.zip(tableDesc.schema).map { case (attr, col) =>
-HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
+ wrapViewTextWithSelect
+}
+
+val viewSchema = {
+ if (tableDesc.schema.isEmpty) {
+childSchema.map { attr =>
+ HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
+}
+ } else {
+childSchema.zip(tableDesc.schema).map { case (attr, col) =>
+ HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
+}
}
}
-val columnNames = childSchema.map(f => verbose(f.name))
+tableDesc.copy(schema = viewSchema, viewText = Some(expandedText))
+ }
+ private def wrapViewTextWithSelect: String = {
// When user specified column names for view, we should create a
project to do the renaming.
// When no column name specified, we still need to create a project to
declare the columns
// we need, to make us more robust to top level `*`s.
-val projectList = if (tableDesc.schema == Nil) {
- columnNames.mkString(", ")
-} else {
- columnNames.zip(tableDesc.schema.map(f => verbose(f.name))).map {
-case (name, alias) => s"$name AS $alias"
- }.mkString(", ")
+val viewOutput = {
+ val columnNames = childSchema.map(f => quote(f.name))
+ if (tableDesc.schema.isEmpty) {
+columnNames.mkString(", ")
+ } else {
+columnNames.zip(tableDesc.schema.map(f => quote(f.name))).map {
+ case (name, alias) => s"$name AS $alias"
+}.mkString(", ")
+ }
}
-val viewName = verbose(tableDesc.name)
+val viewText = tableDesc.viewText.get
+val viewName = quote(tableDesc.name)
+s"SELECT $viewOutput FROM ($viewText) $viewName"
+ }
-val expandedText = s"SELECT $projectList FROM
(${tableDesc.viewText.get}) $viewName"
+ private def rebuildViewQueryString(sqlContext: SQLContext):
Option[String] = {
+val logicalPlan = if (tableDesc.schema.isEmpty) {
+ child
+} else {
+ if (childSchema.length != tableDesc.schema.length) {
--- End diff --
sorry Actually we do not need this check. We have already done it at
https://github.com/apache/spark/pull/10733/files#diff-074b1d8480e0d0d7c212bc4461f3d4acR43
---
If your project is set up for it, you can reply to this email and have y
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175343371 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50140/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r50932944
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateViewAsSelect.scala
---
@@ -44,55 +48,96 @@ private[hive] case class CreateViewAsSelect(
override def run(sqlContext: SQLContext): Seq[Row] = {
val hiveContext = sqlContext.asInstanceOf[HiveContext]
-if (hiveContext.catalog.tableExists(tableIdentifier)) {
- if (allowExisting) {
-// view already exists, will do nothing, to keep consistent with
Hive
- } else if (orReplace) {
-hiveContext.catalog.client.alertView(prepareTable())
- } else {
+hiveContext.catalog.tableExists(tableIdentifier) match {
+ case true if allowExisting =>
+// Handles `CREATE VIEW IF NOT EXISTS v0 AS SELECT ...`. Does
nothing when the target view
+// already exists.
+
+ case true if orReplace =>
+// Handles `CREATE OR REPLACE VIEW v0 AS SELECT ...`
+hiveContext.catalog.client.alertView(prepareTable(sqlContext))
+
+ case true =>
+// Handles `CREATE VIEW v0 AS SELECT ...`. Throws exception when
the target view already
+// exists.
throw new AnalysisException(s"View $tableIdentifier already
exists. " +
"If you want to update the view definition, please use ALTER
VIEW AS or " +
"CREATE OR REPLACE VIEW AS")
- }
-} else {
- hiveContext.catalog.client.createView(prepareTable())
+
+ case false =>
+hiveContext.catalog.client.createView(prepareTable(sqlContext))
}
Seq.empty[Row]
}
- private def prepareTable(): HiveTable = {
-// setup column types according to the schema of child.
-val schema = if (tableDesc.schema == Nil) {
- childSchema.map { attr =>
-HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
- }
+ private def prepareTable(sqlContext: SQLContext): HiveTable = {
+val expandedText = if (sqlContext.conf.canonicalView) {
+ rebuildViewQueryString(sqlContext).getOrElse(wrapViewTextWithSelect)
} else {
- childSchema.zip(tableDesc.schema).map { case (attr, col) =>
-HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
+ wrapViewTextWithSelect
+}
+
+val viewSchema = {
+ if (tableDesc.schema.isEmpty) {
+childSchema.map { attr =>
+ HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
+}
+ } else {
+childSchema.zip(tableDesc.schema).map { case (attr, col) =>
--- End diff --
I just realized that we do the check at
https://github.com/apache/spark/pull/10733/files#diff-074b1d8480e0d0d7c212bc4461f3d4acR43
(`assert(tableDesc.schema == Nil || tableDesc.schema.length ==
childSchema.length)`)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175343370 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r50933587
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateViewAsSelect.scala
---
@@ -44,55 +47,83 @@ private[hive] case class CreateViewAsSelect(
override def run(sqlContext: SQLContext): Seq[Row] = {
val hiveContext = sqlContext.asInstanceOf[HiveContext]
-if (hiveContext.catalog.tableExists(tableIdentifier)) {
- if (allowExisting) {
-// view already exists, will do nothing, to keep consistent with
Hive
- } else if (orReplace) {
-hiveContext.catalog.client.alertView(prepareTable())
- } else {
+hiveContext.catalog.tableExists(tableIdentifier) match {
+ case true if allowExisting =>
+// Handles `CREATE VIEW IF NOT EXISTS v0 AS SELECT ...`. Does
nothing when the target view
+// already exists.
+
+ case true if orReplace =>
+// Handles `CREATE OR REPLACE VIEW v0 AS SELECT ...`
+hiveContext.catalog.client.alertView(prepareTable(sqlContext))
+
+ case true =>
+// Handles `CREATE VIEW v0 AS SELECT ...`. Throws exception when
the target view already
+// exists.
throw new AnalysisException(s"View $tableIdentifier already
exists. " +
"If you want to update the view definition, please use ALTER
VIEW AS or " +
"CREATE OR REPLACE VIEW AS")
- }
-} else {
- hiveContext.catalog.client.createView(prepareTable())
+
+ case false =>
+hiveContext.catalog.client.createView(prepareTable(sqlContext))
}
Seq.empty[Row]
}
- private def prepareTable(): HiveTable = {
-// setup column types according to the schema of child.
-val schema = if (tableDesc.schema == Nil) {
- childSchema.map { attr =>
-HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
- }
+ private def prepareTable(sqlContext: SQLContext): HiveTable = {
+val expandedText = if (sqlContext.conf.canonicalView) {
+ rebuildViewQueryString(sqlContext).getOrElse(wrapViewTextWithSelect)
} else {
- childSchema.zip(tableDesc.schema).map { case (attr, col) =>
-HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
+ wrapViewTextWithSelect
+}
+
+val viewSchema = {
+ if (tableDesc.schema.isEmpty) {
+childSchema.map { attr =>
+ HiveColumn(attr.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), null)
+}
+ } else {
+childSchema.zip(tableDesc.schema).map { case (attr, col) =>
+ HiveColumn(col.name,
HiveMetastoreTypes.toMetastoreType(attr.dataType), col.comment)
+}
}
}
-val columnNames = childSchema.map(f => verbose(f.name))
+tableDesc.copy(schema = viewSchema, viewText = Some(expandedText))
+ }
+ private def wrapViewTextWithSelect: String = {
// When user specified column names for view, we should create a
project to do the renaming.
// When no column name specified, we still need to create a project to
declare the columns
// we need, to make us more robust to top level `*`s.
-val projectList = if (tableDesc.schema == Nil) {
- columnNames.mkString(", ")
-} else {
- columnNames.zip(tableDesc.schema.map(f => verbose(f.name))).map {
-case (name, alias) => s"$name AS $alias"
- }.mkString(", ")
+val viewOutput = {
+ val columnNames = childSchema.map(f => quote(f.name))
+ if (tableDesc.schema.isEmpty) {
+columnNames.mkString(", ")
+ } else {
+columnNames.zip(tableDesc.schema.map(f => quote(f.name))).map {
+ case (name, alias) => s"$name AS $alias"
+}.mkString(", ")
+ }
}
-val viewName = verbose(tableDesc.name)
-
-val expandedText = s"SELECT $projectList FROM
(${tableDesc.viewText.get}) $viewName"
+val viewText = tableDesc.viewText.get
+val viewName = quote(tableDesc.name)
+s"SELECT $viewOutput FROM ($viewText) $viewName"
+ }
-tableDesc.copy(schema = schema, viewText = Some(expandedText))
+ private def rebuildViewQueryString(sqlContext: SQLContext):
Option[String] = {
+val logicalPlan = if (tableDesc.schema.isEmpty) {
+ child
+} else {
+ val projectList = childSchema.zip(tableDesc.schema).map {
--- End diff --
nvm, we already have the test at
https://github.com/apache/spark/pull/10733/files#diff-074b1d8480e0d0d7c212bc4461f3d4acR43.
---
If your project is set up for it
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175354026 **[Test build #50162 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50162/consoleFull)** for PR 10733 at commit [`0ce28d9`](https://github.com/apache/spark/commit/0ce28d99ca4f3eb0c93153bf91d43f81fe217042). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175380158 **[Test build #50162 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50162/consoleFull)** for PR 10733 at commit [`0ce28d9`](https://github.com/apache/spark/commit/0ce28d99ca4f3eb0c93153bf91d43f81fe217042). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175380828 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50162/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175380825 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-175383478 LGTM. Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10733 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
GitHub user liancheng opened a pull request: https://github.com/apache/spark/pull/10733 [SPARK-12728][SQL] Integrates SQL generation with native view This PR is a follow-up of PR #10541. It integrates the newly introduced SQL generation feature with native view to make native view canonical. In this PR, a new SQL option `spark.sql.nativeView.canonical` is added. When this option and `spark.sql.nativeView` are both `true`, Spark SQL tries to handle `CREATE VIEW` DDL statements using SQL query strings generated from view definition logical plans. If we failed to map the plan to SQL, we fallback to the original native view approach (i.e., wrap the original view definition SQL text with an extra `SELECT` clause. You can merge this pull request into a Git repository by running: $ git pull https://github.com/liancheng/spark spark-12728.integrate-sql-gen-with-native-view Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10733.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10733 commit 18f9a4d32f2040663831c218881f487235e6c18e Author: Cheng Lian Date: 2016-01-13T02:12:54Z Integrates SQL generation with native view --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171135747 cc @cloud-fan @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171139222 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49285/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171139220 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171140318 **[Test build #49287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49287/consoleFull)** for PR 10733 at commit [`70c19ce`](https://github.com/apache/spark/commit/70c19cec32da97f73620008b13dc02c0ff08a548). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171140480 **[Test build #49287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49287/consoleFull)** for PR 10733 at commit [`70c19ce`](https://github.com/apache/spark/commit/70c19cec32da97f73620008b13dc02c0ff08a548). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171140483 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49287/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171140482 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49546205
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala ---
@@ -372,6 +372,13 @@ private[spark] object SQLConf {
"possible, or you may get wrong result.",
isPublic = false)
+ val CANONICAL_NATIVE_VIEW = booleanConf("spark.sql.nativeView.canonical",
+defaultValue = Some(false),
+doc = "When this option and spark.sql.nativeView are both true, Spark
SQL tries to handle " +
+ "CREATE VIEW statements using SQL query strings generated from
view definition logical " +
+ "plans.",
--- End diff --
also explain the fallback behaviour?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/10733#discussion_r49546177
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala ---
@@ -372,6 +372,13 @@ private[spark] object SQLConf {
"possible, or you may get wrong result.",
isPublic = false)
+ val CANONICAL_NATIVE_VIEW = booleanConf("spark.sql.nativeView.canonical",
+defaultValue = Some(false),
+doc = "When this option and spark.sql.nativeView are both true, Spark
SQL tries to handle " +
+ "CREATE VIEW statements using SQL query strings generated from
view definition logical " +
+ "plans.",
--- End diff --
no `s`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171140821 Do we have a test that fails with the current native view but works after your PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171261693 @cloud-fan "CTE within view" is such a test case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171383565 **[Test build #49320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49320/consoleFull)** for PR 10733 at commit [`d042c3f`](https://github.com/apache/spark/commit/d042c3fd08853295540440fd4f0a06292462f704). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171383913 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171383895 **[Test build #49320 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49320/consoleFull)** for PR 10733 at commit [`d042c3f`](https://github.com/apache/spark/commit/d042c3fd08853295540440fd4f0a06292462f704). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171383916 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49320/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171392067 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171392072 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49321/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171393561 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171395797 **[Test build #49323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49323/consoleFull)** for PR 10733 at commit [`0d9d55b`](https://github.com/apache/spark/commit/0d9d55b46fd7ac97196f30bf82e358350c61f11a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171424728 **[Test build #49323 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49323/consoleFull)** for PR 10733 at commit [`0d9d55b`](https://github.com/apache/spark/commit/0d9d55b46fd7ac97196f30bf82e358350c61f11a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171425153 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49323/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171425149 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171428307 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12728][SQL] Integrates SQL generation w...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/10733#issuecomment-171429313 Thanks @liancheng. I'd like to also review it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
