[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74562712 [Test build #27566 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27566/consoleFull) for PR 4617 at commit [`ac31b3c`](https://github.com/apache/spark/commit/ac31b3c21b78aaa887d66c26e215b74981ec3993). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74571613 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27566/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74582760 [Test build #27591 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27591/consoleFull) for PR 4617 at commit [`8e3019d`](https://github.com/apache/spark/commit/8e3019df8d27815746825704d773ff39ed7a9a83). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74586231 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27579/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74577482 [Test build #27579 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27579/consoleFull) for PR 4617 at commit [`e76e85a`](https://github.com/apache/spark/commit/e76e85aa1822285173c17e8f0caa53c902e503a7). * This patch **does not merge cleanly**. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74586222 [Test build #27579 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27579/consoleFull) for PR 4617 at commit [`e76e85a`](https://github.com/apache/spark/commit/e76e85aa1822285173c17e8f0caa53c902e503a7). * This patch **passes all tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/4617#discussion_r24776541 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/rules.scala --- @@ -74,3 +74,68 @@ private[sql] object PreInsertCastAndRename extends Rule[LogicalPlan] { } } } + +/** + * A rule to do various checks before inserting into or writing to a data source table. + */ +private[sql] case class PreWriteCheck(catalog: Catalog) extends Rule[LogicalPlan] { + def failAnalysis(msg: String) = { throw new AnalysisException(msg) } + + def apply(plan: LogicalPlan): LogicalPlan = { +plan.foreach { + case i @ logical.InsertIntoTable( +l @ LogicalRelation(t: InsertableRelation), partition, query, overwrite) = +// Right now, we do not support insert into a data source table with partition specs. +if (partition.nonEmpty) { + failAnalysis(sInsert into a partition is not allowed because $l is not partitioned.) +} else { + // Get all input data source relations of the query. + val srcRelations = query.collect { +case LogicalRelation(src: BaseRelation) = src + } + if (srcRelations.exists(src = src == t)) { +failAnalysis( + sIt is not allowed to insert overwrite the table because it is used as + + san input table.) + } else { +// OK + } +} + + case i @ logical.InsertIntoTable(l: LogicalRelation, partition, query, overwrite) = --- End diff -- Make the ! instanceOf here explicit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/4617#discussion_r24776531 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/rules.scala --- @@ -74,3 +74,68 @@ private[sql] object PreInsertCastAndRename extends Rule[LogicalPlan] { } } } + +/** + * A rule to do various checks before inserting into or writing to a data source table. + */ +private[sql] case class PreWriteCheck(catalog: Catalog) extends Rule[LogicalPlan] { + def failAnalysis(msg: String) = { throw new AnalysisException(msg) } + + def apply(plan: LogicalPlan): LogicalPlan = { +plan.foreach { + case i @ logical.InsertIntoTable( +l @ LogicalRelation(t: InsertableRelation), partition, query, overwrite) = +// Right now, we do not support insert into a data source table with partition specs. +if (partition.nonEmpty) { + failAnalysis(sInsert into a partition is not allowed because $l is not partitioned.) +} else { + // Get all input data source relations of the query. + val srcRelations = query.collect { +case LogicalRelation(src: BaseRelation) = src + } + if (srcRelations.exists(src = src == t)) { +failAnalysis( + sIt is not allowed to insert overwrite the table because it is used as + + san input table.) --- End diff -- `Cannot insert overwrite into table that is also being read from` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74580589 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27585/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74580482 [Test build #27585 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27585/consoleFull) for PR 4617 at commit [`499e8e7`](https://github.com/apache/spark/commit/499e8e740c2e826be275b151f1d207ae8edc3a98). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4617 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74592303 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27591/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74592299 [Test build #27591 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27591/consoleFull) for PR 4617 at commit [`8e3019d`](https://github.com/apache/spark/commit/8e3019df8d27815746825704d773ff39ed7a9a83). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74571603 [Test build #27566 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27566/consoleFull) for PR 4617 at commit [`ac31b3c`](https://github.com/apache/spark/commit/ac31b3c21b78aaa887d66c26e215b74981ec3993). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74580586 [Test build #27585 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27585/consoleFull) for PR 4617 at commit [`499e8e7`](https://github.com/apache/spark/commit/499e8e740c2e826be275b151f1d207ae8edc3a98). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class AnalysisException protected[sql] (` * ` implicit class AnalysisErrorAt(t: TreeNode[_]) ` * `case class Origin(` * `case class IsNull(attribute: String) extends Filter` * `case class IsNotNull(attribute: String) extends Filter` * `case class And(left: Filter, right: Filter) extends Filter` * `case class Or(left: Filter, right: Filter) extends Filter` * `case class Not(child: Filter) extends Filter` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74446691 [Test build #27534 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27534/consoleFull) for PR 4617 at commit [`99da57e`](https://github.com/apache/spark/commit/99da57ea60b5a67978423631e81bba38373a6e06). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/4617 [SPARK-5746][SQL] Check invalid cases for the write path of data source API JIRA: https://issues.apache.org/jira/browse/SPARK-5746 @liancheng @marmbrus You can merge this pull request into a Git repository by running: $ git pull https://github.com/yhuai/spark insertOverwrite Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4617.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4617 commit a88c5168901dc914ee9dee79bce5261b49a7e35e Author: Yin Huai yh...@databricks.com Date: 2015-02-13T22:57:23Z DDLParser will take a parsering function to take care CTAS statements. commit 6b7545c4abc4455df07b4508cb192718866c8e21 Author: Yin Huai yh...@databricks.com Date: 2015-02-16T00:21:23Z Add a pre write check to the data source API. commit 99da57ea60b5a67978423631e81bba38373a6e06 Author: Yin Huai yh...@databricks.com Date: 2015-02-16T00:37:12Z Merge remote-tracking branch 'upstream/master' into insertOverwrite --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/4617#discussion_r24728239 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala --- @@ -66,14 +66,17 @@ class HiveContext(sc: SparkContext) extends SQLContext(sc) { override protected[sql] def executePlan(plan: LogicalPlan): this.QueryExecution = new this.QueryExecution(plan) + @transient + protected[sql] val ddlParserWithHiveQL = new DDLParser(HiveQl.parseSql(_)) --- End diff -- I am adding another ddl parser which takes `HiveQl.parseSql` to handle CTAS statements. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74447199 [Test build #27534 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27534/consoleFull) for PR 4617 at commit [`99da57e`](https://github.com/apache/spark/commit/99da57ea60b5a67978423631e81bba38373a6e06). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74447200 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27534/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/4617#discussion_r24728499 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala --- @@ -37,20 +37,22 @@ class ParquetQuerySuite extends QueryTest with ParquetTest { test(s$prefix: appending) { val data = (0 until 10).map(i = (i, i.toString)) + createDataFrame(data).toDataFrame(c1, c2).registerTempTable(tmp) withParquetTable(data, t) { -sql(INSERT INTO TABLE t SELECT * FROM t) +sql(INSERT INTO TABLE t SELECT * FROM tmp) checkAnswer(table(t), (data ++ data).map(Row.fromTuple)) } + catalog.unregisterTable(Seq(tmp)) } -// This test case will trigger the NPE mentioned in -// https://issues.apache.org/jira/browse/PARQUET-151. -ignore(s$prefix: overwriting) { +test(s$prefix: overwriting) { --- End diff -- @liancheng I re-enabled this one. Seems it is fine now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74448216 [Test build #27535 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27535/consoleFull) for PR 4617 at commit [`f30bdad`](https://github.com/apache/spark/commit/f30bdade7b1861f735c0ec3c720012d7317549d9). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74452654 [Test build #27535 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27535/consoleFull) for PR 4617 at commit [`f30bdad`](https://github.com/apache/spark/commit/f30bdade7b1861f735c0ec3c720012d7317549d9). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4617#issuecomment-74452655 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27535/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org