[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18994 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18994 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88456/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18994 **[Test build #88456 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88456/testReport)** for PR 18994 at commit [`c126122`](https://github.com/apache/spark/commit/c126122879db0441abc8e2150d3caffa0d3b700a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user sureshthalamati commented on the issue: https://github.com/apache/spark/pull/18994 @ioana-delaney Thank you for pinging me. I would like to complete this PR. My responses might be slow due other commitments at my workplace , if I am blocking you please feel free to take over the PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18994 **[Test build #88456 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88456/testReport)** for PR 18994 at commit [`c126122`](https://github.com/apache/spark/commit/c126122879db0441abc8e2150d3caffa0d3b700a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18994 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1669/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18994 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user ioana-delaney commented on the issue: https://github.com/apache/spark/pull/18994 @sureshthalamati Hi Suresh, We are planning to proceed with the performance improvements. Will you be able to continue working on this PR? Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user sureshthalamati commented on the issue: https://github.com/apache/spark/pull/18994 Thank you for the input @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18994 Before we review the DDL changes, we need to see the PRs that can get benefits from this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user sureshthalamati commented on the issue: https://github.com/apache/spark/pull/18994 ping @gatorsmile @cloud-fan @rxin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18994 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18994 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82039/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18994 **[Test build #82039 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82039/testReport)** for PR 18994 at commit [`ea39601`](https://github.com/apache/spark/commit/ea39601829a8a1e4c5642e68ef0fab68310bdae1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18994 **[Test build #82039 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82039/testReport)** for PR 18994 at commit [`ea39601`](https://github.com/apache/spark/commit/ea39601829a8a1e4c5642e68ef0fab68310bdae1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user sureshthalamati commented on the issue: https://github.com/apache/spark/pull/18994 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18994 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82019/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18994 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18994 **[Test build #82019 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82019/testReport)** for PR 18994 at commit [`ea39601`](https://github.com/apache/spark/commit/ea39601829a8a1e4c5642e68ef0fab68310bdae1). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18994 **[Test build #82019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82019/testReport)** for PR 18994 at commit [`ea39601`](https://github.com/apache/spark/commit/ea39601829a8a1e4c5642e68ef0fab68310bdae1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18994 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18994 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81123/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18994 **[Test build #81123 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81123/testReport)** for PR 18994 at commit [`f1f6d35`](https://github.com/apache/spark/commit/f1f6d35e3e9968fa1720305daeeb4b10302b2332). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18994 **[Test build #81123 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81123/testReport)** for PR 18994 at commit [`f1f6d35`](https://github.com/apache/spark/commit/f1f6d35e3e9968fa1720305daeeb4b10302b2332). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user sureshthalamati commented on the issue: https://github.com/apache/spark/pull/18994 Created [SPARK-21823](https://issues.apache.org/jira/browse/SPARK-21823) and [SPARK-21824](https://issues.apache.org/jira/browse/SPARK-21824) for fixing the DDL's that impact the informational constraints. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18994 @sureshthalamati Sure. Please create sub-JIRAs for them --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user sureshthalamati commented on the issue: https://github.com/apache/spark/pull/18994 Thank you very much for reviewing @gatorsmile By scanning through the current supported DDL syntax for non-partition columns, I think following DDL statements will impact informational constraints: **ALTER STATEMENTS** ```sql ALTER TABLE name RENAME TO new_name ALTER TABLE name CHANGE column_name new_name new_type ``` Spark SQL can raise errors if the informational constraints are defined on the affected columns and let the user drop constraints before proceeding with the DDL. In the future we can enhance the affected DDL's to automatically fix up the constraint definition when possible, and not raise error When spark adds support for DROP/REPLACE of columns they will impact informational constraints. ```sql ALTER TABLE name DROP [COLUMN] column_name ALTER TABLE name REPLACE COLUMNS (col_spec[, col_spec ...]) ``` **DROP TABLE** ```sql DROP TABLE name ``` Hive drops the referential constraints automatically. Oracle requires user specify _[CASCADE CONSTRAINTS]_ clause to automatically drop the referential constraints, otherwise raises the error. Should we stick to the Hive behavior ? Fixing the affected DDLâs requires carrying additional dependency information as part of storing primary key definition, Is it ok if I fix the affected DDLS in a separate PR ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user sureshthalamati commented on the issue: https://github.com/apache/spark/pull/18994 sure. DDL that changes table name , column name and data type of the referenced primary key will affect foreign key definitions. I will check the spark DDL that does schema changes and get back to you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18994 Could you check whether the impact of the other DDL on the constraints? For example, rename. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18994 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18994 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80855/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18994 **[Test build #80855 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80855/testReport)** for PR 18994 at commit [`4839e84`](https://github.com/apache/spark/commit/4839e8419ca7360f0feafeceec8f3832102e3dba). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class TableConstraints(` * `sealed trait TableConstraint ` * `case class PrimaryKey(` * `case class ForeignKey(` * `case class AlterTableAddConstraintCommand(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18994 **[Test build #80855 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80855/testReport)** for PR 18994 at commit [`4839e84`](https://github.com/apache/spark/commit/4839e8419ca7360f0feafeceec8f3832102e3dba). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user sureshthalamati commented on the issue: https://github.com/apache/spark/pull/18994 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18994 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80851/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18994 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18994 **[Test build #80851 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80851/testReport)** for PR 18994 at commit [`4839e84`](https://github.com/apache/spark/commit/4839e8419ca7360f0feafeceec8f3832102e3dba). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class TableConstraints(` * `sealed trait TableConstraint ` * `case class PrimaryKey(` * `case class ForeignKey(` * `case class AlterTableAddConstraintCommand(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18994 **[Test build #80851 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80851/testReport)** for PR 18994 at commit [`4839e84`](https://github.com/apache/spark/commit/4839e8419ca7360f0feafeceec8f3832102e3dba). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
