[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16938 thanks, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73680/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73680 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73680/testReport)** for PR 16938 at commit [`d78b7d5`](https://github.com/apache/spark/commit/d78b7d5f0dfd45661e30a90c1cabf7a30278eb3b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73680 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73680/testReport)** for PR 16938 at commit [`d78b7d5`](https://github.com/apache/spark/commit/d78b7d5f0dfd45661e30a90c1cabf7a30278eb3b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 I am modifying the hacky code --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73648/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73648 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73648/testReport)** for PR 16938 at commit [`a8dbcca`](https://github.com/apache/spark/commit/a8dbccaff206df4773541527f00e340a5ce3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73648 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73648/testReport)** for PR 16938 at commit [`a8dbcca`](https://github.com/apache/spark/commit/a8dbccaff206df4773541527f00e340a5ce3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73634/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73634 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73634/testReport)** for PR 16938 at commit [`2498dfd`](https://github.com/apache/spark/commit/2498dfd8df6e93d5df2a5d99653e335dd4c9365d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73634 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73634/testReport)** for PR 16938 at commit [`2498dfd`](https://github.com/apache/spark/commit/2498dfd8df6e93d5df2a5d99653e335dd4c9365d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73587/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73587 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73587/testReport)** for PR 16938 at commit [`304ae31`](https://github.com/apache/spark/commit/304ae3112950a80b4ff2a980199c7817c0d0562a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73587 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73587/testReport)** for PR 16938 at commit [`304ae31`](https://github.com/apache/spark/commit/304ae3112950a80b4ff2a980199c7817c0d0562a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 @gatorsmile @cloud-fan could you help to review this pr? thanks :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73424/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73424 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73424/testReport)** for PR 16938 at commit [`416ea37`](https://github.com/apache/spark/commit/416ea37b8c85040cff868007f6c5fea55f9b2d16). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73424/testReport)** for PR 16938 at commit [`416ea37`](https://github.com/apache/spark/commit/416ea37b8c85040cff868007f6c5fea55f9b2d16). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73411/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73411 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73411/testReport)** for PR 16938 at commit [`5a3e5ac`](https://github.com/apache/spark/commit/5a3e5ac98855fe9f474a6c5e44eab42bee6c3d08). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73399/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73411 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73411/testReport)** for PR 16938 at commit [`5a3e5ac`](https://github.com/apache/spark/commit/5a3e5ac98855fe9f474a6c5e44eab42bee6c3d08). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73405/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73405 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73405/testReport)** for PR 16938 at commit [`1f2ce17`](https://github.com/apache/spark/commit/1f2ce17e3d2eca92bc01b6a22e908bd8fd1d9592). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73400/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73400 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73400/testReport)** for PR 16938 at commit [`afa1313`](https://github.com/apache/spark/commit/afa13136d6d24313c8f18bb7ed175bf45079476a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73405 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73405/testReport)** for PR 16938 at commit [`1f2ce17`](https://github.com/apache/spark/commit/1f2ce17e3d2eca92bc01b6a22e908bd8fd1d9592). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73400/testReport)** for PR 16938 at commit [`afa1313`](https://github.com/apache/spark/commit/afa13136d6d24313c8f18bb7ed175bf45079476a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #73399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73399/testReport)** for PR 16938 at commit [`8559e4e`](https://github.com/apache/spark/commit/8559e4e8f9b8e8f773f4d336866a01ff15c9fc5e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 @gatorsmile I have test it for `partition path exists` , the result is still same with `table path exists` **2. CREATE TABLE ...PARTITIONED BY ... LOCATION path AS SELECT ...** a) path exists hive(external) -> not support spark(hive with HiveExternalCatalog) -> ok spark(parquet with HiveExternalCatalog) -> throw exception(path already exists) spark(parquet with InMemoryCatalog) -> throw exception(path already exists) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16938 @tejasapatil Spark doesn't need to be exactly same with Hive, we follow hive behavior if it's reasonable, or use our own logic if hive's behavior doesn't make sense. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user tejasapatil commented on the issue: https://github.com/apache/spark/pull/16938 I looked into the code. Looks like that version is merely for picking the hive shim and metastore interactions and got nothing to do with semantics of SQL operations. So you are most likely correct. @gatorsmile @cloud-fan : Is the goal of hive support in spark to be adherent with a specific release of Hive (as long as the hive behavior is sane and consistent .. otherwise it doesn't make sense to follow it) ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 @tejasapatil In my opinion, test in Hive 2.0.0 just make a compare with Spark, the target is to determine these actions in Spark, not to make consist with Hive 2.0.0 or Hive 1.2.1, isn't it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user tejasapatil commented on the issue: https://github.com/apache/spark/pull/16938 @windpiger : I realised that you are checking the hive behavior against Hive 2.0.0. Spark is expected to support semantics for Hive 1.2.1 : https://github.com/apache/spark/blob/3881f342b49efdb1e0d5ee27f616451ea1928c5d/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala#L58 I am not upto date with the differences between those two releases of hive wrt this discussion. Can you confirm if the observations reported earlier in the discussion are valid against Hive 1.2.1 ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16938 Thank you for your work! Maybe the last question. ``` **2. CREATE TABLE ...PARTITIONED BY ... LOCATION path AS SELECT ...** a) path exists hive(external) -> not support spark(hive with HiveExternalCatalog) -> ok spark(parquet with HiveExternalCatalog) -> throw exception(path already exists) spark(parquet with InMemoryCatalog) -> throw exception(path already exists) ``` In the above case, you used `path exists`. I assumed this is the existence of the table directory. Are these behaviors still the same when the specific partition directory exists? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 oh, you are right~ thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16938 Basically, the rules you proposed are - When users specify the location in CT or CTAS (i.e., creating an external table), we should create a new directory if not existed, or overwrite the directory if already created. - When users do not specify the location in CT or CTAS (i.e., creating a manged table), we should create a new directory if not existed. If the directory already exists, we should issue the error. Is my understanding right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16938 I found you also changed the following cases: **4. CREATE TABLE ** **5. CREATE TABLE ... AS SELECT ...** Actually, they are managed tables. You do not need to update them. Can you roll back the changes? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 @cloud-fan @gatorsmile @tejasapatil As we discussed above, we have three actions to do: # 1. CREATE TABLE ... (PARTITIONED BY ...) LOCATION path *situation:path not exists* Item | Before| After | - | - spark(parquet with HiveExternalCatalog) | throw exception(path does not exists) | ok spark(parquet with InMemoryCatalog) | throw exception(path does not exists) | ok # 2. CREATE TABLE ...(PARTITIONED BY ...) LOCATION path AS SELECT *situation:path exists* Item | Before| After | - | - spark(parquet with HiveExternalCatalog) | throw exception(path already exists) | ok spark(parquet with InMemoryCatalog) | throw exception(path already exists) | ok # 3. CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... *situation:default warehouse table path exists* Item | Before| After | - | - spark(hive with HiveExternalCatalog) | ok | throw exception(path already exists) please help to confirm the actions above, if it is ok, situation 2 is this PR going to resolve, and I will make another PR to resolve situation 1&3, thanks~ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 @gatorsmile sorry, I make a mistake of this, I have updated the compare test above. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16938 Based on [the doc](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTableAsSelect(CTAS)), Hive does not support CTAS when the target table is external. **1. CREATE TABLE ... LOCATION path** **2. CREATE TABLE ... LOCATION path AS SELECT ...** When you testing the above two cases, you need to change the syntax a little bit by manually adding `EXTERNAL`. Spark SQL actually is creating an external table. This is different from Hive. **1. CREATE EXTERNAL TABLE ... LOCATION path** **2. CREATE EXTERNAL TABLE ... LOCATION path AS SELECT ...** Could you update the comparison? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16938 @windpiger yes for both questions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 @cloud-fan situation 2. CREATE TABLE ...(PARTITIONED BY ...) LOCATION path AS SELECT ... is different for `path exists`, which is this PR going to resolve. It is ok to make it consist with hive with HiveExternalCatalog in spark? situation 3. CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... is also different for default warehouse table `path exists`, do you mean that the parquet action is expected that throw an already exist exception, and hive should make consist with it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 @tejasapatil * throw exception is the result of the test, It is really happened in current spark master branch * Hive CTAS not support for partition table [hive-doc](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTableAsSelect(CTAS)) * `default warehouse table path exist` means that we already create a table path under warehouse path, before we create the table. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16938 > CREATE TABLE ... (PARTITIONED BY ...) LOCATION path I think hive's behavior makes more sense. Users may wanna insert data to this table and put the data in a specified location, even it doesn't exist at the beginning. > CREATE TABLE ...(PARTITIONED BY ...) LOCATION path AS SELECT ... The reason applies here too. > CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... When users don't specify the location, mostly they would expect this is a fresh table and the table path should not exist. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user tejasapatil commented on the issue: https://github.com/apache/spark/pull/16938 @windpiger : - what does `throw exception(...)` mean ? Operation is supported OR not ? it might throw exception but the operation itself might have happened. - for 2nd point, you said hive does not support that. Can you share the error message ? I am trying to understand if there is a reason why hive does not allow that and with Spark we would also need to think about that. - I could not understand what `default warehouse table path exists` in 3rd point means --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 @gatorsmile I have test all the cases above updated. The result shows that spark for datasource table with HiveExternalCatalog and InMemoryCatatlog have the same actions. spark for hive table passed all the tests above. there are some different between spark for hive table and spark for datasource table. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 @gatorsmile Sorry, I forget to declare that ,Above tests, spark represents parquet table with HiveExternalCatalog , hive represents hive table in hive2.0.0. I will add hive serde table for spark with HiveExternalCatalogï¼ and parquet/hive serde table with InMemoryCatalog soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16938 Could you check the behaviors for both data source tables and hive serde tables? Later, we also need to check the behaviors of InMemoryCatalog for data source tables without enabling Hive supports. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16938 @windpiger Thank you for your efforts! What you did above need to be written as the test cases. Could you do it as a separate PR? In addition, all the cases you tried are only for hive serve tables, right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 Compare spark-master branch and hive-2.0.0 **1. CREATE TABLE ... PARTITIONED BY ... LOCATION path** ``` a) path exists hive -> ok spark -> ok b) path not exists hive -> ok spark -> throw exception(path does not exists) ``` **2. CREATE TABLE ...PARTITIONED BY ... LOCATION path AS SELECT ...** ``` a) path exists hive -> not support spark -> throw exception(path already exists) b) path not exists hive -> not support spark -> ok ``` **3. ALTER TABLE ... PARTITION(...) ... SET LOCATION path** ``` a) path exists hive -> ok spark -> ok b) path not exists hive -> ok spark -> ok ``` **4. CREATE TABLE PARTITIONED BY ...** ``` a) default warehouse table path exists hive -> ok spark -> ok b) default warehouse table path not exists hive -> ok spark -> ok ``` **5. CREATE TABLE ... AS SELECT ...** ``` a) default warehouse table path exists hive -> not support spark -> throw exception(path already exists) b) default warehouse table path not exists hive -> not support spark -> ok ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 **1. CREATE TABLE ... LOCATION path** ``` a) path exists hive -> ok spark -> ok b) path not exists hive -> ok spark -> throw exception(path does not exists) ``` **2. CREATE TABLE ... LOCATION path AS SELECT ...** ``` a) path exists hive -> throw exception(CREATE-TABLE-AS-SELECT cannot create table with location to a non-empty directory.) spark -> throw exception(path already exists) b) path not exists hive -> ok spark -> ok ``` **3. ALTER TABLE ... SET LOCATION path** ``` a) path exists hive -> ok spark -> ok b) path not exists hive -> ok spark -> ok ``` **4. CREATE TABLE ... ** ``` a) default warehouse table path exists hive -> ok spark -> ok b) default warehouse table path not exists hive -> ok spark -> ok ``` **5. CREATE TABLE ... AS SELECT ...** ``` a) default warehouse table path exists hive -> ok spark -> throw exception(path already exists) b) default warehouse table path not exists hive -> ok spark -> ok ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16938 One more case: 5. `CREATE TABLE` or `CTAS` without the location spec: if the default path exists, should we succeed or fail? After we finishing the TABLE-level DDLs, we also need to do the same things for DATABASE-level DDLs and PARTITION-level DDLs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16938 ok let's discuss it case by case: 1. `CREATE TABLE ... LOCATION path` works if path exists, it's expected 2. `CREATE TABLE ... LOCATION path` fails if path doesn't exist, is it expected? 3. `CREATE TABLE ... LOCATION path AS SELECT ...`, shall we fail if path exists? 4. `ALTER TABLE ... SET LOCATION path`, shall we fail if path not exist? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 @cloud-fan @gatorsmile @tejasapatil let's discuss this together? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 I think in CTASï¼it is not allowed an existed tableï¼ no strict for the path exists. In DataFrameWriter.save with errorifnotexist modeï¼path existed is not allowed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user tejasapatil commented on the issue: https://github.com/apache/spark/pull/16938 From what I understand, this change is applicable for EXTERNAL tables only. There are two main uses of EXTERNAL tables I am aware of (repost from https://github.com/apache/spark/pull/16868#issuecomment-279282420): - Ingest data from non-hive locations into Hive tables. - Create a logical "pointer" to an existing hive table / partition (without creating multiple copies of the underlying data). Ability to point to random location (which already has data) and create an EXTERNAL table over it is important for supporting EXTERNAL tables. If we don't allow this PR, then the options left to users are: - Create an external table and point to some non-existing location. - Later do either of these 2 things: - issue `ALTER TABLE SET LOCATION` to set the external table's location to the source location having desired data. - do a `dfs -mv` from the source location of the data to the new location which the table points at. This will be nasty in case your source data was a managed table location. @cloud-fan : I don't think Spark's interpretation of EXTERNAL tables is different from Hive's. If it is, can you share the differences ? I think we should allow this. If you have specific concerns, lets discuss those. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16938 We need to define a consistent rule in Catalog how to handle the scenario when the to-be-created directory already exists. So far, in most DDL scenarios, when trying to create a directory but it already exists, we just simply use the existing directory without an error message. `mkdir` does not complain if the destination directory exists. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16938 I don't think we should treat it as a bug just because hive supports it, we should think more. Does it make sense to specify an existing directory in CTAS? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/16938 cc @gatorsmile @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72932/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16938 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #72932 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72932/testReport)** for PR 16938 at commit [`058865b`](https://github.com/apache/spark/commit/058865bdac9895c1f810be2dcc3439f2a7d17b70). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16938: [SPARK-19583][SQL]CTAS for data source table with a crea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16938 **[Test build #72932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72932/testReport)** for PR 16938 at commit [`058865b`](https://github.com/apache/spark/commit/058865bdac9895c1f810be2dcc3439f2a7d17b70). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
