[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user dilipbiswal commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-217100315 @liancheng Hi Lian, in this PR, i had implemented "describe table partition" and "describe column". Do you want me to put this on top of your describe table changes ? Let me know please. If you plan to work on it then let me know. @viirya - fyi. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user dilipbiswal closed the pull request at: https://github.com/apache/spark/pull/12460 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-216893487 It looks like this can be closed because https://github.com/apache/spark/pull/12844 was merged --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-215939351 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57400/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-215939350 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-215939319 **[Test build #57400 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57400/consoleFull)** for PR 12460 at commit [`319d45b`](https://github.com/apache/spark/commit/319d45ba2a46e317b6576160284d80df9f73c023). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-215931419 **[Test build #57400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57400/consoleFull)** for PR 12460 at commit [`319d45b`](https://github.com/apache/spark/commit/319d45ba2a46e317b6576160284d80df9f73c023). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user dilipbiswal commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-215195961 @liancheng Thank you for your comment. Actually initially i started with the idea of serving the describe command solely from `CatalogTable`. I then realized that `CatalogTable` may not have all the metadata information that is required for this command. So i have a couple of high level questions: 1. Can we add more fields to CatalogTable ? * Some example of fields that miss are retention, privileges. * When we choose "describe extended partition", quite a few details that are readily available in HivePartition is not present in our CatalogTablePartition object. * Another use case is "describe table column_path". This is served by a call to Hive's deserializer via. Hive.getFieldsFromDeserializer 2. Do we have flexibility on the output of describe command or we need match hive's output completely ? If so, we can remove the describe-related tests from `HiveCompatibiltySuite` and add suitable tests in `SQLQuerySuite`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-215115938 @dilipbiswal One purpose of re-implementing all DDL as native Spark SQL command is to minimize dependency to Hive so that we can move Hive into a separate data source some day. That said, we really don't want to make these new DDL commands rely on classes like `HiveClient`, `HiveClientImpl`, or `HiveSessionCatalog`. When you need to access Hive table metadata, you should access them via `CatalogTable` rather than depending on any Hive data structure. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-215034168 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57106/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-215034163 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-215033667 **[Test build #57106 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57106/consoleFull)** for PR 12460 at commit [`34f6d32`](https://github.com/apache/spark/commit/34f6d32fa6af50bb7b33df665385f2e5ba3b5efe). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-215008628 **[Test build #57106 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57106/consoleFull)** for PR 12460 at commit [`34f6d32`](https://github.com/apache/spark/commit/34f6d32fa6af50bb7b33df665385f2e5ba3b5efe). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-214943531 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-214943532 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57062/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-214943392 **[Test build #57062 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57062/consoleFull)** for PR 12460 at commit [`83c2875`](https://github.com/apache/spark/commit/83c2875fc0d51bd9ff8ebe5b6f52bb9a17911184). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-214928492 **[Test build #57062 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57062/consoleFull)** for PR 12460 at commit [`83c2875`](https://github.com/apache/spark/commit/83c2875fc0d51bd9ff8ebe5b6f52bb9a17911184). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user dilipbiswal commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-214852703 @liancheng Hi Lian, can you please look over this PR and give some comments. Thanks !! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212597432 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56394/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212597429 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212596804 **[Test build #56394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56394/consoleFull)** for PR 12460 at commit [`eb1c30e`](https://github.com/apache/spark/commit/eb1c30e60792db3c8dce545b93b260ac795c728d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212558294 **[Test build #56394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56394/consoleFull)** for PR 12460 at commit [`eb1c30e`](https://github.com/apache/spark/commit/eb1c30e60792db3c8dce545b93b260ac795c728d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user dilipbiswal commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212557057 rebased.. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212537783 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56371/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212537781 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212537369 **[Test build #56371 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56371/consoleFull)** for PR 12460 at commit [`41cf12d`](https://github.com/apache/spark/commit/41cf12d10309adc0ddc9f1cbdc32fbbf4ab22f94). * This patch passes all tests. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212486789 **[Test build #56371 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56371/consoleFull)** for PR 12460 at commit [`41cf12d`](https://github.com/apache/spark/commit/41cf12d10309adc0ddc9f1cbdc32fbbf4ab22f94). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212349369 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56335/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212349365 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212348881 **[Test build #56335 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56335/consoleFull)** for PR 12460 at commit [`868b438`](https://github.com/apache/spark/commit/868b438244de7bdb4d53ba28a9d9e36e7d2c). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212335624 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56333/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212335619 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212335071 **[Test build #56333 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56333/consoleFull)** for PR 12460 at commit [`98df1d8`](https://github.com/apache/spark/commit/98df1d89e91506abeb32c0707db061cb3538588e). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212297857 **[Test build #56335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56335/consoleFull)** for PR 12460 at commit [`868b438`](https://github.com/apache/spark/commit/868b438244de7bdb4d53ba28a9d9e36e7d2c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/12460#discussion_r60357946
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -254,6 +251,21 @@ class SparkSqlAstBuilder extends AstBuilder {
}
}
+ /**
+* A column path can be specified as an parameter to describe command.
It is a dot separated
+* elements where the last element can be a String.
+* TODO - check with Herman
--- End diff --
@hvanhovell Yeah. I have attempted to support the key/value/elem keywords.
Could you please check to see if there are any issues ? I am also trying to
test this a bit more in parallel.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-212287159 **[Test build #56333 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56333/consoleFull)** for PR 12460 at commit [`98df1d8`](https://github.com/apache/spark/commit/98df1d89e91506abeb32c0707db061cb3538588e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12460#discussion_r60354533
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -254,6 +251,21 @@ class SparkSqlAstBuilder extends AstBuilder {
}
}
+ /**
+* A column path can be specified as an parameter to describe command.
It is a dot separated
+* elements where the last element can be a String.
+* TODO - check with Herman
--- End diff --
@dilipbiswal Do you plan on supporting the key/value/elem keywords and
nested elements? Which would be cool.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/12460#discussion_r60347793
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -254,6 +251,21 @@ class SparkSqlAstBuilder extends AstBuilder {
}
}
+ /**
+* A column path can be specified as an parameter to describe command.
It is a dot separated
+* elements where the last element can be a String.
+* TODO - check with Herman
--- End diff --
@hvanhovell Let me work on the grammar change. I will introduce a rule
colPathIdentifier which is basically a regular identifier or the set of key,
value, elem keywords.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/12460#discussion_r60335414
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -254,6 +251,21 @@ class SparkSqlAstBuilder extends AstBuilder {
}
}
+ /**
+* A column path can be specified as an parameter to describe command.
It is a dot separated
+* elements where the last element can be a String.
+* TODO - check with Herman
--- End diff --
@hvanhovell Hi Herman, I tried very simple scenarios of using nested
columns and it seems to work ok. Let me paste the output here.
map
``` SQL
create table mp_t1 (a map , b string) row format delimited
collection items terminated by '$' map keys terminated by '#';
load data local inpath '/data/mapfile' overwrite into table mp_t1;
select * from mp_t1;
a b
{100:"spark"} ABC
describe extended mp_t1.a.$key$;
Result
==
$key$ int from deserializer
```
Struct
``` SQL
create table ct_t (a struct, b string) stored as
textfile;
insert into ct_t values (('abc', 'efg'), 'ABC');
spark-sql> select * from ct_t;
{"n1":"abb","n2":"efg"} ABC
spark-sql> describe extended ct_t.a.n1;
OK
n1 string from deserializer
```
Herman, based on hive syntax diagram, i was expecting the following
command to work.
describe extended mp_t1.a.'$key$';
However, i get a parse exception and when i remove the quotes it works like
following.
describe extended mp_t1.a.$key$
Given this, we can simply change the grammar to use a dot separated list of
identifiers, right ? Please
let me know what you think..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12460#discussion_r60186589
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -254,6 +251,21 @@ class SparkSqlAstBuilder extends AstBuilder {
}
}
+ /**
+* A column path can be specified as an parameter to describe command.
It is a dot separated
+* elements where the last element can be a String.
+* TODO - check with Herman
--- End diff --
Ok, lets remove this from the grammar as well, and just use a dot separated
list of identifiers. Actually, are we currently able to deal with nested
columns?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/12460#discussion_r60180497
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -254,6 +251,21 @@ class SparkSqlAstBuilder extends AstBuilder {
}
}
+ /**
+* A column path can be specified as an parameter to describe command.
It is a dot separated
+* elements where the last element can be a String.
+* TODO - check with Herman
--- End diff --
Yeah Herman. Not supporting it would certainly simplify things. FYI - I
checked that the unit test case describe_xpath.q which exercises this syntax is
not getting tested in HiveCompatibleSuite.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12460#discussion_r60175987
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -254,6 +251,21 @@ class SparkSqlAstBuilder extends AstBuilder {
}
}
+ /**
+* A column path can be specified as an parameter to describe command.
It is a dot separated
+* elements where the last element can be a String.
+* TODO - check with Herman
--- End diff --
It is a bit more complicates than I thought. We allow strings here because
Hive allows us to use the `'$elem'`, `'$keys'` and `'$values'` 'keywords'. That
is why I added strings to the rule. I am not sure if we should support this.
What do you guys think?
This is what I found in the Hive manual:
```SQL
DESCRIBE [EXTENDED|FORMATTED] [db_name.]table_name[ col_name (
[.field_name] | [.'$elem$'] | [.'$key$'] | [.'$value$'] )* ];
```
See also:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Describe
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-211543186 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56071/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-211543181 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-211542850 **[Test build #56071 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56071/consoleFull)** for PR 12460 at commit [`cfb0eeb`](https://github.com/apache/spark/commit/cfb0eeb71a136865154772b2e285b9918d5c9a30). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-211487455 **[Test build #56071 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56071/consoleFull)** for PR 12460 at commit [`cfb0eeb`](https://github.com/apache/spark/commit/cfb0eeb71a136865154772b2e285b9918d5c9a30). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12460#discussion_r60006341
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -254,6 +251,21 @@ class SparkSqlAstBuilder extends AstBuilder {
}
}
+ /**
+* A column path can be specified as an parameter to describe command.
It is a dot separated
+* elements where the last element can be a String.
+* TODO - check with Herman
--- End diff --
cc @hvanhovell
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user dilipbiswal commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-211169665 @gatorsmile Thank you. I have resolved the conflicts. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user gatorsmile commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-211163664 Please resolve the conflicts. : ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL][WIP] Describe table
Github user dilipbiswal commented on the pull request: https://github.com/apache/spark/pull/12460#issuecomment-211163482 @andrewor14 Looking for some early feedback on this as i was thinking to do the same for show table extended. I did have a brief discussion with @gatorsmile on this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
