[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14809 thanks for the review, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14809 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65002/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14809 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14809 **[Test build #65002 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65002/consoleFull)** for PR 14809 at commit [`7caf2fb`](https://github.com/apache/spark/commit/7caf2fb92ccd010fc38f4d79e07e4e67de168062). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14809 **[Test build #65002 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65002/consoleFull)** for PR 14809 at commit [`7caf2fb`](https://github.com/apache/spark/commit/7caf2fb92ccd010fc38f4d79e07e4e67de168062). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14809 LGTM pending test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14809 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14809 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14809 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64988/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14809 **[Test build #64988 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64988/consoleFull)** for PR 14809 at commit [`7caf2fb`](https://github.com/apache/spark/commit/7caf2fb92ccd010fc38f4d79e07e4e67de168062). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14809 **[Test build #64988 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64988/consoleFull)** for PR 14809 at commit [`7caf2fb`](https://github.com/apache/spark/commit/7caf2fb92ccd010fc38f4d79e07e4e67de168062). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14809 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14809 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64981/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14809 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14809 **[Test build #64981 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64981/consoleFull)** for PR 14809 at commit [`7caf2fb`](https://github.com/apache/spark/commit/7caf2fb92ccd010fc38f4d79e07e4e67de168062). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14809 **[Test build #64981 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64981/consoleFull)** for PR 14809 at commit [`7caf2fb`](https://github.com/apache/spark/commit/7caf2fb92ccd010fc38f4d79e07e4e67de168062). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14809 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14809 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14809 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64974/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14809 **[Test build #64974 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64974/consoleFull)** for PR 14809 at commit [`7caf2fb`](https://github.com/apache/spark/commit/7caf2fb92ccd010fc38f4d79e07e4e67de168062). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14809 **[Test build #64974 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64974/consoleFull)** for PR 14809 at commit [`7caf2fb`](https://github.com/apache/spark/commit/7caf2fb92ccd010fc38f4d79e07e4e67de168062). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14809 Yeah, that is a bug. We did not get an exception when we read it, but we can get the error when trying to write it. The error message is confusing ``` Can only write data to relations with a single path.; org.apache.spark.sql.AnalysisException: Can only write data to relations with a single path.; at org.apache.spark.sql.execution.datasources.DataSourceAnalysis$$anonfun$apply$1.applyOrElse(DataSourceStrategy.scala:167) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14809 ah i see, btw in your example when will we throw exception? when we read it? a file-based external table without path is invalid. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14809 If we do not specify the schema, it will behave like what you said. For example, ```Scala sparkSession.catalog.createExternalTable( "createdParquetTable", "parquet", Map.empty[String, String]) ``` ``` Unable to infer schema for ParquetFormat at . It must be specified manually; org.apache.spark.sql.AnalysisException: Unable to infer schema for ParquetFormat at . It must be specified manually; ``` However, if we specify the schema, we will not get an error. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14809 @gatorsmile can you explain more about this example? I think we will throw exception in `CreateDataSourceTableCommand` when we create a `DataSource` and call its `resolveRelation`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14809
Condition 2 is not always true when condition 1 is `true`. I found an
exception.
```Scala
val schema = StructType(StructField("b", StringType, true) :: Nil)
sparkSession.catalog.createExternalTable(
"createdParquetTable",
"parquet",
schema,
Map.empty[String, String])
```
I think this is a bug. Do you want to fix it in this PR? Or I can fix it in
another PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14809 https://github.com/apache/spark/pull/13060 This change is based on another PR. If users specify the location in `CREATE TABLE`, we always set the table type to `EXTERNAL`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14809 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64425/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14809 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14809 **[Test build #64425 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64425/consoleFull)** for PR 14809 at commit [`915d2b5`](https://github.com/apache/spark/commit/915d2b5a1dd8c26a37d0b99ba0503a0d95b6f3f3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14809 **[Test build #64425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64425/consoleFull)** for PR 14809 at commit [`915d2b5`](https://github.com/apache/spark/commit/915d2b5a1dd8c26a37d0b99ba0503a0d95b6f3f3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14809: [SPARK-17238][SQL] simplify the logic for converting dat...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14809 cc @yhuai @gatorsmile @liancheng @clockfly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
