[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/11572 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-194013959 LGTM - merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193947308 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193947313 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52674/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193946796 **[Test build #52674 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52674/consoleFull)** for PR 11572 at commit [`d7472ea`](https://github.com/apache/spark/commit/d7472eafed5dcef0ad5c506cda5e4731c55fa6cf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193903126 **[Test build #52674 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52674/consoleFull)** for PR 11572 at commit [`d7472ea`](https://github.com/apache/spark/commit/d7472eafed5dcef0ad5c506cda5e4731c55fa6cf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193709857 While creating a PR I also saw some nits. At 'OrcRelation', 1. [`import com.google.common.base.Objects`](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala#L22) is not being used. 2. [`import org.apache.spark.sql.hive.HiveContext`](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala#L42) is not being used. 3. the exception message, [`"call writeInternal"`](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala#L176) has double spaces. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11572#discussion_r55340181
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -34,14 +34,39 @@ import org.apache.spark.sql.sources._
import org.apache.spark.sql.types.{CalendarIntervalType, StructType}
import org.apache.spark.util.Utils
-case class ResolvedDataSource(provider: Class[_], relation: BaseRelation)
-
/**
- * Responsible for taking a description of a datasource (either from
- * [[org.apache.spark.sql.DataFrameReader]], or a metastore) and
converting it into a logical
- * relation that can be used in a query plan.
+ * The main class responsible for representing a pluggable Data Source in
Spark SQL. In addition to
+ * acting as the canonical set of parameters that can describe a Data
Source, this class is used to
+ * resolve a description to a concrete implementation that can be used in
a query plan
+ * (either batch or streaming) or to write out out data using an external
library.
--- End diff --
nit: `write out ...`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193686921 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52634/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193686919 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193686598 **[Test build #52634 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52634/consoleFull)** for PR 11572 at commit [`6f6f099`](https://github.com/apache/spark/commit/6f6f0991a7a124390dc4da30c6a193a70fa901d3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193672169 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193672173 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52639/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193671389 **[Test build #52639 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52639/consoleFull)** for PR 11572 at commit [`dadd589`](https://github.com/apache/spark/commit/dadd589d11e2fa70ea756a9ba768a4663badf507). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class MetaStoreFileCatalog(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193663987 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52637/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193663983 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193663395 **[Test build #52637 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52637/consoleFull)** for PR 11572 at commit [`d83c2e2`](https://github.com/apache/spark/commit/d83c2e242fde0081c8215add1fb8381f5aeac2cb). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193639421 **[Test build #52639 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52639/consoleFull)** for PR 11572 at commit [`dadd589`](https://github.com/apache/spark/commit/dadd589d11e2fa70ea756a9ba768a4663badf507). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193634049 **[Test build #52637 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52637/consoleFull)** for PR 11572 at commit [`d83c2e2`](https://github.com/apache/spark/commit/d83c2e242fde0081c8215add1fb8381f5aeac2cb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193628454 **[Test build #52634 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52634/consoleFull)** for PR 11572 at commit [`6f6f099`](https://github.com/apache/spark/commit/6f6f0991a7a124390dc4da30c6a193a70fa901d3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193593900 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52625/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193593899 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193593805 **[Test build #52625 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52625/consoleFull)** for PR 11572 at commit [`cf7c719`](https://github.com/apache/spark/commit/cf7c719b72896450affad9b866ad9077a6140e40). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193565461 **[Test build #52625 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52625/consoleFull)** for PR 11572 at commit [`cf7c719`](https://github.com/apache/spark/commit/cf7c719b72896450affad9b866ad9077a6140e40). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193562059 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52624/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193562058 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193562054 **[Test build #52624 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52624/consoleFull)** for PR 11572 at commit [`36969f8`](https://github.com/apache/spark/commit/36969f8671ed396e9ed2027b8ee2c7435bbf7dfc). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13738][SQL] Cleanup Data Source resolut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11572#issuecomment-193561759 **[Test build #52624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52624/consoleFull)** for PR 11572 at commit [`36969f8`](https://github.com/apache/spark/commit/36969f8671ed396e9ed2027b8ee2c7435bbf7dfc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
