Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/13283#discussion_r64807974
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -448,6 +448,14 @@ class Analyzer(
}
}
+ /**
+ * Data source formats that were not supported in direct query on file
+ */
+ private final val unsupportedFileQuerySource = Set(
+ "org.apache.spark.sql.jdbc",
--- End diff --
@zsxwing @marmbrus Agree. It is hard to maintain the list. First, will add
the missing class names to the list.
To implement what @marmbrus suggested, I think we need to do the following:
- Users' input are not full class names. To dynamically load the class, we
still need to have a list like:
https://github.com/apache/spark/blob/361ebc282b2d09dc6dcf21419a53c5c617b1b6bd/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L80-L108
To avoid duplicating the codes, we need to move this list to a place both
`Analyzer` and `DataSource` can access. Any suggestion here?
- Dynamically loading the class and check whether the class extends
`FileFormat`. We need to do something similar like
https://github.com/apache/spark/blob/361ebc282b2d09dc6dcf21419a53c5c617b1b6bd/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L128-L132
- To avoid duplicate the codes, when unable to loading the class, we still
follow the existing way to handle it. That is, returns the `UnresolvedRelation`
`u`.
Is my understanding right?
Thank you for your suggestions!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
