Github user dbtsai commented on a diff in the pull request:
https://github.com/apache/spark/pull/22573#discussion_r221128544
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -437,53 +436,65 @@ object DataSourceStrategy {
* @return a `Some[Filter]` if the input [[Expression]] is convertible,
otherwise a `None`.
*/
protected[sql] def translateFilter(predicate: Expression):
Option[Filter] = {
+ // Recursively try to find an attribute name from the top level that
can be pushed down.
+ def attrName(e: Expression): Option[String] = e match {
+ // In Spark and many data sources such as parquet, dots are used as
a column path delimiter;
+ // thus, we don't translate such expressions.
+ case a: Attribute if !a.name.contains(".") =>
+ Some(a.name)
--- End diff --
Do we have any data source currently supporting `dot` in the column name
with pushdown? The worst case will be no pushdown for those data sources.
I know ORC doesn't work for now. We can have another followup PR to address
this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
