[
https://issues.apache.org/jira/browse/SPARK-53512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuchuan Huang updated SPARK-53512:
----------------------------------
Description:
Currently, the PushDownUtils has following matching cases.
{code:java}
object PushDownUtils {
def pushFilters(scanBuilder: ScanBuilder, filters: Seq[Expression])
: (Either[Seq[sources.Filter], Seq[Predicate]], Seq[Expression]) = {
scanBuilder match {
case r: SupportsPushDownFilters => ... // public interface extends
ScanBuilder
case r: SupportsPushDownV2Filters => ... // public interface extends
ScanBuilder
case f: FileScanBuilder => ... // which extends with
SupportsPushDownCatalystFilters
case _ => (Left(Nil), filters)
}
}
} {code}
As a result, when a new scanBuilder wants to SupportsPushDownCatalystFilters
(but doesn't want to extend FileScanBuilder), it will not be picked up by
PushDownUtils. To better unify these filter pushdown traits, it would be better
if the matching cases are like following:
{code:java}
scanBuilder match {
case r: SupportsPushDownFilters => ... // public interface extends ScanBuilder
case r: SupportsPushDownV2Filters => ... // public interface extends
ScanBuilder
case r: SupportsPushDownCatalystFilters => ... // trait extends ScanBuilder
case _ => (Left(Nil), filters)
}{code}
was:
Currently, the PushDownUtils has following matching cases.
{code:java}
object PushDownUtils {
def pushFilters(scanBuilder: ScanBuilder, filters: Seq[Expression])
: (Either[Seq[sources.Filter], Seq[Predicate]], Seq[Expression]) = {
scanBuilder match {
case r: SupportsPushDownFilters => ... // public interface extends
ScanBuilder
case r: SupportsPushDownV2Filters => ... // public interface extends
ScanBuilder
case f: FileScanBuilder => ... // which extends with
SupportsPushDownCatalystFilters
case _ => (Left(Nil), filters)
}
}
} {code}
As a result, when a new scanBuilder wants to SupportsPushDownCatalystFilters,
it will not be picked up by PushDownUtils. To better unify these filter
pushdown traits, it would be better if the matching cases are like following:
{code:java}
scanBuilder match {
case r: SupportsPushDownFilters => ... // public interface extends ScanBuilder
case r: SupportsPushDownV2Filters => ... // public interface extends
ScanBuilder
case r: SupportsPushDownCatalystFilters => ... // trait extends ScanBuilder
case _ => (Left(Nil), filters)
}{code}
> Better unification of DSv2 PushDownUtils
> ----------------------------------------
>
> Key: SPARK-53512
> URL: https://issues.apache.org/jira/browse/SPARK-53512
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: Yuchuan Huang
> Priority: Critical
>
> Currently, the PushDownUtils has following matching cases.
> {code:java}
> object PushDownUtils {
> def pushFilters(scanBuilder: ScanBuilder, filters: Seq[Expression])
> : (Either[Seq[sources.Filter], Seq[Predicate]], Seq[Expression]) = {
> scanBuilder match {
> case r: SupportsPushDownFilters => ... // public interface extends
> ScanBuilder
> case r: SupportsPushDownV2Filters => ... // public interface extends
> ScanBuilder
> case f: FileScanBuilder => ... // which extends with
> SupportsPushDownCatalystFilters
> case _ => (Left(Nil), filters)
> }
> }
> } {code}
> As a result, when a new scanBuilder wants to SupportsPushDownCatalystFilters
> (but doesn't want to extend FileScanBuilder), it will not be picked up by
> PushDownUtils. To better unify these filter pushdown traits, it would be
> better if the matching cases are like following:
> {code:java}
> scanBuilder match {
> case r: SupportsPushDownFilters => ... // public interface extends
> ScanBuilder
> case r: SupportsPushDownV2Filters => ... // public interface extends
> ScanBuilder
> case r: SupportsPushDownCatalystFilters => ... // trait extends ScanBuilder
> case _ => (Left(Nil), filters)
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
