[
https://issues.apache.org/jira/browse/SPARK-55869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18063753#comment-18063753
]
Scott Schenkein commented on SPARK-55869:
-----------------------------------------
[~imback82] [~huaxingao] [~aokolnychyi]
I see you have predicate pushdown commits. Would very much appreciate your
help finding a reviewer!
> Extensible predicate pushdown for DataSource V2
> ------------------------------------------------
>
> Key: SPARK-55869
> URL: https://issues.apache.org/jira/browse/SPARK-55869
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Scott Schenkein
> Priority: Major
> Labels: SQL, connector, datasource, predicate,
> pull-request-available, pushdown
> Attachments: SPARK-55869-extended-predicate-pushdown.md
>
>
> h2. Problem
>
> DataSource V2 predicate pushdown is limited to a fixed, closed set of
> expression types hardcoded in {{{}V2ExpressionBuilder{}}}. Data source
> authors who need to push predicates involving:
> *Custom operators* (e.g. {{{}my_col indexquery 'param'{}}},
> {{{}my_boolean_function(col1, col2, 'p'){}}})
> *Builtin Spark expressions not in the pushdown whitelist* (e.g.
> {{{}RLIKE{}}}, {{{}LIKE{}}}, {{{}ILIKE{}}})
> are forced to resort to fragile workarounds: intercepting the logical plan
> via {{SparkSessionExtensions}} injected rules, using thread-local state to
> smuggle filter information past the optimizer, and effectively reimplementing
> pushdown outside Spark's designed architecture.
>
> These hacks are:
> * Brittle across Spark versions
> * Invisible to Spark's query planning (no EXPLAIN output, no metrics)
> * Unable to participate in Spark's post-scan filter safety net
> * Duplicative — every data source author reinvents the same machinery
> h2. Proposed Solution
> Three independently adoptable layers:
> *Layer 1: Capability-gated builtin predicate translation*
> New {{SupportsPushDownPredicateCapabilities}} interface (mix-in for
> {{{}ScanBuilder{}}}) lets data sources declare which additional V2 predicate
> names they support (e.g. {{{}"RLIKE"{}}}, {{{}"LIKE"{}}}).
> {{V2ExpressionBuilder}} uses this set to conditionally translate builtin
> Catalyst expressions that currently have no match case.
> Tier 1 builtins: {{{}LIKE{}}}, {{{}RLIKE{}}}, {{{}IS_NAN{}}},
> {{{}ARRAY_CONTAINS{}}}, {{{}MAP_CONTAINS_KEY{}}}.
> *Layer 2: Custom predicate functions*
> New {{SupportsCustomPredicates}} interface (mix-in for {{{}Table{}}}) lets
> data sources declare custom boolean predicate functions with dot-qualified
> canonical names (e.g. {{{}"com.mycompany.INDEXQUERY"{}}}).
> A new analyzer rule ({{{}ResolveCustomPredicates{}}}) resolves these
> during analysis. A new {{CustomPredicateExpression}} Catalyst node translates
> to a namespaced V2 {{Predicate}} during pushdown. A safety rule
> ({{{}EnsureCustomPredicatesPushed{}}}) fails queries if a custom predicate
> wasn't pushed.
>
> *Layer 3: Custom infix operator syntax*
> Helper base class {{CustomOperatorParserExtension}} simplifies parser
> extensions that rewrite custom infix operators (e.g. {{{}col INDEXQUERY
> 'param'{}}}) to function call syntax, composing cleanly with Layer 2.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
