Just like our discussion in thread https://lists.apache.org/thread/whh75f6rtwdyqxt47gb39j6m6m0cpphq , +1 for this Flip.
-- Best! Xuyang 在 2023-10-24 18:03:36,"Jiabao Sun" <jiabao....@xtransfer.cn.INVALID> 写道: >Thanks Jane, > >I believe that the configuration "table.optimizer.source.predicate" has a >higher priority at the planner level than the configuration at the source >level, >and it seems easy to implement now. > >Best, >Jiabao > > >> 2023年10月24日 17:36,Jane Chan <qingyue....@gmail.com> 写道: >> >> Hi Jiabao, >> >> Thanks for driving this discussion. I have a small question that will >> "scan.filter-push-down.enabled" take precedence over >> "table.optimizer.source.predicate" when the two parameters might conflict >> each other? >> >> Best, >> Jane >> >> On Tue, Oct 24, 2023 at 5:05 PM Jiabao Sun <jiabao....@xtransfer.cn.invalid> >> wrote: >> >>> Thanks Jark, >>> >>> If we only add configuration without adding the enableFilterPushDown >>> method in the SupportsFilterPushDown interface, >>> each connector would have to handle the same logic in the applyFilters >>> method to determine whether filter pushdown is needed. >>> This would increase complexity and violate the original behavior of the >>> applyFilters method. >>> >>> On the contrary, we only need to pass the configuration parameter in the >>> newly added enableFilterPushDown method >>> to decide whether to perform predicate pushdown. >>> >>> I think this approach would be clearer and simpler. >>> WDYT? >>> >>> Best, >>> Jiabao >>> >>> >>>> 2023年10月24日 16:58,Jark Wu <imj...@gmail.com> 写道: >>>> >>>> Hi JIabao, >>>> >>>> I think the current interface can already satisfy your requirements. >>>> The connector can reject all the filters by returning the input filters >>>> as `Result#remainingFilters`. >>>> >>>> So maybe we don't need to introduce a new method to disable >>>> pushdown, but just introduce an option for the specific connector. >>>> >>>> Best, >>>> Jark >>>> >>>> On Tue, 24 Oct 2023 at 16:38, Leonard Xu <xbjt...@gmail.com> wrote: >>>> >>>>> Thanks @Jiabao for kicking off this discussion. >>>>> >>>>> Could you add a section to explain the difference between proposed >>>>> connector level config `scan.filter-push-down.enabled` and existing >>> query >>>>> level config `table.optimizer.source.predicate-pushdown-enabled` ? >>>>> >>>>> Best, >>>>> Leonard >>>>> >>>>>> 2023年10月24日 下午4:18,Jiabao Sun <jiabao....@xtransfer.cn.INVALID> 写道: >>>>>> >>>>>> Hi Devs, >>>>>> >>>>>> I would like to start a discussion on FLIP-377: support configuration >>> to >>>>> disable filter pushdown for Table/SQL Sources[1]. >>>>>> >>>>>> Currently, Flink Table/SQL does not expose fine-grained control for >>>>> users to enable or disable filter pushdown. >>>>>> However, filter pushdown has some side effects, such as additional >>>>> computational pressure on external systems. >>>>>> Moreover, Improper queries can lead to issues such as full table scans, >>>>> which in turn can impact the stability of external systems. >>>>>> >>>>>> Suppose we have an SQL query with two sources: Kafka and a database. >>>>>> The database is sensitive to pressure, and we want to configure it to >>>>> not perform filter pushdown to the database source. >>>>>> However, we still want to perform filter pushdown to the Kafka source >>> to >>>>> decrease network IO. >>>>>> >>>>>> I propose to support configuration to disable filter push down for >>>>> Table/SQL sources to let user decide whether to perform filter pushdown. >>>>>> >>>>>> Looking forward to your feedback. >>>>>> >>>>>> [1] >>>>> >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768 >>>>>> >>>>>> Best, >>>>>> Jiabao >>>>> >>>>> >>> >>>