I agree with Jane that fine-grained configurations should have higher priority than job level configurations.
For current proposal, we can achieve that: - Set "table.optimizer.source.predicate" = "true" to enable by default, and set ""scan.filter-push-down.enabled" = "false" to disable it per table source - Set "table.optimizer.source.predicate" = "false" to disable by default, and set ""scan.filter-push-down.enabled" = "true" to enable it per table source Jane Chan <qingyue....@gmail.com> 于2023年10月24日周二 23:55写道： > > > > > I believe that the configuration "table.optimizer.source.predicate" has a > > higher priority at the planner level than the configuration at the source > > level, > > and it seems easy to implement now. > > > > Correct me if I'm wrong, but I think the fine-grained configuration > "scan.filter-push-down.enabled" should have a higher priority because the > default value of "table.optimizer.source.predicate" is true. As a result, > turning off filter push-down for a specific source will not take effect > unless the default value of "table.optimizer.source.predicate" is changed > to false, or, alternatively, let users manually set > "table.optimizer.source.predicate" to false first and then selectively > enable filter push-down for the desired sources, which is less intuitive. > WDYT? > > Best, > Jane > > On Tue, Oct 24, 2023 at 6:05 PM Jiabao Sun <jiabao....@xtransfer.cn.invalid> > wrote: > > > Thanks Jane, > > > > I believe that the configuration "table.optimizer.source.predicate" has a > > higher priority at the planner level than the configuration at the source > > level, > > and it seems easy to implement now. > > > > Best, > > Jiabao > > > > > > > 2023年10月24日 17:36，Jane Chan <qingyue....@gmail.com> 写道： > > > > > > Hi Jiabao, > > > > > > Thanks for driving this discussion. I have a small question that will > > > "scan.filter-push-down.enabled" take precedence over > > > "table.optimizer.source.predicate" when the two parameters might conflict > > > each other? > > > > > > Best, > > > Jane > > > > > > On Tue, Oct 24, 2023 at 5:05 PM Jiabao Sun <jiabao....@xtransfer.cn > > .invalid> > > > wrote: > > > > > >> Thanks Jark, > > >> > > >> If we only add configuration without adding the enableFilterPushDown > > >> method in the SupportsFilterPushDown interface, > > >> each connector would have to handle the same logic in the applyFilters > > >> method to determine whether filter pushdown is needed. > > >> This would increase complexity and violate the original behavior of the > > >> applyFilters method. > > >> > > >> On the contrary, we only need to pass the configuration parameter in the > > >> newly added enableFilterPushDown method > > >> to decide whether to perform predicate pushdown. > > >> > > >> I think this approach would be clearer and simpler. > > >> WDYT? > > >> > > >> Best, > > >> Jiabao > > >> > > >> > > >>> 2023年10月24日 16:58，Jark Wu <imj...@gmail.com> 写道： > > >>> > > >>> Hi JIabao, > > >>> > > >>> I think the current interface can already satisfy your requirements. > > >>> The connector can reject all the filters by returning the input filters > > >>> as `Result#remainingFilters`. > > >>> > > >>> So maybe we don't need to introduce a new method to disable > > >>> pushdown, but just introduce an option for the specific connector. > > >>> > > >>> Best, > > >>> Jark > > >>> > > >>> On Tue, 24 Oct 2023 at 16:38, Leonard Xu <xbjt...@gmail.com> wrote: > > >>> > > >>>> Thanks @Jiabao for kicking off this discussion. > > >>>> > > >>>> Could you add a section to explain the difference between proposed > > >>>> connector level config `scan.filter-push-down.enabled` and existing > > >> query > > >>>> level config `table.optimizer.source.predicate-pushdown-enabled` ? > > >>>> > > >>>> Best, > > >>>> Leonard > > >>>> > > >>>>> 2023年10月24日 下午4:18，Jiabao Sun <jiabao....@xtransfer.cn.INVALID> 写道： > > >>>>> > > >>>>> Hi Devs, > > >>>>> > > >>>>> I would like to start a discussion on FLIP-377: support configuration > > >> to > > >>>> disable filter pushdown for Table/SQL Sources[1]. > > >>>>> > > >>>>> Currently, Flink Table/SQL does not expose fine-grained control for > > >>>> users to enable or disable filter pushdown. > > >>>>> However, filter pushdown has some side effects, such as additional > > >>>> computational pressure on external systems. > > >>>>> Moreover, Improper queries can lead to issues such as full table > > scans, > > >>>> which in turn can impact the stability of external systems. > > >>>>> > > >>>>> Suppose we have an SQL query with two sources: Kafka and a database. > > >>>>> The database is sensitive to pressure, and we want to configure it to > > >>>> not perform filter pushdown to the database source. > > >>>>> However, we still want to perform filter pushdown to the Kafka source > > >> to > > >>>> decrease network IO. > > >>>>> > > >>>>> I propose to support configuration to disable filter push down for > > >>>> Table/SQL sources to let user decide whether to perform filter > > pushdown. > > >>>>> > > >>>>> Looking forward to your feedback. > > >>>>> > > >>>>> [1] > > >>>> > > >> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768 > > >>>>> > > >>>>> Best, > > >>>>> Jiabao > > >>>> > > >>>> > > >> > > >> > > > > -- Best, Benchao Li
