Hi Jiabao, Thanks for driving this discussion. I have a small question that will "scan.filter-push-down.enabled" take precedence over "table.optimizer.source.predicate" when the two parameters might conflict each other?
Best, Jane On Tue, Oct 24, 2023 at 5:05 PM Jiabao Sun <jiabao....@xtransfer.cn.invalid> wrote: > Thanks Jark, > > If we only add configuration without adding the enableFilterPushDown > method in the SupportsFilterPushDown interface, > each connector would have to handle the same logic in the applyFilters > method to determine whether filter pushdown is needed. > This would increase complexity and violate the original behavior of the > applyFilters method. > > On the contrary, we only need to pass the configuration parameter in the > newly added enableFilterPushDown method > to decide whether to perform predicate pushdown. > > I think this approach would be clearer and simpler. > WDYT? > > Best, > Jiabao > > > > 2023年10月24日 16:58，Jark Wu <imj...@gmail.com> 写道： > > > > Hi JIabao, > > > > I think the current interface can already satisfy your requirements. > > The connector can reject all the filters by returning the input filters > > as `Result#remainingFilters`. > > > > So maybe we don't need to introduce a new method to disable > > pushdown, but just introduce an option for the specific connector. > > > > Best, > > Jark > > > > On Tue, 24 Oct 2023 at 16:38, Leonard Xu <xbjt...@gmail.com> wrote: > > > >> Thanks @Jiabao for kicking off this discussion. > >> > >> Could you add a section to explain the difference between proposed > >> connector level config `scan.filter-push-down.enabled` and existing > query > >> level config `table.optimizer.source.predicate-pushdown-enabled` ? > >> > >> Best, > >> Leonard > >> > >>> 2023年10月24日 下午4:18，Jiabao Sun <jiabao....@xtransfer.cn.INVALID> 写道： > >>> > >>> Hi Devs, > >>> > >>> I would like to start a discussion on FLIP-377: support configuration > to > >> disable filter pushdown for Table/SQL Sources[1]. > >>> > >>> Currently, Flink Table/SQL does not expose fine-grained control for > >> users to enable or disable filter pushdown. > >>> However, filter pushdown has some side effects, such as additional > >> computational pressure on external systems. > >>> Moreover, Improper queries can lead to issues such as full table scans, > >> which in turn can impact the stability of external systems. > >>> > >>> Suppose we have an SQL query with two sources: Kafka and a database. > >>> The database is sensitive to pressure, and we want to configure it to > >> not perform filter pushdown to the database source. > >>> However, we still want to perform filter pushdown to the Kafka source > to > >> decrease network IO. > >>> > >>> I propose to support configuration to disable filter push down for > >> Table/SQL sources to let user decide whether to perform filter pushdown. > >>> > >>> Looking forward to your feedback. > >>> > >>> [1] > >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768 > >>> > >>> Best, > >>> Jiabao > >> > >> > >
