Just like our discussion in thread
https://lists.apache.org/thread/whh75f6rtwdyqxt47gb39j6m6m0cpphq , +1 for this
Flip.
--
Best！
Xuyang
在 2023-10-24 18:03:36，"Jiabao Sun" <[email protected]> 写道：
>Thanks Jane,
>
>I believe that the configuration "table.optimizer.source.predicate" has a
>higher priority at the planner level than the configuration at the source
>level,
>and it seems easy to implement now.
>
>Best,
>Jiabao
>
>
>> 2023年10月24日 17:36，Jane Chan <[email protected]> 写道：
>>
>> Hi Jiabao,
>>
>> Thanks for driving this discussion. I have a small question that will
>> "scan.filter-push-down.enabled" take precedence over
>> "table.optimizer.source.predicate" when the two parameters might conflict
>> each other?
>>
>> Best,
>> Jane
>>
>> On Tue, Oct 24, 2023 at 5:05 PM Jiabao Sun <[email protected]>
>> wrote:
>>
>>> Thanks Jark,
>>>
>>> If we only add configuration without adding the enableFilterPushDown
>>> method in the SupportsFilterPushDown interface,
>>> each connector would have to handle the same logic in the applyFilters
>>> method to determine whether filter pushdown is needed.
>>> This would increase complexity and violate the original behavior of the
>>> applyFilters method.
>>>
>>> On the contrary, we only need to pass the configuration parameter in the
>>> newly added enableFilterPushDown method
>>> to decide whether to perform predicate pushdown.
>>>
>>> I think this approach would be clearer and simpler.
>>> WDYT?
>>>
>>> Best,
>>> Jiabao
>>>
>>>
>>>> 2023年10月24日 16:58，Jark Wu <[email protected]> 写道：
>>>>
>>>> Hi JIabao,
>>>>
>>>> I think the current interface can already satisfy your requirements.
>>>> The connector can reject all the filters by returning the input filters
>>>> as `Result#remainingFilters`.
>>>>
>>>> So maybe we don't need to introduce a new method to disable
>>>> pushdown, but just introduce an option for the specific connector.
>>>>
>>>> Best,
>>>> Jark
>>>>
>>>> On Tue, 24 Oct 2023 at 16:38, Leonard Xu <[email protected]> wrote:
>>>>
>>>>> Thanks @Jiabao for kicking off this discussion.
>>>>>
>>>>> Could you add a section to explain the difference between proposed
>>>>> connector level config `scan.filter-push-down.enabled` and existing
>>> query
>>>>> level config `table.optimizer.source.predicate-pushdown-enabled` ?
>>>>>
>>>>> Best,
>>>>> Leonard
>>>>>
>>>>>> 2023年10月24日 下午4:18，Jiabao Sun <[email protected]> 写道：
>>>>>>
>>>>>> Hi Devs,
>>>>>>
>>>>>> I would like to start a discussion on FLIP-377: support configuration
>>> to
>>>>> disable filter pushdown for Table/SQL Sources[1].
>>>>>>
>>>>>> Currently, Flink Table/SQL does not expose fine-grained control for
>>>>> users to enable or disable filter pushdown.
>>>>>> However, filter pushdown has some side effects, such as additional
>>>>> computational pressure on external systems.
>>>>>> Moreover, Improper queries can lead to issues such as full table scans,
>>>>> which in turn can impact the stability of external systems.
>>>>>>
>>>>>> Suppose we have an SQL query with two sources: Kafka and a database.
>>>>>> The database is sensitive to pressure, and we want to configure it to
>>>>> not perform filter pushdown to the database source.
>>>>>> However, we still want to perform filter pushdown to the Kafka source
>>> to
>>>>> decrease network IO.
>>>>>>
>>>>>> I propose to support configuration to disable filter push down for
>>>>> Table/SQL sources to let user decide whether to perform filter pushdown.
>>>>>>
>>>>>> Looking forward to your feedback.
>>>>>>
>>>>>> [1]
>>>>>
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768
>>>>>>
>>>>>> Best,
>>>>>> Jiabao
>>>>>
>>>>>
>>>
>>>
