Thanks Becket, The behavior description has been added to the Public Interfaces section.
Best, Jiabao > 2023年12月20日 08:17，Becket Qin <becket....@gmail.com> 写道： > > Hi Jiabao, > > Thanks for updating the FLIP. > Can you add the behavior of the policies that are only applicable to some > but not all of the databases? This is a part of the intended behavior of > the proposed configuration. So, we should include that in the FLIP. > Otherwise, the FLIP looks good to me. > > Cheers, > > Jiangjie (Becket) Qin > > On Tue, Dec 19, 2023 at 11:00 PM Jiabao Sun <jiabao....@xtransfer.cn.invalid> > wrote: > >> Hi Becket, >> >> I share the same view as you regarding the prefix for this configuration >> option. >> >> For the JDBC connector, I prefer setting 'filter.handling.policy' = 'FOO' >> and throwing an exception when the database do not support that specific >> policy. >> >> Not using a prefix can reduce the learning curve for users and avoid >> introducing a new set of configuration options for every supported JDBC >> database. >> I think the policies we provide can be compatible with most databases that >> follow the JDBC protocol. >> However, there may be cases where certain databases cannot support some >> policies. >> Nevertheless, we can ensure fast failure and allow users to choose a >> suitable policy in such situations. >> >> I have removed the contents about the configuration prefix. >> Please help review it again. >> >> Thanks, >> Jiabao >> >> >>> 2023年12月19日 19:46，Becket Qin <becket....@gmail.com> 写道： >>> >>> Hi Jiabao, >>> >>> Thanks for updating the FLIP. >>> >>> One more question regarding the JDBC connector, since it is a connector >>> shared by multiple databases, what if there is a filter handling policy >>> that is only applicable to one of the databases, but not the others? In >>> that case, how would the users specify that policy? >>> Unlike the example of orc format with 2nd+ level config, JDBC connector >>> only looks at the URL to decide which driver to use. >>> >>> For example, MySql supports policy FOO while other databases do not. If >>> users want to use FOO for MySql, what should they do? Will they set >>> '*mysql.filter.hanlding.policy' >>> = 'FOO', *which will only be picked up when the MySql driver is used? >>> Or they should just set* 'filter.handling.policy' = 'FOO', *and throw >>> exceptions when other JDBC drivers are used? Personally, I prefer the >>> latter. If we pick that, do we still need to mention the following? >>> >>> *The prefix is needed when the option is for a 2nd+ level. * >>>> 'connector' = 'filesystem', >>>> 'format' = 'orc', >>>> 'orc.filter.handling.policy' = 'NUBERIC_ONY' >>>> >>>> *In this case, the values of this configuration may be different >> depending >>>> on the format option. For example, orc format may have INDEXED_ONLY >> while >>>> parquet format may have something else. * >>>> >>> >>> I found this is somewhat misleading, because the example here is not a >> part >>> of the proposal of this FLIP. It is just an example explaining when a >>> prefix is needed, which seems orthogonal to the proposal in this FLIP. >>> >>> Thanks, >>> >>> Jiangjie (Becket) Qin >>> >>> >>> On Tue, Dec 19, 2023 at 10:09 AM Jiabao Sun <jiabao....@xtransfer.cn >> .invalid> >>> wrote: >>> >>>> Thanks Becket for the suggestions, >>>> >>>> Updated. >>>> Please help review it again when you have time. >>>> >>>> Best, >>>> Jiabao >>>> >>>> >>>>> 2023年12月19日 09:06，Becket Qin <becket....@gmail.com> 写道： >>>>> >>>>> Hi JIabao, >>>>> >>>>> Thanks for updating the FLIP. It looks better. Some suggestions / >>>> questions: >>>>> >>>>> 1. In the motivation section: >>>>> >>>>>> *Currently, Flink Table/SQL does not expose fine-grained control for >>>> users >>>>>> to control filter pushdown. **However, filter pushdown has some side >>>>>> effects, such as additional computational pressure on external >>>>>> systems. Moreover, Improper queries can lead to issues such as full >>>> table >>>>>> scans, which in turn can impact the stability of external systems.* >>>>> >>>>> This statement sounds like the side effects are there for all the >>>> systems, >>>>> which is inaccurate. Maybe we can say: >>>>> *Currently, Flink Table/SQL does not expose fine-grained control for >>>> users >>>>> to control filter pushdown. **However, filter pushdown may have side >>>>> effects in some cases, **such as additional computational pressure on >>>>> external systems. The JDBC source is a typical example of that. If a >>>> filter >>>>> is pushed down to the database, an expensive full table scan may happen >>>> if >>>>> the filter involves unindexed columns.* >>>>> >>>>> 2. Regarding the prefix, usually a prefix is not required for the top >>>> level >>>>> connector options. This is because the *connector* option is already >>>> there. >>>>> So >>>>> 'connector' = 'jdbc', >>>>> 'filter.handling.policy' = 'ALWAYS' >>>>> is sufficient. >>>>> >>>>> The prefix is needed when the option is for a 2nd+ level. For example, >>>>> 'connector' = 'jdbc', >>>>> 'format' = 'orc', >>>>> 'orc.some.option' = 'some_value' >>>>> In this case, the prefix of "orc" is needed to make it clear this >> option >>>> is >>>>> for the format. >>>>> >>>>> I am guessing that the reason that currently the connector prefix is >>>> there >>>>> is because the values of this configuration may be different depending >> on >>>>> the connectors. For example, jdbc may have INDEXED_ONLY while MongoDB >> may >>>>> have something else. Personally speaking, I am fine if we do not have a >>>>> prefix in this case because users have already specified the connector >>>> type >>>>> and it is intuitive enough that the option value is for that connector, >>>> not >>>>> others. >>>>> >>>>> 3. can we clarify on the following statement: >>>>> >>>>>> *Introduce the native configuration [prefix].filter.handling.policy in >>>> the >>>>>> connector.* >>>>> >>>>> What do you mean by "native configuration"? From what I understand, the >>>>> FLIP does the following: >>>>> - introduces a new configuration to the JDBC and MongoDB connector. >>>>> - Suggests a convention option name if other connectors are going to >> add >>>> an >>>>> option for the same purpose. >>>>> >>>>> Thanks, >>>>> >>>>> Jiangjie (Becket) Qin >>>>> >>>>> >>>>> >>>>> On Mon, Dec 18, 2023 at 5:45 PM Jiabao Sun <jiabao....@xtransfer.cn >>>> .invalid> >>>>> wrote: >>>>> >>>>>> Hi Becket, >>>>>> >>>>>> The FLIP document[1] has been updated. >>>>>> Could you help take a look again? >>>>>> >>>>>> Thanks, >>>>>> Jiabao >>>>>> >>>>>> [1] >>>>>> >>>> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768 >>>>>> >>>>>> >>>>>>> 2023年12月18日 16:53，Becket Qin <becket....@gmail.com> 写道： >>>>>>> >>>>>>> Yes, that sounds reasonable to me. We can start with ALWAYS and >> NEVER, >>>>>> and >>>>>>> add more policies as needed. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Jiangjie (Becket) Qin >>>>>>> >>>>>>> On Mon, Dec 18, 2023 at 4:48 PM Jiabao Sun <jiabao....@xtransfer.cn >>>>>> .invalid> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks Bucket, >>>>>>>> >>>>>>>> The jdbc.filter.handling.policy is good to me as it provides >>>> sufficient >>>>>>>> extensibility for future filter pushdown optimizations. >>>>>>>> However, currently, we don't have an implementation for the AUTO >> mode, >>>>>> and >>>>>>>> it seems that the AUTO mode can easily be confused with the ALWAYS >>>> mode >>>>>>>> because users don't have the opportunity to MANUALLY decide which >>>>>> filters >>>>>>>> to push down. >>>>>>>> >>>>>>>> I suggest that we only introduce the ALWAYS and NEVER modes for now, >>>> and >>>>>>>> we can consider introducing more flexible policies in the future, >>>>>>>> such as INDEX_ONLY, NUMBERIC_ONLY and so on. >>>>>>>> >>>>>>>> WDYT? >>>>>>>> >>>>>>>> Best, >>>>>>>> Jiabao >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> 2023年12月18日 16:27，Becket Qin <becket....@gmail.com> 写道： >>>>>>>>> >>>>>>>>> Hi Jiabao, >>>>>>>>> >>>>>>>>> Please see the reply inline. >>>>>>>>> >>>>>>>>> >>>>>>>>>> The MySQL connector is currently in the flink-connector-jdbc >>>>>> repository >>>>>>>>>> and is not a standalone connector. >>>>>>>>>> Is it too unique to use "mysql" as the configuration option >> prefix? >>>>>>>>> >>>>>>>>> If the intended behavior makes sense to all the supported JDBC >>>> drivers, >>>>>>>> we >>>>>>>>> can make this a JDBC connector configuration. >>>>>>>>> >>>>>>>>> Also, I would like to ask about the difference in behavior between >>>> AUTO >>>>>>>> and >>>>>>>>>> ALWAYS. >>>>>>>>>> It seems that we cannot guarantee the pushing down of all filters >> to >>>>>> the >>>>>>>>>> external system under the ALWAYS >>>>>>>>>> mode because not all filters in Flink SQL are supported by the >>>>>> external >>>>>>>>>> system. >>>>>>>>>> Should we throw an error when encountering a filter that cannot be >>>>>>>> pushed >>>>>>>>>> down in the ALWAYS mode? >>>>>>>>> >>>>>>>>> The idea of AUTO is to do efficiency-aware pushdowns. The source >> will >>>>>>>> query >>>>>>>>> the external system (MySQL, Oracle, SQL Server, etc) first to >>>> retrieve >>>>>>>> the >>>>>>>>> information of the table. With that information, the source will >>>> decide >>>>>>>>> whether to further push a filter to the external system based on >> the >>>>>>>>> efficiency. E.g. only push the indexed fields. In contrast, ALWAYS >>>> will >>>>>>>>> just always push the supported filters to the external system, >>>>>> regardless >>>>>>>>> of the efficiency. In case there are filters that are not >> supported, >>>>>>>>> according to the current contract of SupportsFilterPushdown, these >>>>>>>>> unsupported filters should just be returned by the >>>>>>>>> *SupportsFilterPushdown.applyFilters()* method as remaining >> filters. >>>>>>>>> Therefore, there is no need to throw exceptions here. This is >> likely >>>>>> the >>>>>>>>> desired behavior for most users, IMO. If there are cases that users >>>>>>>> really >>>>>>>>> want to get alerted when a filter cannot be pushed to the external >>>>>>>> system, >>>>>>>>> we can add another value like "ENFORCED_ALWAYS", which behaves like >>>>>>>> ALWAYS, >>>>>>>>> but throws exceptions when a filter cannot be applied to the >> external >>>>>>>>> system. But personally I don't see much value in doing this. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Dec 18, 2023 at 3:54 PM Jiabao Sun < >> jiabao....@xtransfer.cn >>>>>>>> .invalid> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Becket, >>>>>>>>>> >>>>>>>>>> The MySQL connector is currently in the flink-connector-jdbc >>>>>> repository >>>>>>>>>> and is not a standalone connector. >>>>>>>>>> Is it too unique to use "mysql" as the configuration option >> prefix? >>>>>>>>>> >>>>>>>>>> Also, I would like to ask about the difference in behavior between >>>>>> AUTO >>>>>>>>>> and ALWAYS. >>>>>>>>>> It seems that we cannot guarantee the pushing down of all filters >> to >>>>>> the >>>>>>>>>> external system under the ALWAYS >>>>>>>>>> mode because not all filters in Flink SQL are supported by the >>>>>> external >>>>>>>>>> system. >>>>>>>>>> Should we throw an error when encountering a filter that cannot be >>>>>>>> pushed >>>>>>>>>> down in the ALWAYS mode? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Jiabao >>>>>>>>>> >>>>>>>>>>> 2023年12月18日 15:34，Becket Qin <becket....@gmail.com> 写道： >>>>>>>>>>> >>>>>>>>>>> Hi JIabao, >>>>>>>>>>> >>>>>>>>>>> Thanks for updating the FLIP. Maybe I did not explain it clearly >>>>>>>> enough. >>>>>>>>>> My >>>>>>>>>>> point is that given there are various good flavors of behaviors >>>>>>>> handling >>>>>>>>>>> filters pushed down, we should not have a common config of >>>>>>>>>>> "ignore.filter.pushdown", because the behavior is not *common*. >>>>>>>>>>> >>>>>>>>>>> It looks like the original motivation of this FLIP is just for >>>> MySql. >>>>>>>>>> Let's >>>>>>>>>>> focus on what is the best solution for MySql connector here >> first. >>>>>>>> After >>>>>>>>>>> that, if people think the best behavior for MySql happens to be a >>>>>>>> common >>>>>>>>>>> one, we can then discuss whether that is worth being added to the >>>>>> base >>>>>>>>>>> implementation of source. For MySQL, if we are going to >> introduce a >>>>>>>>>> config >>>>>>>>>>> to MySql, why not have something like >>>> "mysql.filter.handling.policy" >>>>>>>> with >>>>>>>>>>> value of AUTO / NEVER / ALWAYS? Isn't that better than >>>>>>>>>>> "ignore.filter.pushdown"? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sun, Dec 17, 2023 at 11:30 PM Jiabao Sun < >>>> jiabao....@xtransfer.cn >>>>>>>>>> .invalid> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Becket, >>>>>>>>>>>> >>>>>>>>>>>> The FLIP document has been updated as well. >>>>>>>>>>>> Please take a look when you have time. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Jiabao >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> 2023年12月17日 22:54，Jiabao Sun <jiabao....@xtransfer.cn.INVALID> >>>> 写道： >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks Becket, >>>>>>>>>>>>> >>>>>>>>>>>>> I apologize for not being able to continue with this proposal >> due >>>>>> to >>>>>>>>>>>> being too busy during this period. >>>>>>>>>>>>> >>>>>>>>>>>>> The viewpoints you shared about the design of Flink Source make >>>>>> sense >>>>>>>>>> to >>>>>>>>>>>> me >>>>>>>>>>>>> The native configuration ‘ignore.filter.pushdown’ is good to >> me. >>>>>>>>>>>>> Having a unified name or naming style can indeed prevent >>>> confusion >>>>>>>> for >>>>>>>>>>>> users regarding >>>>>>>>>>>>> the inconsistent naming of this configuration across different >>>>>>>>>>>> connectors. >>>>>>>>>>>>> >>>>>>>>>>>>> Currently, there are not many external connectors that support >>>>>> filter >>>>>>>>>>>> pushdown. >>>>>>>>>>>>> I propose that we first introduce it in flink-connector-jdbc >> and >>>>>>>>>>>> flink-connector-mongodb. >>>>>>>>>>>>> Do you think this is feasible? >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> Jiabao >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> 2023年11月16日 17:45，Becket Qin <becket....@gmail.com> 写道： >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Arguments like "because Spark has it so Flink should also have >>>> it" >>>>>>>>>> does >>>>>>>>>>>> not >>>>>>>>>>>>>> make sense. Different projects have different API flavors and >>>>>>>> styles. >>>>>>>>>>>> What >>>>>>>>>>>>>> is really important is the rationale and the design principle >>>>>> behind >>>>>>>>>> the >>>>>>>>>>>>>> API. They should conform to the convention of the project. >>>>>>>>>>>>>> >>>>>>>>>>>>>> First of all, Spark Source API itself has a few issues and >> they >>>>>>>> ended >>>>>>>>>> up >>>>>>>>>>>>>> introduce DataSource V2 in Spark 3.0, which added the >> decorative >>>>>>>>>>>> interfaces >>>>>>>>>>>>>> like SupportsPushdownXXX. Some of the configurations predating >>>>>>>>>>>> DataSource >>>>>>>>>>>>>> V2 may still be there. >>>>>>>>>>>>>> >>>>>>>>>>>>>> For the Spark configurations you mentioned, they are all the >>>>>>>>>>>> configurations >>>>>>>>>>>>>> for FileScanBuilder, which is equivalent to FileSource in >> Flink. >>>>>>>>>>>> Currently, >>>>>>>>>>>>>> regardless of the format (ORC, Parquet, Avro, etc), the >>>> FileSource >>>>>>>>>>>> pushes >>>>>>>>>>>>>> back all the filters to ensure correctness. The actual filters >>>>>> that >>>>>>>>>> got >>>>>>>>>>>>>> applied to the specific format might still be different. This >>>>>>>>>>>>>> implementation is the same in FileScanBuilder.pushFilters() >> for >>>>>>>>>> Spark. I >>>>>>>>>>>>>> don't know why Spark got separate configurations for each >>>> format. >>>>>>>>>> Maybe >>>>>>>>>>>> it >>>>>>>>>>>>>> is because the filters are actually implemented differently >> for >>>>>>>>>>>> different >>>>>>>>>>>>>> format. >>>>>>>>>>>>>> >>>>>>>>>>>>>> At least for the current implementation in FileScanBuilder, >>>> these >>>>>>>>>>>>>> configurations can be merged to one configuration like >>>>>>>>>>>>>> `apply.filters.to.format.enabled`. Note that this config, as >>>> well >>>>>> as >>>>>>>>>> the >>>>>>>>>>>>>> separate configs you mentioned, are just visible and used by >> the >>>>>>>>>>>>>> FileScanBuilder. It determines whether the filters should be >>>>>> passed >>>>>>>>>>>> down to >>>>>>>>>>>>>> the format of the FileScanBuilder instance. Regardless of the >>>>>> value >>>>>>>> of >>>>>>>>>>>>>> these configs, FileScanBuilder.pushFilters() will always be >>>>>> called, >>>>>>>>>> and >>>>>>>>>>>>>> FileScanBuilder (as well as FileSource in Flink) will always >>>> push >>>>>>>> back >>>>>>>>>>>> all >>>>>>>>>>>>>> the filters to the framework. >>>>>>>>>>>>>> >>>>>>>>>>>>>> A MySql source can have a very different way to handle this. >> For >>>>>>>>>>>> example, A >>>>>>>>>>>>>> MySql source A config in this case might be >> "my.apply.filters" >>>>>> with >>>>>>>>>>>> three >>>>>>>>>>>>>> different values: >>>>>>>>>>>>>> - AUTO: The Source will issue a DESC Table request to >> understand >>>>>>>>>>>> whether a >>>>>>>>>>>>>> filter can be applied efficiently. And decide which filters >> can >>>> be >>>>>>>>>>>> applied >>>>>>>>>>>>>> and which cannot based on that. >>>>>>>>>>>>>> - NEVER: Never apply filtering. It will always do a full table >>>>>> read >>>>>>>>>> and >>>>>>>>>>>>>> let Flink do the filtering. >>>>>>>>>>>>>> - ALWAYS: Always apply the filtering to the MySql server. >>>>>>>>>>>>>> >>>>>>>>>>>>>> In the above examples of FileSource and MySql Source, I don't >>>>>> think >>>>>>>> it >>>>>>>>>>>> is a >>>>>>>>>>>>>> good idea to shoehorn the behaviors into a naive config of >>>>>>>>>>>>>> `ignore.filter.pushdown`. That is why I don't think this is a >>>>>> common >>>>>>>>>>>> config. >>>>>>>>>>>>>> >>>>>>>>>>>>>> To recap, like I said, I do agree that in some cases, we may >>>> want >>>>>> to >>>>>>>>>>>> behave >>>>>>>>>>>>>> differently when filters are pushed down to the sources, even >>>> if a >>>>>>>>>>>> source >>>>>>>>>>>>>> implements SupportsFilterPushDown, but I don't think there is >> a >>>>>>>>>> suitable >>>>>>>>>>>>>> common config for this. The behavior is very likely source >>>>>> specific. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Nov 16, 2023 at 3:41 PM Jiabao Sun < >>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>> .invalid> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks Becket, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I still believe that adding a configuration at the source >> level >>>>>> to >>>>>>>>>>>> disable >>>>>>>>>>>>>>> filter pushdown is needed. This demand exists in spark as >>>>>> well[1]. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In Spark, most sources that support filter pushdown provide >>>> their >>>>>>>> own >>>>>>>>>>>>>>> corresponding configuration options to enable or disable >> filter >>>>>>>>>>>> pushdown. >>>>>>>>>>>>>>> For PRs[2-4] that support filter pushdown capability, they >> also >>>>>>>>>> provide >>>>>>>>>>>>>>> configuration options to disable this capability. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I believe this configuration is applicable to most scenarios, >>>> and >>>>>>>>>>>> there is >>>>>>>>>>>>>>> no need to dwell on why this configuration option was not >>>>>>>> introduced >>>>>>>>>>>>>>> earlier than the SupportsFilterPushDown interface. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> spark.sql.parquet.filterPushdown >>>>>>>>>>>>>>> spark.sql.orc.filterPushdown >>>>>>>>>>>>>>> spark.sql.csv.filterPushdown.enabled >>>>>>>>>>>>>>> spark.sql.json.filterPushdown.enabled >>>>>>>>>>>>>>> spark.sql.avro.filterPushdown.enabled >>>>>>>>>>>>>>> JDBC Option: pushDownPredicate >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We can see that the lack of consistency is caused by each >>>>>> connector >>>>>>>>>>>>>>> introducing different configuration options for the same >>>>>> behavior. >>>>>>>>>>>>>>> This is one of the motivations for advocating the >> introduction >>>>>> of a >>>>>>>>>>>>>>> unified configuration name. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/SPARK-24288 >>>>>>>>>>>>>>> [2] https://github.com/apache/spark/pull/27366 >>>>>>>>>>>>>>> [3]https://github.com/apache/spark/pull/26973 >>>>>>>>>>>>>>> [4] https://github.com/apache/spark/pull/29145 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2023年11月16日 08:10，Becket Qin <becket....@gmail.com> 写道： >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> While we can always fix the formality of the config, a more >>>>>>>>>>>> fundamental >>>>>>>>>>>>>>>> issue here is whether this configuration is common enough. >>>>>>>>>> Personally >>>>>>>>>>>> I >>>>>>>>>>>>>>> am >>>>>>>>>>>>>>>> still not convinced it is. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Remember we don't have a common implementation for >>>>>>>>>>>> SupportsFilterPushdown >>>>>>>>>>>>>>>> itself. Why does a potential behavior of the >>>>>>>>>>>>>>>> SupportsFilterPushdown.applyFilters() method deserve a >> common >>>>>>>>>>>>>>>> configuration? A common implementation should always come >>>> first, >>>>>>>>>> then >>>>>>>>>>>> its >>>>>>>>>>>>>>>> configuration becomes a common configuration as a natural >>>>>> result. >>>>>>>>>> But >>>>>>>>>>>>>>> here >>>>>>>>>>>>>>>> we are trying to add an impl to a configuration just to fix >>>> its >>>>>>>>>>>>>>> formality. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I agree that there might be a few Source implementations >> that >>>>>> may >>>>>>>>>>>> want to >>>>>>>>>>>>>>>> avoid additional burdens on the remote system in some >>>>>>>> circumstances. >>>>>>>>>>>> And >>>>>>>>>>>>>>>> these circumstances are very specific: >>>>>>>>>>>>>>>> 1. The source talks to a remote service that can help >> perform >>>>>> the >>>>>>>>>>>> actual >>>>>>>>>>>>>>>> filtering. >>>>>>>>>>>>>>>> 2. The filtering done by the remote service is inefficient >> for >>>>>>>> some >>>>>>>>>>>>>>> reason >>>>>>>>>>>>>>>> (e.g. missing index) >>>>>>>>>>>>>>>> 3. The external service does not want to perform the >>>> inefficient >>>>>>>>>>>>>>> filtering >>>>>>>>>>>>>>>> for some reason (e.g. it is a shared service with others) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There are multiple approaches to address the issue. Pushing >>>> back >>>>>>>> the >>>>>>>>>>>>>>>> filters is just one way of achieving this. So here we are >>>>>> talking >>>>>>>>>>>> about a >>>>>>>>>>>>>>>> config for one of the possible solutions to a scenario with >>>> all >>>>>>>> the >>>>>>>>>>>> above >>>>>>>>>>>>>>>> situations. I don't think there is enough justification for >>>> the >>>>>>>>>>>> config to >>>>>>>>>>>>>>>> be common. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There is always this trade-off between the proliferation of >>>>>> public >>>>>>>>>>>>>>>> interfaces and the API standardization. As an extreme >> example, >>>>>> we >>>>>>>>>> can >>>>>>>>>>>>>>> make >>>>>>>>>>>>>>>> our public API a union of all the configs potentially used >> in >>>>>> all >>>>>>>>>> the >>>>>>>>>>>>>>> cases >>>>>>>>>>>>>>>> in the name of standardization. Apparently this won't work. >> So >>>>>>>> there >>>>>>>>>>>> must >>>>>>>>>>>>>>>> be a bar here and this bar might be somewhat subjective. For >>>>>> this >>>>>>>>>>>> FLIP, >>>>>>>>>>>>>>>> personally I don't think the config meets my bar for the >>>> reason >>>>>>>>>> stated >>>>>>>>>>>>>>>> above. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Therefore, my suggestion remains the same. Keep the config >> as >>>> a >>>>>>>>>> Source >>>>>>>>>>>>>>>> implementation specific configuration. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, Nov 16, 2023 at 12:36 AM Jiabao Sun < >>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks Becket for the feedback, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regarding concerns about common configurations, I think we >>>> can >>>>>>>>>>>> introduce >>>>>>>>>>>>>>>>> FiltersApplier to unify the behavior of various connectors. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> public static class FiltersApplier { >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> private final ReadableConfig config; >>>>>>>>>>>>>>>>> private final Function<List<ResolvedExpression>, Result> >>>>>> action; >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> private FiltersApplier( >>>>>>>>>>>>>>>>> ReadableConfig config, >>>>>>>>>>>>>>>>> Function<List<ResolvedExpression>, Result> action) { >>>>>>>>>>>>>>>>> this.config = config; >>>>>>>>>>>>>>>>> this.action = action; >>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> public Result applyFilters(List<ResolvedExpression> >> filters) >>>> { >>>>>>>>>>>>>>>>> if (config.get(ENABLE_FILTER_PUSH_DOWN)) { >>>>>>>>>>>>>>>>> return action.apply(filters); >>>>>>>>>>>>>>>>> } else { >>>>>>>>>>>>>>>>> return Result.of(Collections.emptyList(), filters); >>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> public static FiltersApplier of( >>>>>>>>>>>>>>>>> ReadableConfig config, >>>>>>>>>>>>>>>>> Function<List<ResolvedExpression>, Result> action) { >>>>>>>>>>>>>>>>> return new FiltersApplier(config, action); >>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> For connectors implementation: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> @Override >>>>>>>>>>>>>>>>> public Result applyFilters(List<ResolvedExpression> >> filters) >>>> { >>>>>>>>>>>>>>>>> return FiltersApplier.of(config, >>>>>>>>>>>>>>>>> f -> Result.of(new ArrayList<>(filters), >>>>>>>>>>>>>>>>> Collections.emptyList())); >>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> As for the name, whether it is >>>>>> "source.filter-push-down.enabled" >>>>>>>> or >>>>>>>>>>>>>>>>> "source.ignore-pushed-down-filters.enabled", I think both >> are >>>>>>>> okay. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Do you think this change is feasible? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2023年11月15日 23:44，Becket Qin <becket....@gmail.com> 写道： >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Yes, I still have concerns. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The FLIP violates the following two principles regarding >>>>>>>>>>>> configuration: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 1.* A config of a class should never negate the semantic >> of >>>> a >>>>>>>>>>>>>>> decorative >>>>>>>>>>>>>>>>>> interface implemented by that class. * >>>>>>>>>>>>>>>>>> A decorative interface is a public contract with other >>>>>>>> components, >>>>>>>>>>>>>>> while >>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>> config is only internal to the class itself. The >>>>>> configurations >>>>>>>>>> for >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> Sources are not (and should never be) visible or >> understood >>>> to >>>>>>>>>>>>>>>>>> other components (e.g. optimizer). A configuration of a >>>> Source >>>>>>>>>> only >>>>>>>>>>>>>>>>>> controls the behavior of that Source, provided it is not >>>>>>>> violating >>>>>>>>>>>> the >>>>>>>>>>>>>>>>> API >>>>>>>>>>>>>>>>>> contract / semantic defined by the decorative interface. >> So >>>>>>>> when a >>>>>>>>>>>>>>> Source >>>>>>>>>>>>>>>>>> implementation implements SupportsFilterPushdown, this is >> a >>>>>>>> clear >>>>>>>>>>>>>>> public >>>>>>>>>>>>>>>>>> contract with Flink that filters should be pushed down to >>>> that >>>>>>>>>>>> Source. >>>>>>>>>>>>>>>>>> Therefore, for the same source, there should not be a >>>>>>>>>> configuration >>>>>>>>>>>>>>>>>> "source.filter-push-down.enabled" which stops the filters >>>> from >>>>>>>>>> being >>>>>>>>>>>>>>>>> pushed >>>>>>>>>>>>>>>>>> down to that Source. However, that specific source >>>>>>>> implementation >>>>>>>>>>>> can >>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>> its own config to control its internal behavior, e.g. >>>>>>>>>>>>>>>>>> "ignore-pushed-down-filters.enabled" which may push back >> all >>>>>> the >>>>>>>>>>>> pushed >>>>>>>>>>>>>>>>>> down filters back to the Flink optimizer. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2. When we are talking about "common configs", in fact we >>>> are >>>>>>>>>>>> talking >>>>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>> "configs for common (abstract) implementation classes". >> With >>>>>>>> that >>>>>>>>>>>> as a >>>>>>>>>>>>>>>>>> context, *a common config should always be backed by a >>>> common >>>>>>>>>>>>>>>>>> implementation class, so that consistent behavior can be >>>>>>>>>>>> guaranteed. * >>>>>>>>>>>>>>>>>> The LookupOptions you mentioned are configurations defined >>>> for >>>>>>>>>>>> classes >>>>>>>>>>>>>>>>>> DefaultLookupCache / PeriodicCacheReloadTrigger / >>>>>>>>>>>>>>>>> TimedCacheReloadTrigger. >>>>>>>>>>>>>>>>>> These configs are considered as "common" only because the >>>>>>>>>>>>>>> implementation >>>>>>>>>>>>>>>>>> classes using them are common building blocks for lookup >>>> table >>>>>>>>>>>>>>>>>> implementations. It would not make sense to have a >> dangling >>>>>>>> config >>>>>>>>>>>> in >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> LookupOptions without the underlying common implementation >>>>>>>> class, >>>>>>>>>>>> but >>>>>>>>>>>>>>>>> only >>>>>>>>>>>>>>>>>> relies on a specific source to implement the stated >>>> behavior. >>>>>>>>>>>>>>>>>> As a bad example, there is this outlier config >> "max-retries" >>>>>> in >>>>>>>>>>>>>>>>>> LookupOptions, which I don't think should be here. This is >>>>>>>> because >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> retry behavior can be very implementation specific. For >>>>>> example, >>>>>>>>>>>> there >>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>> be many different flavors of retry related configurations, >>>>>>>>>>>>>>> retry-backoff, >>>>>>>>>>>>>>>>>> retry-timeout, retry-async, etc. Why only max-retry is put >>>>>> here? >>>>>>>>>>>> should >>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>> of them be put here? If we put all such kinds of configs >> in >>>>>> the >>>>>>>>>>>> common >>>>>>>>>>>>>>>>>> configs for "standardization and unification", the number >> of >>>>>>>>>> "common >>>>>>>>>>>>>>>>>> configs" can easily go crazy. And I don't see material >>>>>> benefits >>>>>>>> of >>>>>>>>>>>>>>> doing >>>>>>>>>>>>>>>>>> that. So here I don't think the configuration "max-retry" >>>>>> should >>>>>>>>>> be >>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> LookupOptions, because it is not backed by any common >>>>>>>>>> implementation >>>>>>>>>>>>>>>>>> classes. If max-retry is implemented in the HBase source, >> it >>>>>>>>>> should >>>>>>>>>>>>>>> stay >>>>>>>>>>>>>>>>>> there. For the same reason, the config proposed in this >> FLIP >>>>>>>>>>>> (probably >>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>> a name less confusing for the first reason mentioned >> above) >>>>>>>>>> should >>>>>>>>>>>>>>> stay >>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> the specific Source implementation. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> For the two reasons above, I am -1 to what the FLIP >>>> currently >>>>>>>>>>>> proposes. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I think the right way to address the motivation here is >>>> still >>>>>> to >>>>>>>>>>>> have a >>>>>>>>>>>>>>>>>> config like "ignore-pushed-down-filters.enabled" for the >>>>>>>> specific >>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>> implementation. Please let me know if this solves the >>>> problem >>>>>>>> you >>>>>>>>>>>> are >>>>>>>>>>>>>>>>>> facing. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Wed, Nov 15, 2023 at 11:52 AM Jiabao Sun < >>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Becket, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The purpose of introducing this configuration is that not >>>> all >>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>> pushdowns can improve overall performance. >>>>>>>>>>>>>>>>>>> If the filter can hit the external index, then pushdown >> is >>>>>>>>>>>> definitely >>>>>>>>>>>>>>>>>>> worth it, as it can not only improve query time but also >>>>>>>> decrease >>>>>>>>>>>>>>>>> network >>>>>>>>>>>>>>>>>>> overhead. >>>>>>>>>>>>>>>>>>> However, for filters that do not hit the external index, >> it >>>>>> may >>>>>>>>>>>>>>>>> increase a >>>>>>>>>>>>>>>>>>> lot of performance overhead on the external system. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Undeniably, if the connector can make accurate decisions >>>> for >>>>>>>> good >>>>>>>>>>>> and >>>>>>>>>>>>>>>>> bad >>>>>>>>>>>>>>>>>>> filters, we may not need to introduce this configuration >>>>>> option >>>>>>>>>> to >>>>>>>>>>>>>>>>> disable >>>>>>>>>>>>>>>>>>> pushing down filters to the external system. >>>>>>>>>>>>>>>>>>> However, it is currently not easy to achieve. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> IMO, supporting filter pushdown does not mean that always >>>>>>>> filter >>>>>>>>>>>>>>>>> pushdown >>>>>>>>>>>>>>>>>>> is better. >>>>>>>>>>>>>>>>>>> In the absence of automatic decision-making, I think we >>>>>> should >>>>>>>>>>>> leave >>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>> decision to users. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The newly introduced configuration option is similar to >>>>>>>>>>>> LookupOptions, >>>>>>>>>>>>>>>>>>> providing unified naming and default values to avoid >>>>>> confusion >>>>>>>>>>>> caused >>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>> inconsistent naming in different connectors for users. >>>>>>>>>>>>>>>>>>> Setting the default value to true allows it to maintain >>>>>>>>>>>> compatibility >>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>> the default behavior of "always pushdown". >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Do you have any other concerns about this proposal? >> Please >>>>>> let >>>>>>>> me >>>>>>>>>>>>>>> know. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 2023年10月31日 17:29，Jiabao Sun <jiabao....@xtransfer.cn >>>>>>>> .INVALID> >>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Becket, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Actually, for FileSystemSource, it is not always >> desired, >>>>>> only >>>>>>>>>> OCR >>>>>>>>>>>>>>> file >>>>>>>>>>>>>>>>>>> formats support filter pushdown. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> We can disable predicate pushdown for FileSystemSource >> by >>>>>>>>>> setting >>>>>>>>>>>>>>>>>>> 'table.optimizer.source.predicate-pushdown-enabled' to >>>> false. >>>>>>>>>>>>>>>>>>>> I think we can also disable filter pushdown at a more >>>>>> granular >>>>>>>>>>>> level >>>>>>>>>>>>>>>>>>> through fine-grained configuration. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 2023年10月31日 16:50，Becket Qin <becket....@gmail.com> >> 写道： >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks for the explanation. Maybe it's easier to >> explain >>>>>> with >>>>>>>>>> an >>>>>>>>>>>>>>>>>>> example. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Let's take FileSystemTableSource as an example. >> Currently >>>>>> it >>>>>>>>>>>>>>>>> implements >>>>>>>>>>>>>>>>>>>>> SupportsFilterPushDown interface. With your proposal, >>>> does >>>>>> it >>>>>>>>>>>> have >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> support `source.filter-push-down.enabled` as well? But >>>> this >>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>> does not quite make sense for the FileSystemTableSource >>>>>>>> because >>>>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>>> pushdown is always desired. However, because this >>>>>>>> configuration >>>>>>>>>>>> is a >>>>>>>>>>>>>>>>>>> part >>>>>>>>>>>>>>>>>>>>> of the SupportsFilterPushDown interface (which sounds >>>>>>>> confusing >>>>>>>>>>>> to >>>>>>>>>>>>>>>>> begin >>>>>>>>>>>>>>>>>>>>> with), the FileSystemTableSource can only do one of the >>>>>>>>>>>> following: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 1. Ignore the user configuration to always apply the >>>> pushed >>>>>>>>>> down >>>>>>>>>>>>>>>>>>> filters - >>>>>>>>>>>>>>>>>>>>> this is an apparent anti-pattern because a >> configuration >>>>>>>> should >>>>>>>>>>>>>>> always >>>>>>>>>>>>>>>>>>> do >>>>>>>>>>>>>>>>>>>>> what it says. >>>>>>>>>>>>>>>>>>>>> 2. Throw an exception telling users that this >>>> configuration >>>>>>>> is >>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>> applicable to the FileSystemTableSource. >>>>>>>>>>>>>>>>>>>>> 3. Implement this configuration to push back the pushed >>>>>> down >>>>>>>>>>>>>>> filters, >>>>>>>>>>>>>>>>>>> even >>>>>>>>>>>>>>>>>>>>> though this is never desired. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> None of the above options looks awkward. I am curious >>>> what >>>>>>>> your >>>>>>>>>>>>>>>>>>> solution is >>>>>>>>>>>>>>>>>>>>> here? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 31, 2023 at 3:11 PM Jiabao Sun < >>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks Becket for the further explanation. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Perhaps I didn't explain it clearly. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 1. If a source does not implement the >>>>>> SupportsFilterPushDown >>>>>>>>>>>>>>>>> interface, >>>>>>>>>>>>>>>>>>>>>> the newly added configurations do not need to be added >>>> to >>>>>>>>>> either >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> requiredOptions or optionalOptions. >>>>>>>>>>>>>>>>>>>>>> Similar to LookupOptions, if a source does not >> implement >>>>>>>>>>>>>>>>>>>>>> LookupTableSource, there is no need to add >> LookupOptions >>>>>> to >>>>>>>>>>>> either >>>>>>>>>>>>>>>>>>>>>> requiredOptions or optionalOptions. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 2. "And these configs are specific to those sources, >>>>>> instead >>>>>>>>>> of >>>>>>>>>>>>>>>>> common >>>>>>>>>>>>>>>>>>>>>> configs." >>>>>>>>>>>>>>>>>>>>>> The newly introduced configurations define >> standardized >>>>>>>> names >>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> default >>>>>>>>>>>>>>>>>>>>>> values. >>>>>>>>>>>>>>>>>>>>>> They still belong to the configuration at the >> individual >>>>>>>>>> source >>>>>>>>>>>>>>>>> level. >>>>>>>>>>>>>>>>>>>>>> The purpose is to avoid scattered configuration items >>>> when >>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>> sources implement the same logic. >>>>>>>>>>>>>>>>>>>>>> Whether a source should accept these configurations is >>>>>>>>>>>> determined >>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> source's Factory. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 2023年10月31日 13:47，Becket Qin <becket....@gmail.com> >>>> 写道： >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Please see the replies inline. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Introducing common configurations does not mean that >>>> all >>>>>>>>>>>> sources >>>>>>>>>>>>>>>>> must >>>>>>>>>>>>>>>>>>>>>>>> accept these configuration options. >>>>>>>>>>>>>>>>>>>>>>>> The configuration options supported by a source are >>>>>>>>>>>> determined by >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> requiredOptions and optionalOptions in the Factory >>>>>>>>>> interface. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> This is not true. Both required and optional options >>>> are >>>>>>>>>>>>>>> SUPPORTED. >>>>>>>>>>>>>>>>>>> That >>>>>>>>>>>>>>>>>>>>>>> means they are implemented and if one specifies an >>>>>> optional >>>>>>>>>>>> config >>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>> still take effect. An OptionalConfig is "Optional" >>>>>> because >>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>> configuration has a default value. Hence, it is OK >> that >>>>>>>> users >>>>>>>>>>>> do >>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>> specify their own value. In another word, it is >>>>>> "optional" >>>>>>>>>> for >>>>>>>>>>>> the >>>>>>>>>>>>>>>>> end >>>>>>>>>>>>>>>>>>>>>>> users to set the config, but the implementation and >>>>>> support >>>>>>>>>> for >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>> config >>>>>>>>>>>>>>>>>>>>>>> is NOT optional. In case a source does not support a >>>>>> common >>>>>>>>>>>>>>> config, >>>>>>>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>>> exception must be thrown when the config is provided >> by >>>>>> the >>>>>>>>>> end >>>>>>>>>>>>>>>>> users. >>>>>>>>>>>>>>>>>>>>>>> However, the config we are talking about in this FLIP >>>> is >>>>>> a >>>>>>>>>>>> common >>>>>>>>>>>>>>>>>>> config >>>>>>>>>>>>>>>>>>>>>>> optional to implement, meaning that sometimes the >>>> claimed >>>>>>>>>>>> behavior >>>>>>>>>>>>>>>>>>> won't >>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>> there even if users specify that config. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Similar to sources that do not implement the >>>>>>>>>> LookupTableSource >>>>>>>>>>>>>>>>>>> interface, >>>>>>>>>>>>>>>>>>>>>>>> sources that do not implement the >>>> SupportsFilterPushDown >>>>>>>>>>>>>>> interface >>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>> do >>>>>>>>>>>>>>>>>>>>>>>> not need to accept newly introduced options. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> First of all, filter pushdown is a behavior of the >>>> query >>>>>>>>>>>>>>> optimizer, >>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> behavior of Sources. The Sources tells the optimizer >>>> that >>>>>>>> it >>>>>>>>>>>> has >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> ability to accept pushed down filters by implementing >>>> the >>>>>>>>>>>>>>>>>>>>>>> SupportsFilterPushDown interface. And this is the >> only >>>>>>>>>> contract >>>>>>>>>>>>>>>>>>> between >>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> Source and Optimizer regarding whether filters should >>>> be >>>>>>>>>> pushed >>>>>>>>>>>>>>>>> down. >>>>>>>>>>>>>>>>>>> As >>>>>>>>>>>>>>>>>>>>>>> long as a specific source implements this decorative >>>>>>>>>> interface, >>>>>>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>>>>> pushdown should always take place, i.e. >>>>>>>>>>>>>>>>>>>>>>> *SupportsFilterPushDown.applyFilters()* will be >> called. >>>>>>>> There >>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>> no >>>>>>>>>>>>>>>>>>>>>>> other config to disable that call. However, Sources >> can >>>>>>>>>> decide >>>>>>>>>>>> how >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> behave based on their own configurations after >>>>>>>>>>>> *applyFilters()* is >>>>>>>>>>>>>>>>>>>>>> called. >>>>>>>>>>>>>>>>>>>>>>> And these configs are specific to those sources, >>>> instead >>>>>> of >>>>>>>>>>>> common >>>>>>>>>>>>>>>>>>>>>> configs. >>>>>>>>>>>>>>>>>>>>>>> Please see the examples I mentioned in the previous >>>>>> email. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 31, 2023 at 10:27 AM Jiabao Sun < >>>>>>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi Becket, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Sorry, there was a typo in the second point. Let me >>>>>>>> correct >>>>>>>>>>>> it: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Introducing common configurations does not mean that >>>> all >>>>>>>>>>>> sources >>>>>>>>>>>>>>>>> must >>>>>>>>>>>>>>>>>>>>>>>> accept these configuration options. >>>>>>>>>>>>>>>>>>>>>>>> The configuration options supported by a source are >>>>>>>>>>>> determined by >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> requiredOptions and optionalOptions in the Factory >>>>>>>>>> interface. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Similar to sources that do not implement the >>>>>>>>>> LookupTableSource >>>>>>>>>>>>>>>>>>>>>> interface, >>>>>>>>>>>>>>>>>>>>>>>> sources that do not implement the >>>> SupportsFilterPushDown >>>>>>>>>>>>>>> interface >>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>> do >>>>>>>>>>>>>>>>>>>>>>>> not need to accept newly introduced options. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月31日 10:13，Jiabao Sun < >>>> jiabao....@xtransfer.cn >>>>>>>>>>>> .INVALID> >>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks Becket for the feedback. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> 1. Currently, the >> SupportsFilterPushDown#applyFilters >>>>>>>>>> method >>>>>>>>>>>>>>>>>>> returns a >>>>>>>>>>>>>>>>>>>>>>>> result that includes acceptedFilters and >>>>>> remainingFilters. >>>>>>>>>> The >>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>> decide to push down some filters or not accept any >> of >>>>>>>> them. >>>>>>>>>>>>>>>>>>>>>>>>> 2. Introducing common configuration options does >> not >>>>>> mean >>>>>>>>>>>> that a >>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>> that supports the SupportsFilterPushDown capability >>>> must >>>>>>>>>>>> accept >>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>> configuration. Similar to LookupOptions, only >> sources >>>>>> that >>>>>>>>>>>>>>>>> implement >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> LookupTableSource interface are necessary to accept >>>>>> these >>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>> options. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月31日 07:49，Becket Qin < >> becket....@gmail.com> >>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiabao and Ruanhang, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Adding a configuration of >>>>>>>> source.filter-push-down.enabled >>>>>>>>>>>> as a >>>>>>>>>>>>>>>>>>> common >>>>>>>>>>>>>>>>>>>>>>>>>> source configuration seems problematic. >>>>>>>>>>>>>>>>>>>>>>>>>> 1. The config name is misleading. filter pushdown >>>>>> should >>>>>>>>>>>> only >>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>> determined >>>>>>>>>>>>>>>>>>>>>>>>>> by whether the SupportsFilterPushdown interface is >>>>>>>>>>>> implemented >>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>> not. >>>>>>>>>>>>>>>>>>>>>>>>>> 2. The behavior of this configuration is only >>>>>> applicable >>>>>>>>>> to >>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>>> implementations. Why is it a common configuration? >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Here's my suggestion for design principles: >>>>>>>>>>>>>>>>>>>>>>>>>> 1. Only add source impl specific configuration to >>>>>>>>>>>> corresponding >>>>>>>>>>>>>>>>>>>>>> sources. >>>>>>>>>>>>>>>>>>>>>>>>>> 2. The configuration name should not overrule >>>> existing >>>>>>>>>>>> common >>>>>>>>>>>>>>>>>>>>>> contracts. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> For example, in the case of MySql source. There >> are >>>>>>>>>> several >>>>>>>>>>>>>>>>>>> options: >>>>>>>>>>>>>>>>>>>>>>>>>> 1. Have a configuration of >>>>>>>>>>>>>>>>> `*mysql.avoid.remote.full.table.scan`*. >>>>>>>>>>>>>>>>>>> If >>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>> configuration is set, and a filter pushdown does >> not >>>>>> hit >>>>>>>>>> an >>>>>>>>>>>>>>>>> index, >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>> MySql source impl would not further pushdown the >>>>>> filter >>>>>>>> to >>>>>>>>>>>>>>> MySql >>>>>>>>>>>>>>>>>>>>>>>> servers. >>>>>>>>>>>>>>>>>>>>>>>>>> Note that this assumes the MySql source can >> retrieve >>>>>> the >>>>>>>>>>>> index >>>>>>>>>>>>>>>>>>>>>>>> information >>>>>>>>>>>>>>>>>>>>>>>>>> from the MySql servers. >>>>>>>>>>>>>>>>>>>>>>>>>> 2. If the MySql index information is not available >>>> to >>>>>>>> the >>>>>>>>>>>> MySql >>>>>>>>>>>>>>>>>>>>>> source, >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>> configuration could be something like >>>>>>>>>>>>>>>>>>>>>>>> *`mysql.pushback.pushed.down.filters`*. >>>>>>>>>>>>>>>>>>>>>>>>>> Once set to true, MySql source would just add all >>>> the >>>>>>>>>>>> filters >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>> RemainingFilters in the Result returned by >>>>>>>>>>>>>>>>>>>>>>>>>> *SupportsFilterPushdown.applyFilters().* >>>>>>>>>>>>>>>>>>>>>>>>>> 3. An alternative to option 2 is to have a ` >>>>>>>>>>>>>>>>>>>>>>>>>> *mysql.apply.predicates.after.scan*`. When it is >> set >>>>>> to >>>>>>>>>>>> true, >>>>>>>>>>>>>>>>> MySql >>>>>>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>>> will not push the filter down to the MySql >> servers, >>>>>> but >>>>>>>>>>>> apply >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> filters >>>>>>>>>>>>>>>>>>>>>>>>>> inside the MySql source itself. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> As you may see, the above configurations do not >>>>>> disable >>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>>>> pushdown >>>>>>>>>>>>>>>>>>>>>>>>>> itself. They just allow various implementations of >>>>>>>> filter >>>>>>>>>>>>>>>>> pushdown. >>>>>>>>>>>>>>>>>>>>>> And >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>> configuration name does not give any illusion that >>>>>>>> filter >>>>>>>>>>>>>>>>> pushdown >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>> disabled. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 30, 2023 at 11:58 PM Jiabao Sun < >>>>>>>>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Hang for the suggestion. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I think the configuration of TableSource is not >>>>>> closely >>>>>>>>>>>>>>> related >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>> SourceReader, >>>>>>>>>>>>>>>>>>>>>>>>>>> so I prefer to introduce a independent >>>> configuration >>>>>>>>>> class >>>>>>>>>>>>>>>>>>>>>>>>>>> TableSourceOptions in the flink-table-common >>>> module, >>>>>>>>>>>> similar >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>> LookupOptions. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> For the second point, I suggest adding Java doc >> to >>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> SupportsXXXPushDown >>>>>>>>>>>>>>>>>>>>>>>>>>> interfaces, providing detailed information on >> these >>>>>>>>>> options >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>> needs >>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>> be supported. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I have made updates in the FLIP document. >>>>>>>>>>>>>>>>>>>>>>>>>>> Please help check it again. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月30日 17:23，Hang Ruan < >>>> ruanhang1...@gmail.com >>>>>>> >>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the improvements, Jiabao. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> There are some details that I am not sure about. >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. The new option >>>> `source.filter-push-down.enabled` >>>>>>>> will >>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>> added to >>>>>>>>>>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>>>>>>>>> class? I think it should be >> `SourceReaderOptions`. >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. How are the connector developers able to know >>>> and >>>>>>>>>>>> follow >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> FLIP? >>>>>>>>>>>>>>>>>>>>>>>> Do >>>>>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>>> need an abstract base class or provide a default >>>>>>>> method? >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hang >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao Sun <jiabao....@xtransfer.cn.invalid> >>>>>>>>>>>> 于2023年10月30日周一 >>>>>>>>>>>>>>>>>>>>>> 14:45写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, all, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the lively discussion. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Based on the discussion, I have made some >>>>>> adjustments >>>>>>>>>> to >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> FLIP >>>>>>>>>>>>>>>>>>>>>>>>>>> document: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. The name of the newly added option has been >>>>>>>> changed >>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>> "source.filter-push-down.enabled". >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Considering compatibility with older >> versions, >>>>>> the >>>>>>>>>>>> newly >>>>>>>>>>>>>>>>>>> added >>>>>>>>>>>>>>>>>>>>>>>>>>>>> "source.filter-push-down.enabled" option needs >> to >>>>>>>>>> respect >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> optimizer's >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>> option. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> But there is a consideration to remove the old >>>>>> option >>>>>>>>>> in >>>>>>>>>>>>>>> Flink >>>>>>>>>>>>>>>>>>> 2.0. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. We can provide more options to disable other >>>>>>>> source >>>>>>>>>>>>>>>>> abilities >>>>>>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>>>>>>>>>> side >>>>>>>>>>>>>>>>>>>>>>>>>>>>> effects, such as “source.aggregate.enabled” and >>>>>>>>>>>>>>>>>>>>>>>>>>> “source.projection.enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is not urgent and can be continuously >>>>>>>> introduced. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to your feedback again. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月29日 08:45，Becket Qin < >>>>>> becket....@gmail.com> >>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for digging into the git history, >> Jark. I >>>>>>>> agree >>>>>>>>>>>> it >>>>>>>>>>>>>>>>> makes >>>>>>>>>>>>>>>>>>>>>>>> sense >>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deprecate this API in 2.0. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 27, 2023 at 5:47 PM Jark Wu < >>>>>>>>>>>> imj...@gmail.com> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Becket, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I checked the history of " >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> *table.optimizer.source.predicate-pushdown-enabled*", >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it seems it was introduced since the legacy >>>>>>>>>>>>>>>>>>> FilterableTableSource >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interface >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which might be an experiential feature at >> that >>>>>>>> time. >>>>>>>>>> I >>>>>>>>>>>>>>> don't >>>>>>>>>>>>>>>>>>> see >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessity >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of this option at the moment. Maybe we can >>>>>>>> deprecate >>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>> option >>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>> drop >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in Flink 2.0[1] if it is not necessary >> anymore. >>>>>>>> This >>>>>>>>>>>> may >>>>>>>>>>>>>>>>> help >>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> simplify this discussion. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jark >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]: >>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-32383 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, 26 Oct 2023 at 10:14, Becket Qin < >>>>>>>>>>>>>>>>>>> becket....@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the proposal, Jiabao. My two >> cents >>>>>>>> below: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. If I understand correctly, the motivation >>>> of >>>>>>>> the >>>>>>>>>>>> FLIP >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>> mainly to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make predicate pushdown optional on SOME of >>>> the >>>>>>>>>>>> Sources. >>>>>>>>>>>>>>> If >>>>>>>>>>>>>>>>>>> so, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> intuitively >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the configuration should be Source specific >>>>>>>> instead >>>>>>>>>> of >>>>>>>>>>>>>>>>>>> general. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we will end up with general configurations >>>> that >>>>>>>> may >>>>>>>>>>>> not >>>>>>>>>>>>>>>>> take >>>>>>>>>>>>>>>>>>>>>>>> effect >>>>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some of the Source implementations. This >>>>>> violates >>>>>>>>>> the >>>>>>>>>>>>>>> basic >>>>>>>>>>>>>>>>>>> rule >>>>>>>>>>>>>>>>>>>>>>>> of a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration - it does what it says, >>>> regardless >>>>>>>> of >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> implementation. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> While configuration standardization is >>>> usually a >>>>>>>>>> good >>>>>>>>>>>>>>>>> thing, >>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> break the basic rules. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we really want to have this general >>>>>>>>>> configuration, >>>>>>>>>>>> for >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> sources >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this configuration does not apply, they >> should >>>>>>>> throw >>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>> exception >>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>> make >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it clear that this configuration is not >>>>>> supported. >>>>>>>>>>>>>>> However, >>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>> seems >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ugly. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I think the actual motivation of this >> FLIP >>>> is >>>>>>>>>> about >>>>>>>>>>>>>>>>> "how a >>>>>>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should implement predicate pushdown >>>>>> efficiently", >>>>>>>>>> not >>>>>>>>>>>>>>>>>>> "whether >>>>>>>>>>>>>>>>>>>>>>>>>>>>> predicate >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown should be applied to the source." >> For >>>>>>>>>>>> example, >>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wants >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to avoid additional computing load in the >>>>>> external >>>>>>>>>>>>>>> system, >>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>>>>>>> always >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> read the entire record and apply the >>>> predicates >>>>>> by >>>>>>>>>>>>>>> itself. >>>>>>>>>>>>>>>>>>>>>>>> However, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Flink perspective, the predicate >> pushdown >>>> is >>>>>>>>>>>> applied, >>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>> just >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented differently by the source. So >> the >>>>>>>> design >>>>>>>>>>>>>>>>>>> principle >>>>>>>>>>>>>>>>>>>>>>>> here >>>>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Flink only cares about whether a source >>>> supports >>>>>>>>>>>>>>> predicate >>>>>>>>>>>>>>>>>>>>>>>> pushdown >>>>>>>>>>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>>>>>>>> not, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it does not care about the implementation >>>>>>>>>> efficiency / >>>>>>>>>>>>>>> side >>>>>>>>>>>>>>>>>>>>>>>> effect of >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> predicates pushdown. It is the Source >>>>>>>>>> implementation's >>>>>>>>>>>>>>>>>>>>>>>> responsibility >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ensure the predicates pushdown is >> implemented >>>>>>>>>>>> efficiently >>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>> does >>>>>>>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> impose excessive pressure on the external >>>>>> system. >>>>>>>>>> And >>>>>>>>>>>> it >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>> OK >>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> additional configurations to achieve this >>>> goal. >>>>>>>>>>>>>>> Obviously, >>>>>>>>>>>>>>>>>>> such >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations will be source specific in >> this >>>>>>>> case. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Regarding the existing configurations of >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> *table.optimizer.source.predicate-pushdown-enabled. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *I am not sure why we need it. Supposedly, >> if >>>> a >>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>> implements >>>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SupportsXXXPushDown interface, the optimizer >>>>>>>> should >>>>>>>>>>>> push >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> corresponding >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> predicates to the Source. I am not sure in >>>> which >>>>>>>>>> case >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would be used. Any ideas @Jark Wu < >>>>>>>> imj...@gmail.com >>>>>>>>>>> ? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 25, 2023 at 11:55 PM Jiabao Sun >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <jiabao....@xtransfer.cn.invalid> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Jane for the detailed explanation. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think that for users, we should respect >>>>>>>>>> conventions >>>>>>>>>>>>>>> over >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Conventions can be default values >> explicitly >>>>>>>>>>>> specified >>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations, or they can be behaviors >> that >>>>>>>>>> follow >>>>>>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>>>>>>>>>>>>>> versions. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If the same code has different behaviors in >>>>>>>>>> different >>>>>>>>>>>>>>>>>>> versions, >>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be a very bad thing. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree that for regular users, it is not >>>>>>>> necessary >>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>> understand >>>>>>>>>>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the configurations related to Flink. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> By following conventions, they can have a >>>> good >>>>>>>>>>>>>>> experience. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Let's get back to the practical situation >> and >>>>>>>>>>>> consider >>>>>>>>>>>>>>> it. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Case 1: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The user is not familiar with the purpose >> of >>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> table.optimizer.source.predicate-pushdown-enabled >>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>>>>>>>>> follows >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the convention of allowing predicate >> pushdown >>>>>> to >>>>>>>>>> the >>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>>>>>>>> default. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just understanding the >>>>>>>>>>>> source.predicate-pushdown-enabled >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and performing fine-grained toggle control >>>> will >>>>>>>>>> work >>>>>>>>>>>>>>> well. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Case 2: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The user understands the meaning of the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> table.optimizer.source.predicate-pushdown-enabled >>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>> has set >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> its value to false. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We have reason to believe that the user >>>>>>>> understands >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> meaning >>>>>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> predicate pushdown configuration and the >>>>>>>> intention >>>>>>>>>>>> is to >>>>>>>>>>>>>>>>>>>>>> disable >>>>>>>>>>>>>>>>>>>>>>>>>>>>> predicate >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown (rather than whether or not to >> allow >>>>>>>> it). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The previous choice of globally disabling >> it >>>> is >>>>>>>>>>>> likely >>>>>>>>>>>>>>>>>>> because >>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> couldn't be disabled on individual sources. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From this perspective, if we provide more >>>>>>>>>>>> fine-grained >>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support and provide detailed explanations >> of >>>>>> the >>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>> behaviors in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the documentation, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users can clearly understand the >> differences >>>>>>>>>> between >>>>>>>>>>>>>>> these >>>>>>>>>>>>>>>>>>> two >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations and use them correctly. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also, I don't agree that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> table.optimizer.source.predicate-pushdown-enabled = >>>>>>>>>>>> true >>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source.predicate-pushdown-enabled = false >>>> means >>>>>>>>>> that >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> local >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration overrides the global >>>>>> configuration. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On the contrary, both configurations are >>>>>>>>>> functioning >>>>>>>>>>>>>>>>>>> correctly. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The optimizer allows predicate pushdown to >>>> all >>>>>>>>>>>> sources, >>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>>>>>>>>>>>>>>> sources >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can reject the filters pushed down by the >>>>>>>>>> optimizer. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is natural, just like different >>>> components >>>>>>>> at >>>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>>> levels >>>>>>>>>>>>>>>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> responsible for different tasks. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The more serious issue is that if >>>>>>>>>>>>>>>>>>>>>>>>>>> "source.predicate-pushdown-enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does not respect >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> "table.optimizer.source.predicate-pushdown-enabled”, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be invalidated. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This means that regardless of whether >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>> is >>>>>>>>>>>>>>> set >>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>> true >>>>>>>>>>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> false, it will have no effect. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月25日 22:24，Jane Chan < >>>>>>>>>> qingyue....@gmail.com> >>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the in-depth clarification. >> Here >>>>>> are >>>>>>>> my >>>>>>>>>>>>>>> cents >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, >>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" are >>>>>>>>>> configurations >>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> components(optimizer and source >> operator). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We cannot assume that every user would be >>>>>>>>>>>> interested in >>>>>>>>>>>>>>>>>>>>>>>>>>> understanding >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal components of Flink, such as the >>>>>>>>>> optimizer >>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>>> connectors, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific configurations associated with >> each >>>>>>>>>>>> component. >>>>>>>>>>>>>>>>>>>>>> Instead, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> users >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> might be more concerned about knowing >> which >>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>> enables >>>>>>>>>>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disables the filter push-down feature for >>>> all >>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>> connectors, >>>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter provides the flexibility to >>>> override >>>>>>>>>> this >>>>>>>>>>>>>>>>>>> behavior >>>>>>>>>>>>>>>>>>>>>>>> for a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> single >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source if needed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, from this perspective, I am inclined >> to >>>>>>>> divide >>>>>>>>>>>>>>> these >>>>>>>>>>>>>>>>>>> two >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameters >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> based on the scope of their impact from >> the >>>>>>>> user's >>>>>>>>>>>>>>>>>>> perspective >>>>>>>>>>>>>>>>>>>>>>>>>>> (i.e. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> global-level or operator-level), rather >> than >>>>>>>>>>>>>>> categorizing >>>>>>>>>>>>>>>>>>> them >>>>>>>>>>>>>>>>>>>>>>>>>>> based >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> component hierarchy from a developer's >> point >>>>>> of >>>>>>>>>>>> view. >>>>>>>>>>>>>>>>>>>>>> Therefore, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> based >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this premise, it is intuitive and natural >>>> for >>>>>>>>>> users >>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand fine-grained configuration >>>> options >>>>>>>> can >>>>>>>>>>>>>>>>> override >>>>>>>>>>>>>>>>>>>>>>>> global >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Additionally, if >>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>> doesn't >>>>>>>>>>>>>>>>>>>>>>>> respect to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> default >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of "scan.filter-push-down.enabled" is >>>> defined >>>>>>>> as >>>>>>>>>>>> true, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it means that just modifying >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" as >>>>>>>>>>>>>>>>>>> false >>>>>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have no >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> effect, and filter pushdown will still be >>>>>>>>>>>> performed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we define the default value of >>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> false, it would introduce a difference in >>>>>>>>>> behavior >>>>>>>>>>>>>>>>>>> compared >>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <1>If I understand correctly, >>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>>>>>>>>>> is a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> connector option, which means the only way >>>> to >>>>>>>>>>>> configure >>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicitly >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specify it in DDL (no matter whether >> disable >>>>>> or >>>>>>>>>>>>>>> enable), >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> SET >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> command is not applicable, so I think it's >>>>>>>> natural >>>>>>>>>>>> to >>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>>>>>> respect >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user's >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification here. Otherwise, users might >>>> be >>>>>>>> more >>>>>>>>>>>>>>>>> confused >>>>>>>>>>>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>>>>>>>>>>> why >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DDL does not work as expected, and the >>>> reason >>>>>> is >>>>>>>>>>>> just >>>>>>>>>>>>>>>>>>> because >>>>>>>>>>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "optimizer" configuration is set to a >>>>>> different >>>>>>>>>>>> value. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <2> From the implementation side, I am >>>>>> inclined >>>>>>>> to >>>>>>>>>>>> keep >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter's >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> priority consistent for all conditions. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Let "global" denote >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled", >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and let "per-source" denote >>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source T, the following Truth table >> (based >>>> on >>>>>>>> the >>>>>>>>>>>>>>>>> current >>>>>>>>>>>>>>>>>>>>>>>> design) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> indicates the inconsistent behavior for >>>>>>>>>> "per-source >>>>>>>>>>>>>>>>>>> override >>>>>>>>>>>>>>>>>>>>>>>>>>> global". >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> .------------.---------------.------------------- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> ----.-------------------------------------. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | global | per-source | push-down for T >> | >>>>>>>>>>>> per-source >>>>>>>>>>>>>>>>>>>>>> override >>>>>>>>>>>>>>>>>>>>>>>>>>>>> global >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> >>>> >> :-----------+--------------+-----------------------+------------------------------------: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | true | false | false >>>>>>>>>>>>>>>>> | Y >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> >>>> >> :-----------+--------------+-----------------------+------------------------------------: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | false | true | false >>>>>>>>>>>>>>>>> | N >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> >>>> >> .------------.---------------.-----------------------.-------------------------------------. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jane >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 25, 2023 at 6:22 PM Jiabao >> Sun < >>>>>>>>>>>>>>>>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Benchao for the feedback. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I understand that the configuration of >>>> global >>>>>>>>>>>>>>>>> parallelism >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>> task >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parallelism is at different granularities >>>> but >>>>>>>>>> with >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> same >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, >>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" are >>>>>>>>>> configurations >>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> components(optimizer and source >> operator). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From a user's perspective, there are two >>>>>>>>>> scenarios: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Disabling all filter pushdown >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In this case, setting >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to false is sufficient to meet the >>>>>> requirement. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Disabling filter pushdown for specific >>>>>>>> sources >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In this scenario, there is no need to >>>> adjust >>>>>>>> the >>>>>>>>>>>> value >>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled". >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Instead, the focus should be on the >>>>>>>> configuration >>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" to meet >> the >>>>>>>>>>>>>>> requirement. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In this case, users do not need to set >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" to >>>>>>>>>>>>>>>>>>> false >>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> manually >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enable filter pushdown for specific >>>> sources. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Additionally, if >>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>> doesn't >>>>>>>>>>>>>>>>>>>>>>>> respect >>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> default >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of "scan.filter-push-down.enabled" is >>>> defined >>>>>>>> as >>>>>>>>>>>> true, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it means that just modifying >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" as >>>>>>>>>>>>>>>>>>> false >>>>>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have no >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> effect, and filter pushdown will still be >>>>>>>>>>>> performed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we define the default value of >>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> false, it would introduce a difference in >>>>>>>>>> behavior >>>>>>>>>>>>>>>>>>> compared >>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The same SQL query that could >> successfully >>>>>> push >>>>>>>>>>>> down >>>>>>>>>>>>>>>>>>> filters >>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> old >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version but would no longer do so after >> the >>>>>>>>>>>> upgrade. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月25日 17:10，Benchao Li < >>>>>>>>>>>> libenc...@apache.org> >>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Jiabao for the detailed >>>> explanations, >>>>>>>>>> that >>>>>>>>>>>>>>>>> helps a >>>>>>>>>>>>>>>>>>>>>>>> lot, I >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand your rationale now. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Correct me if I'm wrong. Your >> perspective >>>> is >>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>> "developer", >>>>>>>>>>>>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means there is an optimizer and >> connector >>>>>>>>>>>> component, >>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>> want >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enable this feature (pushing filters >> down >>>>>> into >>>>>>>>>>>>>>>>>>> connectors), >>>>>>>>>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>>>>>>>>>>>>>> must >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enable it firstly in optimizer, and only >>>>>> then >>>>>>>>>>>>>>> connector >>>>>>>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chance >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to decide to use it or not. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My perspective is from "user" that (Why >> a >>>>>> user >>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>> care >>>>>>>>>>>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> difference of optimizer/connector) , >> this >>>>>> is a >>>>>>>>>>>>>>> feature, >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>>>>>>>>>>>> two >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> way to control it, one way is to config >> it >>>>>>>>>>>> job-level, >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> other >>>>>>>>>>>>>>>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in table properties. What a user expects >>>> is >>>>>>>> that >>>>>>>>>>>> they >>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>>>>> control a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feature in a tiered way, that setting it >>>> per >>>>>>>>>> job, >>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fine-grained tune it per table. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is some kind of similar to other >>>>>>>> concepts, >>>>>>>>>>>> such >>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parallelism, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users can set a job level default >>>>>> parallelism, >>>>>>>>>> and >>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>>>>>>>>>>>>>>> fine-grained >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tune it per operator. There may be more >>>> such >>>>>>>>>>>> debate >>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> future >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> e.g., we can have a job level config >> about >>>>>>>>>> adding >>>>>>>>>>>>>>>>> key-by >>>>>>>>>>>>>>>>>>>>>>>> before >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> join, and also a hint/table property way >>>> to >>>>>>>>>>>>>>>>> fine-grained >>>>>>>>>>>>>>>>>>>>>>>> control >>>>>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> per lookup operator. Hence we'd better >>>> find >>>>>> a >>>>>>>>>>>> unified >>>>>>>>>>>>>>>>> way >>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> those similar kind of features. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao Sun <jiabao....@xtransfer.cn >>>>>> .invalid> >>>>>>>>>>>>>>>>>>> 于2023年10月25日周三 >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 15:27写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Jane for further explanation. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> These two configurations correspond to >>>>>>>>>> different >>>>>>>>>>>>>>>>> levels. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" does not >>>> make >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >> invalid. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The planner will still push down >>>> predicates >>>>>>>> to >>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>> sources. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Whether filter pushdown is allowed or >> not >>>>>> is >>>>>>>>>>>>>>>>> determined >>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source's "scan.filter-push-down.enabled" >>>>>>>>>>>>>>> configuration. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, >>>> "table.optimizer.source.predicate" >>>>>>>>>> does >>>>>>>>>>>>>>>>>>> directly >>>>>>>>>>>>>>>>>>>>>>>> affect >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled”. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When the planner disables predicate >>>>>> pushdown, >>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> source-level >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown will also not be executed, even >> if >>>>>> the >>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>> allows >>>>>>>>>>>>>>>>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Whatever, in point 1 and 2, our >>>> expectation >>>>>>>> is >>>>>>>>>>>>>>>>>>> consistent. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the 3rd point, I still think that >> the >>>>>>>>>>>>>>>>> planner-level >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> takes precedence over the source-level >>>>>>>>>>>> configuration. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It may seem counterintuitive when we >>>>>> globally >>>>>>>>>>>>>>> disable >>>>>>>>>>>>>>>>>>>>>>>> predicate >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown but allow filter pushdown at the >>>>>>>> source >>>>>>>>>>>>>>> level. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月25日 14:35，Jane Chan < >>>>>>>>>>>> qingyue....@gmail.com >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for clarifying this. While by >>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> takes a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> higher priority" I meant that this >> value >>>>>>>>>> should >>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>> respected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whenever >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> set explicitly. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The conclusion that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. "table.optimizer.source.predicate" >> = >>>>>>>> "true" >>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" = >>>> "false" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Allow the planner to perform >> predicate >>>>>>>>>>>> pushdown, >>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>>>>>>> individual >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sources do >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not enable filter pushdown. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This indicates that the option >>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled = >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> false" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an individual source connector does >>>> indeed >>>>>>>>>>>> override >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> global-level >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> planner settings to make a difference. >>>> And >>>>>>>>>> thus >>>>>>>>>>>>>>> "has >>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>> higher >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> priority". >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> While for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. "table.optimizer.source.predicate" >> = >>>>>>>>>> "false" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Predicate pushdown is not allowed for >>>> the >>>>>>>>>>>> planner. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regardless of the value of the >>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration, filter pushdown is >>>>>> disabled. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In this scenario, the behavior >> remains >>>>>>>>>>>> consistent >>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> old >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> well. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I still think >>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> respected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is enabled for individual >> connectors. >>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jane >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 25, 2023 at 1:27 PM Jiabao >>>>>> Sun < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Benchao for the feedback. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the current proposal, we >> recommend >>>>>>>>>> keeping >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> default >>>>>>>>>>>>>>>>>>>>>>>>>>> value >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" as >>>>>> true, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and setting the the default value of >>>>>> newly >>>>>>>>>>>>>>>>> introduced >>>>>>>>>>>>>>>>>>>>>>>> option >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" to >> true >>>>>> as >>>>>>>>>>>> well. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main purpose of doing this is to >>>>>>>> maintain >>>>>>>>>>>>>>>>>>> consistency >>>>>>>>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versions, as whether to perform >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> filter pushdown in the old version >>>> solely >>>>>>>>>>>> depends >>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >>>>>> option. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That means by default, as long as a >>>>>>>>>> TableSource >>>>>>>>>>>>>>>>>>>>>> implements >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SupportsFilterPushDown interface, >>>> filter >>>>>>>>>>>> pushdown >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>> allowed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And it seems that we don't have much >>>>>>>> benefit >>>>>>>>>> in >>>>>>>>>>>>>>>>>>> changing >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of "table.optimizer.source.predicate" >>>> to >>>>>>>>>> false. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding the priority of these two >>>>>>>>>>>>>>> configurations, >>>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>>>>>>>> believe >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> takes precedence over >>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exhibits >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> following behavior. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. >> "table.optimizer.source.predicate" = >>>>>>>>>> "true" >>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" = >>>> "true" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is the default behavior, >> allowing >>>>>>>> filter >>>>>>>>>>>>>>>>> pushdown >>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sources. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. >> "table.optimizer.source.predicate" = >>>>>>>>>> "true" >>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" = >>>> "false" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Allow the planner to perform >> predicate >>>>>>>>>>>> pushdown, >>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>>>>>>> individual >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sources do >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not enable filter pushdown. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. >> "table.optimizer.source.predicate" = >>>>>>>>>> "false" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Predicate pushdown is not allowed for >>>> the >>>>>>>>>>>> planner. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regardless of the value of the >>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration, filter pushdown is >>>>>> disabled. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In this scenario, the behavior >> remains >>>>>>>>>>>> consistent >>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> old >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> well. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From an implementation perspective, >>>>>> setting >>>>>>>>>> the >>>>>>>>>>>>>>>>>>> priority >>>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >> higher >>>>>> than >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" is >>>>>>>>>>>> difficult to >>>>>>>>>>>>>>>>>>>>>> achieve >>>>>>>>>>>>>>>>>>>>>>>>>>> now. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Because the >>>>>>>> PushFilterIntoSourceScanRuleBase >>>>>>>>>> at >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> planner >>>>>>>>>>>>>>>>>>>>>>>>>>>>> level >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> takes >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> precedence over the source-level >>>>>>>>>>>>>>> FilterPushDownSpec. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Only when the >>>>>>>>>> PushFilterIntoSourceScanRuleBase >>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> enabled, >>>>>>>>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Source-level filter pushdown be >>>>>> performed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Additionally, in my opinion, there >>>>>> doesn't >>>>>>>>>>>> seem to >>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>> much >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> benefit in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> setting a higher priority for >>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled". >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It may instead affect compatibility >> and >>>>>>>>>>>> increase >>>>>>>>>>>>>>>>>>>>>>>>>>> implementation >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> complexity. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月25日 11:56，Benchao Li < >>>>>>>>>>>>>>> libenc...@apache.org >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with Jane that fine-grained >>>>>>>>>>>>>>> configurations >>>>>>>>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> higher >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> priority than job level >>>> configurations. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For current proposal, we can achieve >>>>>> that: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Set >>>>>> "table.optimizer.source.predicate" = >>>>>>>>>>>> "true" >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>> enable >>>>>>>>>>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default, and set >>>>>>>>>>>>>>> ""scan.filter-push-down.enabled" = >>>>>>>>>>>>>>>>>>>>>>>> "false" >>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it per table source >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Set >>>>>> "table.optimizer.source.predicate" = >>>>>>>>>>>>>>> "false" >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>> disable >>>>>>>>>>>>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default, and set >>>>>>>>>>>>>>> ""scan.filter-push-down.enabled" = >>>>>>>>>>>>>>>>>>>>>>>> "true" to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it per table source >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jane Chan <qingyue....@gmail.com> >>>>>>>>>>>> 于2023年10月24日周二 >>>>>>>>>>>>>>>>>>>>>> 23:55写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe that the configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> higher priority at the planner >> level >>>>>>>> than >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and it seems easy to implement >> now. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Correct me if I'm wrong, but I >> think >>>>>> the >>>>>>>>>>>>>>>>>>> fine-grained >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>> should >>>>>>>>>> have a >>>>>>>>>>>>>>>>> higher >>>>>>>>>>>>>>>>>>>>>>>>>>> priority >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> because >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default value of >>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>> true. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> As >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turning off filter push-down for a >>>>>>>> specific >>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>>>> take >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> effect >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unless the default value of >>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> changed >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to false, or, alternatively, let >>>> users >>>>>>>>>>>> manually >>>>>>>>>>>>>>>>> set >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >> to >>>>>>>> false >>>>>>>>>>>>>>> first >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> selectively >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enable filter push-down for the >>>> desired >>>>>>>>>>>> sources, >>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>> less >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intuitive. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jane >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 24, 2023 at 6:05 PM >>>> Jiabao >>>>>>>> Sun >>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Jane, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe that the configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> higher priority at the planner >> level >>>>>>>> than >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and it seems easy to implement >> now. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月24日 17:36，Jane Chan < >>>>>>>>>>>>>>>>>>> qingyue....@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving this >>>> discussion. I >>>>>>>>>> have >>>>>>>>>>>> a >>>>>>>>>>>>>>>>> small >>>>>>>>>>>>>>>>>>>>>>>>>>> question >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>> take >>>>>>>>>>>>>>> precedence >>>>>>>>>>>>>>>>>>> over >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> "table.optimizer.source.predicate" >>>>>> when >>>>>>>>>> the >>>>>>>>>>>>>>> two >>>>>>>>>>>>>>>>>>>>>>>> parameters >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> might >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conflict >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> each other? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jane >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 24, 2023 at 5:05 PM >>>>>> Jiabao >>>>>>>>>> Sun >>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Jark, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we only add configuration >>>> without >>>>>>>>>>>> adding >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enableFilterPushDown >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method in the >>>> SupportsFilterPushDown >>>>>>>>>>>>>>> interface, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> each connector would have to >>>> handle >>>>>>>> the >>>>>>>>>>>> same >>>>>>>>>>>>>>>>>>> logic >>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applyFilters >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method to determine whether >> filter >>>>>>>>>>>> pushdown >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> needed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This would increase complexity >> and >>>>>>>>>> violate >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> original >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applyFilters method. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On the contrary, we only need to >>>>>> pass >>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly added enableFilterPushDown >>>>>>>> method >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to decide whether to perform >>>>>> predicate >>>>>>>>>>>>>>>>> pushdown. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think this approach would be >>>>>> clearer >>>>>>>>>> and >>>>>>>>>>>>>>>>>>> simpler. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月24日 16:58，Jark Wu < >>>>>>>>>>>> imj...@gmail.com >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi JIabao, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think the current interface >> can >>>>>>>>>> already >>>>>>>>>>>>>>>>>>> satisfy >>>>>>>>>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirements. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The connector can reject all >> the >>>>>>>>>> filters >>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>> returning >>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> input >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> filters >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as `Result#remainingFilters`. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So maybe we don't need to >>>>>> introduce a >>>>>>>>>> new >>>>>>>>>>>>>>>>>>> method to >>>>>>>>>>>>>>>>>>>>>>>>>>>>> disable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown, but just introduce an >>>>>>>> option >>>>>>>>>>>> for >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> specific >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> connector. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jark >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, 24 Oct 2023 at 16:38, >>>>>> Leonard >>>>>>>>>> Xu >>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>>> xbjt...@gmail.com >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks @Jiabao for kicking off >>>>>> this >>>>>>>>>>>>>>>>> discussion. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Could you add a section to >>>> explain >>>>>>>> the >>>>>>>>>>>>>>>>>>> difference >>>>>>>>>>>>>>>>>>>>>>>>>>> between >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> connector level config >>>>>>>>>>>>>>>>>>>>>>>> `scan.filter-push-down.enabled` >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> query >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level config >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> `table.optimizer.source.predicate-pushdown-enabled` ? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Leonard >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月24日 下午4:18，Jiabao >> Sun < >>>>>>>>>>>>>>>>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .INVALID> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Devs, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a >>>>>> discussion >>>>>>>> on >>>>>>>>>>>>>>>>>>> FLIP-377: >>>>>>>>>>>>>>>>>>>>>>>>>>> support >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disable filter pushdown for >>>>>>>> Table/SQL >>>>>>>>>>>>>>>>>>> Sources[1]. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently, Flink Table/SQL >> does >>>>>> not >>>>>>>>>>>> expose >>>>>>>>>>>>>>>>>>>>>>>>>>> fine-grained >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users to enable or disable >>>> filter >>>>>>>>>>>> pushdown. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, filter pushdown has >>>> some >>>>>>>>>> side >>>>>>>>>>>>>>>>>>> effects, >>>>>>>>>>>>>>>>>>>>>>>> such >>>>>>>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> additional >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> computational pressure on >>>> external >>>>>>>>>>>> systems. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moreover, Improper queries >> can >>>>>> lead >>>>>>>>>> to >>>>>>>>>>>>>>>>> issues >>>>>>>>>>>>>>>>>>>>>> such >>>>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>>>>>> full >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scans, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which in turn can impact the >>>>>>>> stability >>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>> external >>>>>>>>>>>>>>>>>>>>>>>>>>>>> systems. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Suppose we have an SQL query >>>> with >>>>>>>> two >>>>>>>>>>>>>>>>> sources: >>>>>>>>>>>>>>>>>>>>>>>> Kafka >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> database. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The database is sensitive to >>>>>>>>>> pressure, >>>>>>>>>>>> and >>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>> want >>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configure >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not perform filter pushdown to >>>> the >>>>>>>>>>>> database >>>>>>>>>>>>>>>>>>>>>> source. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, we still want to >>>> perform >>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>> pushdown >>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kafka >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decrease network IO. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I propose to support >>>>>> configuration >>>>>>>> to >>>>>>>>>>>>>>>>> disable >>>>>>>>>>>>>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>>>>>>>>>>> push >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> down for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Table/SQL sources to let user >>>>>> decide >>>>>>>>>>>>>>> whether >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>> perform >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to your >>>> feedback. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> >>>> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Benchao Li >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Benchao Li >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> >>>> >>>> >> >>
