Sorry, i meant white-list ~ Danny Chan <danny0...@apache.org> 于2020年3月27日周五 下午12:40写道：
> Thanks everyone for the feedback ~ > > - For the global config option belongs to `ExecutionConfigOptions` or > `OptimizerConfigOptions`, i have to strong objections, switch > to `OptimizerConfigOptions` is okey to me and i have updated the WIKI > - For use while-list or black-list, i have opinion with Timo, so black-list > > I would fire a Vote if there are no other objections soon, thanks ~ > > Timo Walther <twal...@apache.org> 于2020年3月26日周四 下午6:31写道： > >> Hi everyone, >> >> it is not only about security concerns. Hint options should be >> well-defined. We had a couple of people that were concerned about >> changing the semantics with a concept that is called "hint". These >> options are more like "debugging options" while someone is developing a >> connector or using a notebook to quickly produce some rows. >> >> The final pipeline should use a temporary table instead. I suggest to >> use a whitelist and force people to think about what should be exposed >> as a hint. By default, no option should be exposed. It is better to be >> conservative here. >> >> Regards, >> Timo >> >> >> On 26.03.20 10:31, Danny Chan wrote: >> > Thanks Kurt for the suggestion ~ >> > >> > In my opinion: >> > - There is no need for TableFormatFactory#supportedHintOptions because >> all >> > the format options can be configured dynamically, they have no security >> > issues >> > - Dynamic table options is not an optimization, it is more like an >> > execution behavior from my side >> > >> > Kurt Young <ykt...@gmail.com> 于2020年3月26日周四 下午4:47写道： >> > >> >> Hi Danny, >> >> >> >> Thanks for the updates. I have 2 comments regarding to latest document: >> >> >> >> 1) I think we also need `*supportedHintOptions*` for >> >> `*TableFormatFactory*` >> >> 2) IMO "dynamic-table-options.enabled" should belong to ` >> >> *OptimizerConfigOptions*` >> >> >> >> Best, >> >> Kurt >> >> >> >> >> >> On Thu, Mar 26, 2020 at 4:40 PM Timo Walther <twal...@apache.org> >> wrote: >> >> >> >>> Thanks for the update Danny. +1 for this proposal. >> >>> >> >>> Regards, >> >>> Timo >> >>> >> >>> On 26.03.20 04:51, Danny Chan wrote: >> >>>> Thanks everyone who engaged in this discussion ~ >> >>>> >> >>>> Our goal is "Supports Dynamic Table Options for Flink SQL". After an >> >>>> offline discussion with Kurt, Timo and Dawid, we have made the final >> >>>> conclusion, here is the summary: >> >>>> >> >>>> >> >>>> - Use comment style syntax to specify the dynamic table options: >> >> "/*+ >> >>>> *OPTIONS*(k1='v1', k2='v2') */" >> >>>> - Have constraint on the options keys: the options that may >> bring >> >> in >> >>>> security problems should not be allowed, i.e. Kafka connector >> >>> zookeeper >> >>>> endpoint URL and topic name >> >>>> - Use white-list to control the allowed options for each >> connector, >> >>>> which is more safe for future extention >> >>>> - We allow to enable/disable this feature globally >> >>>> - Implement based on the current code base first, and when >> FLIP-95 >> >> is >> >>>> checked in, implement this feature based on new interface >> >>>> >> >>>> Any suggestions are appreciated ~ >> >>>> >> >>>> [1] >> >>>> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL >> >>>> >> >>>> Best, >> >>>> Danny Chan >> >>>> >> >>>> Jark Wu <imj...@gmail.com> 于2020年3月18日周三 下午10:38写道： >> >>>> >> >>>>> Hi everyone, >> >>>>> >> >>>>> Sorry, but I'm not sure about the `supportedHintOptions`. I'm afraid >> >> it >> >>>>> doesn't solve the problems but increases some development and >> learning >> >>>>> burdens. >> >>>>> >> >>>>> # increase development and learning burden >> >>>>> >> >>>>> According to the discussion so far, we want to support overriding a >> >>> subset >> >>>>> of options in hints which doesn't affect semantics. >> >>>>> With the `supportedHintOptions`, it's up to the connector developers >> >> to >> >>>>> decide which options will not affect semantics, and to be hint >> >> options. >> >>>>> However, the question is how to distinguish whether an option will >> >>> *affect >> >>>>> semantics*? What happens if an option will affect semantics but >> >>> provided as >> >>>>> hint options? >> >>>>> From my point of view, it's not easy to distinguish. For example, >> the >> >>>>> "format.ignore-parse-error" can be a very useful dynamic option but >> >> that >> >>>>> will affect semantic, because the result is different (null vs >> >>> exception). >> >>>>> Another example, the "connector.lookup.cache.*" options are also >> very >> >>>>> useful to tune jobs, however, it will also affect the job results. I >> >> can >> >>>>> come up many more useful options but may affect semantics. >> >>>>> >> >>>>> I can see that the community will under endless discussion around >> "can >> >>> this >> >>>>> option to be a hint option?", "wether this option will affect >> >>> semantics?". >> >>>>> You can also find that we already have different opinions on >> >>>>> "ignore-parse-error". Those discussion is a waste of time! That's >> not >> >>> what >> >>>>> users want! >> >>>>> The problem is user need this, this, this options and HOW to expose >> >>> them? >> >>>>> We should focus on that. >> >>>>> >> >>>>> Then there could be two endings in the future: >> >>>>> 1) compromise on the usability, we drop the rule that hints don't >> >> affect >> >>>>> semantics, allow all the useful options in the hints list. >> >>>>> 2) stick on the rule, users will find this is a stumbling feature >> >> which >> >>>>> doesn't solve their problems. >> >>>>> And they will be surprised why this option can't be set, but >> the >> >>> other >> >>>>> could. *semantic* is hard to be understood by users. >> >>>>> >> >>>>> # doesn't solve the problems >> >>>>> >> >>>>> I think the purpose of this FLIP is to allow users to quickly >> override >> >>> some >> >>>>> connectors' properties to tune their jobs. >> >>>>> However, `supportedHintOptions` is off track. It only allows a >> subset >> >>>>> options and for the users it's not *clear* which subset is allowed. >> >>>>> >> >>>>> Besides, I'm not sure `supportedHintOptions` can work well for all >> >>> cases. >> >>>>> How could you support kafka properties (`connector.properties.*`) as >> >>> hint >> >>>>> options? Some kafka properties may affect semantics >> >> (bootstrap.servers), >> >>>>> some may not (max.poll.records). Besides, I think it's not possible >> to >> >>> list >> >>>>> all the possible kafka properties [1]. >> >>>>> >> >>>>> In summary, IMO, `supportedHintOptions` >> >>>>> (1) it increase the complexity to develop a connector >> >>>>> (2) it confuses users which options can be used in hint, which are >> >> not, >> >>>>> they have to check the docs again and again. >> >>>>> (3) it doesn't solve the problems which we want to solve by this >> FLIP. >> >>>>> >> >>>>> I think we should avoid introducing some partial solutions. >> Otherwise, >> >>> we >> >>>>> will be stuck in a loop that introduce new API -> deprecate API -> >> >>>>> introduce new API.... >> >>>>> >> >>>>> I personally in favor of an explicit WITH syntax after the table as >> a >> >>> part >> >>>>> of the query which is mentioned by Kurt before, e.g. SELECT * from T >> >>>>> WITH('key' = 'value') . >> >>>>> It allows users to dynamically set options which can affect >> semantics. >> >>> It >> >>>>> will be very flexible to solve users' problems so far. >> >>>>> >> >>>>> Best, >> >>>>> Jark >> >>>>> >> >>>>> [1]: https://kafka.apache.org/documentation/#consumerconfigs >> >>>>> >> >>>>> On Wed, 18 Mar 2020 at 21:44, Danny Chan <yuzhao....@gmail.com> >> >> wrote: >> >>>>> >> >>>>>> My POC is here for the hints options merge [1]. >> >>>>>> >> >>>>>> Personally, I have no strong objections for splitting hints with >> the >> >>>>>> CatalogTable, the only cons is a more complex implementation but >> the >> >>>>>> concept is more clear, and I have updated the WIKI. >> >>>>>> >> >>>>>> I think it would be nice if we can support the format “ignore-parse >> >>>>> error” >> >>>>>> option key, the CSV source already has a key [2] and we can use >> that >> >> in >> >>>>> the >> >>>>>> supportedHIntOptions, for the common CSV and JSON formats, we cal >> >> also >> >>>>> give >> >>>>>> a support. This is the only kind of key in formats that “do not >> >> change >> >>>>> the >> >>>>>> semantics” (somehow), what do you think about this ~ >> >>>>>> >> >>>>>> [1] >> >>>>>> >> >>>>> >> >>> >> >> >> https://github.com/danny0405/flink/commit/5d925fa16c3c553423c4b7d93001521b8e6e6bee#diff-6e569a6dd124fd2091c18e2790fb49c5 >> >>>>>> [2] >> >>>>>> >> >>>>> >> >>> >> >> >> https://github.com/apache/flink/blob/b83060dff6d403b6994b6646b3f29a374f599530/flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/table/sources/CsvTableSourceFactoryBase.java#L92 >> >>>>>> >> >>>>>> Best, >> >>>>>> Danny Chan >> >>>>>> 在 2020年3月18日 +0800 PM9:10，Timo Walther <twal...@apache.org>，写道： >> >>>>>>> Hi everyone, >> >>>>>>> >> >>>>>>> +1 to Kurt's suggestion. Let's just have it in source and sink >> >>>>> factories >> >>>>>>> for now. We can still move this method up in the future. >> Currently, >> >> I >> >>>>>>> don't see a need for catalogs or formats. Because how would you >> >> target >> >>>>> a >> >>>>>>> format in the query? >> >>>>>>> >> >>>>>>> @Danny: Can you send a link to your PoC? I'm very skeptical about >> >>>>>>> creating a new CatalogTable in planner. Actually CatalogTable >> should >> >>> be >> >>>>>>> immutable between Catalog and Factory. Because a catalog can >> return >> >>> its >> >>>>>>> own factory and fully control the instantiation. Depending on the >> >>>>>>> implementation, that means it can be possible that the catalog has >> >>>>>>> encoded more information in a concrete subclass implementing the >> >>>>>>> interface. I vote for separating the concerns of catalog >> information >> >>>>> and >> >>>>>>> hints in the factory explicitly. >> >>>>>>> >> >>>>>>> Regards, >> >>>>>>> Timo >> >>>>>>> >> >>>>>>> >> >>>>>>> On 18.03.20 05:41, Jingsong Li wrote: >> >>>>>>>> Hi, >> >>>>>>>> >> >>>>>>>> I am thinking we can provide hints to *table* related instances. >> >>>>>>>> - TableFormatFactory: of cause we need hints support, there are >> >> many >> >>>>>> format >> >>>>>>>> options in DDL too. >> >>>>>>>> - catalog and module: I don't know, maybe in future we can >> provide >> >>>>> some >> >>>>>>>> hints for them. >> >>>>>>>> >> >>>>>>>> Best, >> >>>>>>>> Jingsong Lee >> >>>>>>>> >> >>>>>>>> On Wed, Mar 18, 2020 at 12:28 PM Danny Chan < >> yuzhao....@gmail.com> >> >>>>>> wrote: >> >>>>>>>> >> >>>>>>>>> Yes, I think we should move the `supportedHintOptions` from >> >>>>>> TableFactory >> >>>>>>>>> to TableSourceFactory, and we also need to add the interface to >> >>>>>>>>> TableSinkFactory though because sink target table may also have >> >>>>> hints >> >>>>>>>>> attached. >> >>>>>>>>> >> >>>>>>>>> Best, >> >>>>>>>>> Danny Chan >> >>>>>>>>> 在 2020年3月18日 +0800 AM11:08，Kurt Young <ykt...@gmail.com>，写道： >> >>>>>>>>>> Have one question for adding `supportedHintOptions` method to >> >>>>>>>>>> `TableFactory`. It seems >> >>>>>>>>>> `TableFactory` is a base factory interface for all *table >> module* >> >>>>>> related >> >>>>>>>>>> instances, such as >> >>>>>>>>>> catalog, module, format and so on. It's not created only for >> >>>>>> *table*. Is >> >>>>>>>>> it >> >>>>>>>>>> possible to move it >> >>>>>>>>>> to `TableSourceFactory`? >> >>>>>>>>>> >> >>>>>>>>>> Best, >> >>>>>>>>>> Kurt >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Wed, Mar 18, 2020 at 10:59 AM Danny Chan < >> >>>>> yuzhao....@gmail.com> >> >>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> Thanks Timo ~ >> >>>>>>>>>>> >> >>>>>>>>>>> For the naming itself, I also think the PROPERTIES is not that >> >>>>>>>>> concise, so >> >>>>>>>>>>> +1 for OPTIONS (I had thought about that, but there are many >> >>>>>> codes in >> >>>>>>>>>>> current Flink called it properties, i.e. the >> >>>>>> DescriptorProperties, >> >>>>>>>>>>> #getSupportedProperties), let’s use OPTIONS if this is our new >> >>>>>>>>> preference. >> >>>>>>>>>>> >> >>>>>>>>>>> +1 to `Set<ConfigOption> supportedHintOptions()` because the >> >>>>>>>>> ConfigOption >> >>>>>>>>>>> can take more info. AFAIK, Spark also call their table options >> >>>>>> instead >> >>>>>>>>> of >> >>>>>>>>>>> properties. [1] >> >>>>>>>>>>> >> >>>>>>>>>>> In my local POC, I did create a new CatalogTable, and it works >> >>>>>> for >> >>>>>>>>> current >> >>>>>>>>>>> connectors well, all the DDL tables would finally yield a >> >>>>>> CatalogTable >> >>>>>>>>>>> instance and we can apply the options to that(in the >> >>>>>> CatalogSourceTable >> >>>>>>>>>>> when we generating the TableSource), the pros is that we do >> not >> >>>>>> need to >> >>>>>>>>>>> modify the codes of connectors itself. If we split the options >> >>>>>> from >> >>>>>>>>>>> CatalogTable, we may need to add some additional logic in each >> >>>>>>>>> connector >> >>>>>>>>>>> factories in order to merge these properties (and the logic >> are >> >>>>>> almost >> >>>>>>>>> the >> >>>>>>>>>>> same), what do you think about this? >> >>>>>>>>>>> >> >>>>>>>>>>> [1] >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>> >> >>>>> >> >>> >> >> >> https://docs.databricks.com/spark/latest/spark-sql/language-manual/create-table.html >> >>>>>>>>>>> >> >>>>>>>>>>> Best, >> >>>>>>>>>>> Danny Chan >> >>>>>>>>>>> 在 2020年3月17日 +0800 PM10:10，Timo Walther <twal...@apache.org >> >>>>>> ，写道： >> >>>>>>>>>>>> Hi Danny, >> >>>>>>>>>>>> >> >>>>>>>>>>>> thanks for updating the FLIP. I think your current design is >> >>>>>>>>> sufficient >> >>>>>>>>>>>> to separate hints from result-related properties. >> >>>>>>>>>>>> >> >>>>>>>>>>>> One remark to the naming itself: I would vote for calling the >> >>>>>> hints >> >>>>>>>>>>>> around table scan `OPTIONS('k'='v')`. We used the term >> >>>>>> "properties" >> >>>>>>>>> in >> >>>>>>>>>>>> the past but since we want to unify the Flink configuration >> >>>>>>>>> experience, >> >>>>>>>>>>>> we should use consistent naming and classes around >> >>>>>> `ConfigOptions`. >> >>>>>>>>>>>> >> >>>>>>>>>>>> It would be nice to use `Set<ConfigOption> >> >>>>>> supportedHintOptions();` >> >>>>>>>>> to >> >>>>>>>>>>>> start using config options instead of pure string properties. >> >>>>>> This >> >>>>>>>>> will >> >>>>>>>>>>>> also allow us to generate documentation in the future around >> >>>>>>>>> supported >> >>>>>>>>>>>> data types, ranges, etc. for options. At some point we would >> >>>>>> also >> >>>>>>>>> like >> >>>>>>>>>>>> to drop `DescriptorProperties` class. "Options" is also used >> >>>>>> in the >> >>>>>>>>>>>> documentation [1] and in the SQL/MED standard [2]. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Furthermore, I would still vote for separating CatalogTable >> >>>>>> and hint >> >>>>>>>>>>>> options. Otherwise the planner would need to create a new >> >>>>>>>>> CatalogTable >> >>>>>>>>>>>> instance which might not always be easy. We should offer them >> >>>>>> via: >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>> >> org.apache.flink.table.factories.TableSourceFactory.Context#getHints: >> >>>>>>>>>>>> ReadableConfig >> >>>>>>>>>>>> >> >>>>>>>>>>>> What do you think? >> >>>>>>>>>>>> >> >>>>>>>>>>>> Regards, >> >>>>>>>>>>>> Timo >> >>>>>>>>>>>> >> >>>>>>>>>>>> [1] >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>> >> >>>>> >> >>> >> >> >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/create.html#create-table >> >>>>>>>>>>>> [2] https://wiki.postgresql.org/wiki/SQL/MED >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> On 12.03.20 15:06, Stephan Ewen wrote: >> >>>>>>>>>>>>> @Danny sounds good. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Maybe it is worth listing all the classes of problems that >> >>>>>> you >> >>>>>>>>> want to >> >>>>>>>>>>>>> address and then look at each class and see if hints are a >> >>>>>> good >> >>>>>>>>> default >> >>>>>>>>>>>>> solution or a good optional way of simplifying things? >> >>>>>>>>>>>>> The discussion has grown a lot and it is starting to be >> >>>>> hard >> >>>>>> to >> >>>>>>>>>>> distinguish >> >>>>>>>>>>>>> the parts where everyone agrees from the parts were there >> >>>>> are >> >>>>>>>>> concerns. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:31 PM Danny Chan < >> >>>>>> danny0...@apache.org> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> Thanks Stephan ~ >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> We can remove the support for properties that may change >> >>>>>> the >> >>>>>>>>>>> semantics of >> >>>>>>>>>>>>>> query if you think that is a trouble. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> How about we support the /*+ properties() */ hint only >> >>>>> for >> >>>>>> those >> >>>>>>>>>>> optimize >> >>>>>>>>>>>>>> parameters, such as the fetch size of source or something >> >>>>>> like >> >>>>>>>>> that, >> >>>>>>>>>>> does >> >>>>>>>>>>>>>> that make sense? >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Stephan Ewen <se...@apache.org>于2020年3月12日 周四下午7:45写道： >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> I think Bowen has actually put it very well. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> (1) Hints that change semantics looks like trouble >> >>>>>> waiting to >> >>>>>>>>>>> happen. For >> >>>>>>>>>>>>>>> example Kafka offset handling should be in filters. The >> >>>>>> Kafka >> >>>>>>>>>>> source >> >>>>>>>>>>>>>> should >> >>>>>>>>>>>>>>> support predicate pushdown. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> (2) Hints should not be a workaround for current >> >>>>>> shortcomings. >> >>>>>>>>> A >> >>>>>>>>>>> lot of >> >>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>> suggested above sounds exactly like that. Working >> >>>>> around >> >>>>>>>>>>> catalog/DDL >> >>>>>>>>>>>>>>> shortcomings, missing exposure of metadata (offsets), >> >>>>>> missing >> >>>>>>>>>>> predicate >> >>>>>>>>>>>>>>> pushdown in Kafka. Abusing a feature like hints now as >> >>>>> a >> >>>>>> quick >> >>>>>>>>> fix >> >>>>>>>>>>> for >> >>>>>>>>>>>>>>> these issues, rather than fixing the root causes, will >> >>>>>> much >> >>>>>>>>> likely >> >>>>>>>>>>> bite >> >>>>>>>>>>>>>> us >> >>>>>>>>>>>>>>> back badly in the future. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>> Stephan >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:43 AM Kurt Young < >> >>>>>> ykt...@gmail.com> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> It seems this FLIP's name is somewhat misleading. >> >>>>> From >> >>>>>> my >> >>>>>>>>>>>>>> understanding, >> >>>>>>>>>>>>>>>> this FLIP is trying to >> >>>>>>>>>>>>>>>> address the dynamic parameter issue, and table hints >> >>>>>> is the >> >>>>>>>>> way >> >>>>>>>>>>> we wan >> >>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>> choose. I think we should >> >>>>>>>>>>>>>>>> be focus on "what's the right way to solve dynamic >> >>>>>> property" >> >>>>>>>>>>> instead of >> >>>>>>>>>>>>>>>> discussing "whether table >> >>>>>>>>>>>>>>>> hints can affect query semantics". >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> For now, there are two proposed ways to achieve >> >>>>> dynamic >> >>>>>>>>> property: >> >>>>>>>>>>>>>>>> 1. FLIP-110: create temporary table xx like xx with >> >>>>>> (xxx) >> >>>>>>>>>>>>>>>> 2. use custom "from t with (xxx)" syntax >> >>>>>>>>>>>>>>>> 3. "Borrow" the table hints to have a special >> >>>>>> PROPERTIES >> >>>>>>>>> hint. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> The first one didn't break anything, but the only >> >>>>>> problem i >> >>>>>>>>> see >> >>>>>>>>>>> is a >> >>>>>>>>>>>>>>> little >> >>>>>>>>>>>>>>>> more verbose than the table hint >> >>>>>>>>>>>>>>>> approach. I can imagine when someone using SQL CLI to >> >>>>>> have a >> >>>>>>>>> sql >> >>>>>>>>>>>>>>>> experience, it's quite often that >> >>>>>>>>>>>>>>>> he will modify the table property, some use cases i >> >>>>> can >> >>>>>>>>> think of: >> >>>>>>>>>>>>>>>> 1. the source contains some corrupted data, i want to >> >>>>>> turn >> >>>>>>>>> on the >> >>>>>>>>>>>>>>>> "ignore-error" flag for certain formats. >> >>>>>>>>>>>>>>>> 2. I have a kafka table and want to see some sample >> >>>>>> data >> >>>>>>>>> from the >> >>>>>>>>>>>>>>>> beginning, so i change the offset >> >>>>>>>>>>>>>>>> to "earliest", and then I want to observe the latest >> >>>>>> data >> >>>>>>>>> which >> >>>>>>>>>>> keeps >> >>>>>>>>>>>>>>>> coming in. I would write another query >> >>>>>>>>>>>>>>>> to select from the latest table. >> >>>>>>>>>>>>>>>> 3. I want to my jdbc sink flush data more eagerly >> >>>>> then >> >>>>>> i can >> >>>>>>>>>>> observe >> >>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>> data from database side. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Most of such use cases are quite ad-hoc. If every >> >>>>> time >> >>>>>> I >> >>>>>>>>> want to >> >>>>>>>>>>> have a >> >>>>>>>>>>>>>>>> different experience, i need to create >> >>>>>>>>>>>>>>>> a temporary table and then also modify my query, it >> >>>>>> doesn't >> >>>>>>>>> feel >> >>>>>>>>>>>>>> smooth. >> >>>>>>>>>>>>>>>> Embed such dynamic property into >> >>>>>>>>>>>>>>>> query would have better user experience. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Both 2 & 3 can make this happen. The cons of #2 is >> >>>>>> breaking >> >>>>>>>>> SQL >> >>>>>>>>>>>>>>> compliant, >> >>>>>>>>>>>>>>>> and for #3, it only breaks some >> >>>>>>>>>>>>>>>> unwritten rules, but we can have an explanation on >> >>>>>> that. And >> >>>>>>>>> I >> >>>>>>>>>>> really >> >>>>>>>>>>>>>>> doubt >> >>>>>>>>>>>>>>>> whether user would complain about >> >>>>>>>>>>>>>>>> this when they actually have flexible and good >> >>>>>> experience >> >>>>>>>>> using >> >>>>>>>>>>> this. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> My tendency would be #3 > #1 > #2, what do you think? >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>> Kurt >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> On Thu, Mar 12, 2020 at 1:11 PM Danny Chan < >> >>>>>>>>> yuzhao....@gmail.com >> >>>>>>>>>>>> >> >>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Thanks Aljoscha ~ >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> I agree for most of the query hints, they are >> >>>>>> optional as >> >>>>>>>>> an >> >>>>>>>>>>>>>> optimizer >> >>>>>>>>>>>>>>>>> instruction, especially for the traditional RDBMS. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> But, just like BenChao said, Flink as a computation >> >>>>>> engine >> >>>>>>>>> has >> >>>>>>>>>>> many >> >>>>>>>>>>>>>>>>> different kind of data sources, thus, dynamic >> >>>>>> parameters >> >>>>>>>>> like >> >>>>>>>>>>>>>>>> start_offest >> >>>>>>>>>>>>>>>>> can only bind to each table scope, we can not set a >> >>>>>> session >> >>>>>>>>>>> config >> >>>>>>>>>>>>>> like >> >>>>>>>>>>>>>>>>> KSQL because they are all about Kafka: >> >>>>>>>>>>>>>>>>>> SET ‘auto.offset.reset’=‘earliest’; >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Thus the most flexible way to set up these dynamic >> >>>>>> params >> >>>>>>>>> is >> >>>>>>>>>>> to bind >> >>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>> the table scope in the query when we want to >> >>>>> override >> >>>>>>>>>>> something, so >> >>>>>>>>>>>>>> we >> >>>>>>>>>>>>>>>> have >> >>>>>>>>>>>>>>>>> these solutions above (with pros and cons from my >> >>>>>> side): >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> • 1. Select * from t(offset=123) (from Timo) >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Pros: >> >>>>>>>>>>>>>>>>> - Easy to add >> >>>>>>>>>>>>>>>>> - Parameters are part of the main query >> >>>>>>>>>>>>>>>>> Cons: >> >>>>>>>>>>>>>>>>> - Not SQL compliant >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> • 2. Select * from t /*+ PROPERTIES(offset=123) */ >> >>>>>> (from >> >>>>>>>>> me) >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Pros: >> >>>>>>>>>>>>>>>>> - Easy to add >> >>>>>>>>>>>>>>>>> - SQL compliant because it is nested in the >> >>>>> comments >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Cons: >> >>>>>>>>>>>>>>>>> - Parameters are not part of the main query >> >>>>>>>>>>>>>>>>> - Cryptic syntax for new users >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> The biggest problem for hints way may be the “if >> >>>>>> hints >> >>>>>>>>> must be >> >>>>>>>>>>>>>>> optional”, >> >>>>>>>>>>>>>>>>> actually we have though about 1 for a while but >> >>>>>> aborted >> >>>>>>>>>>> because it >> >>>>>>>>>>>>>>> breaks >> >>>>>>>>>>>>>>>>> the SQL standard too much. And we replace it with >> >>>>> 2, >> >>>>>>>>> because >> >>>>>>>>>>> the >> >>>>>>>>>>>>>> hints >> >>>>>>>>>>>>>>>>> syntax do not break SQL standard(nested in >> >>>>> comments). >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> What if we have the special /*+ PROPERTIES */ hint >> >>>>>> that >> >>>>>>>>> allows >> >>>>>>>>>>>>>> override >> >>>>>>>>>>>>>>>>> some properties of table dynamically, it does not >> >>>>>> break >> >>>>>>>>>>> anything, at >> >>>>>>>>>>>>>>>> lease >> >>>>>>>>>>>>>>>>> for current Flink use cases. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Planner hints are optional just because they are >> >>>>>> naturally >> >>>>>>>>>>> enforcers >> >>>>>>>>>>>>>> of >> >>>>>>>>>>>>>>>>> the planner, most of them aim to instruct the >> >>>>>> optimizer, >> >>>>>>>>> but, >> >>>>>>>>>>> the >> >>>>>>>>>>>>>> table >> >>>>>>>>>>>>>>>>> hints is a little different, table hints can >> >>>>> specify >> >>>>>> the >> >>>>>>>>> table >> >>>>>>>>>>> meta >> >>>>>>>>>>>>>>> like >> >>>>>>>>>>>>>>>>> index column, and it is very convenient to specify >> >>>>>> table >> >>>>>>>>>>> properties. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Or shall we not call /*+ PROPERTIES(offset=123) */ >> >>>>>> table >> >>>>>>>>> hint, >> >>>>>>>>>>> we >> >>>>>>>>>>>>>> can >> >>>>>>>>>>>>>>>>> call it table dynamic parameters. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>> Danny Chan >> >>>>>>>>>>>>>>>>> 在 2020年3月11日 +0800 PM9:20，Aljoscha Krettek < >> >>>>>>>>>>> aljos...@apache.org>，写道： >> >>>>>>>>>>>>>>>>>> Hi, >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> I don't understand this discussion. Hints, as I >> >>>>>>>>> understand >> >>>>>>>>>>> them, >> >>>>>>>>>>>>>>> should >> >>>>>>>>>>>>>>>>>> work like this: >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> - hints are *optional* advice for the optimizer >> >>>>> to >> >>>>>> try >> >>>>>>>>> and >> >>>>>>>>>>> help it >> >>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>>> find a good execution strategy >> >>>>>>>>>>>>>>>>>> - hints should not change query semantics, i.e. >> >>>>>> they >> >>>>>>>>> should >> >>>>>>>>>>> not >> >>>>>>>>>>>>>>> change >> >>>>>>>>>>>>>>>>>> connector properties executing a query with >> >>>>> taking >> >>>>>> into >> >>>>>>>>>>> account the >> >>>>>>>>>>>>>>>>>> hints *must* produce the same result as executing >> >>>>>> the >> >>>>>>>>> query >> >>>>>>>>>>> without >> >>>>>>>>>>>>>>>>>> taking into account the hints >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> From these simple requirements you can derive a >> >>>>>> solution >> >>>>>>>>>>> that makes >> >>>>>>>>>>>>>>>>>> sense. I don't have a strong preference for the >> >>>>>> syntax >> >>>>>>>>> but we >> >>>>>>>>>>>>>> should >> >>>>>>>>>>>>>>>>>> strive to be in line with prior work. >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>> Aljoscha >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> On 11.03.20 11:53, Danny Chan wrote: >> >>>>>>>>>>>>>>>>>>> Thanks Timo for summarize the 3 options ~ >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> I agree with Kurt that option2 is too >> >>>>>> complicated to >> >>>>>>>>> use >> >>>>>>>>>>> because: >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> • As a Kafka topic consumer, the user must >> >>>>>> define both >> >>>>>>>>> the >> >>>>>>>>>>>>>> virtual >> >>>>>>>>>>>>>>>>> column for start offset and he must apply a special >> >>>>>> filter >> >>>>>>>>>>> predicate >> >>>>>>>>>>>>>>>> after >> >>>>>>>>>>>>>>>>> each query >> >>>>>>>>>>>>>>>>>>> • And for the internal implementation, the >> >>>>>> metadata >> >>>>>>>>> column >> >>>>>>>>>>> push >> >>>>>>>>>>>>>>> down >> >>>>>>>>>>>>>>>>> is another hard topic, each kind of message queue >> >>>>>> may have >> >>>>>>>>> its >> >>>>>>>>>>> offset >> >>>>>>>>>>>>>>>>> attribute, we need to consider the expression type >> >>>>>> for >> >>>>>>>>>>> different >> >>>>>>>>>>>>>> kind; >> >>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> source also need to recognize the constant column >> >>>>> as >> >>>>>> a >> >>>>>>>>> config >> >>>>>>>>>>>>>>>> option(which >> >>>>>>>>>>>>>>>>> is weird because usually what we pushed down is a >> >>>>>> table >> >>>>>>>>> column) >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> For option 1 and option3, I think there is no >> >>>>>>>>> difference, >> >>>>>>>>>>> option1 >> >>>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>>> also a hint syntax which is introduced in Sybase >> >>>>> and >> >>>>>>>>>>> referenced then >> >>>>>>>>>>>>>>>>> deprecated by MS-SQL in 199X years because of the >> >>>>>>>>>>> ambitiousness. >> >>>>>>>>>>>>>>>> Personally >> >>>>>>>>>>>>>>>>> I prefer /*+ */ style table hint than WITH keyword >> >>>>>> for >> >>>>>>>>> these >> >>>>>>>>>>> reasons: >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> • We do not break the standard SQL, the hints >> >>>>> are >> >>>>>>>>> nested >> >>>>>>>>>>> in SQL >> >>>>>>>>>>>>>>>>> comments >> >>>>>>>>>>>>>>>>>>> • We do not need to introduce additional WITH >> >>>>>> keyword >> >>>>>>>>>>> which may >> >>>>>>>>>>>>>>>> appear >> >>>>>>>>>>>>>>>>> in a query if we use that because a table can be >> >>>>>>>>> referenced in >> >>>>>>>>>>> all >> >>>>>>>>>>>>>>> kinds >> >>>>>>>>>>>>>>>> of >> >>>>>>>>>>>>>>>>> SQL contexts: INSERT/DELETE/FROM/JOIN …. That would >> >>>>>> make >> >>>>>>>>> our >> >>>>>>>>>>> sql >> >>>>>>>>>>>>>> query >> >>>>>>>>>>>>>>>>> break too much of the SQL from standard >> >>>>>>>>>>>>>>>>>>> • We would have uniform syntax for hints as >> >>>>> query >> >>>>>>>>> hint, one >> >>>>>>>>>>>>>> syntax >> >>>>>>>>>>>>>>>>> fits all and more easy to use >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> And here is the reason why we choose a uniform >> >>>>>> Oracle >> >>>>>>>>>>> style query >> >>>>>>>>>>>>>>>>> hint syntax which is addressed by Julian Hyde when >> >>>>> we >> >>>>>>>>> design >> >>>>>>>>>>> the >> >>>>>>>>>>>>>> syntax >> >>>>>>>>>>>>>>>>> from the Calcite community: >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> I don’t much like the MSSQL-style syntax for >> >>>>>> table >> >>>>>>>>> hints. >> >>>>>>>>>>> It >> >>>>>>>>>>>>>> adds a >> >>>>>>>>>>>>>>>>> new use of the WITH keyword that is unrelated to >> >>>>> the >> >>>>>> use of >> >>>>>>>>>>> WITH for >> >>>>>>>>>>>>>>>>> common-table expressions. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> A historical note. Microsoft SQL Server >> >>>>>> inherited its >> >>>>>>>>> hint >> >>>>>>>>>>> syntax >> >>>>>>>>>>>>>>>> from >> >>>>>>>>>>>>>>>>> Sybase a very long time ago. (See “Transact SQL >> >>>>>>>>>>> Programming”[1], page >> >>>>>>>>>>>>>>>> 632, >> >>>>>>>>>>>>>>>>> “Optimizer hints”. The book was written in 1999, >> >>>>> and >> >>>>>> covers >> >>>>>>>>>>> Microsoft >> >>>>>>>>>>>>>>> SQL >> >>>>>>>>>>>>>>>>> Server 6.5 / 7.0 and Sybase Adaptive Server 11.5, >> >>>>>> but the >> >>>>>>>>>>> syntax very >> >>>>>>>>>>>>>>>>> likely predates Sybase 4.3, from which Microsoft >> >>>>> SQL >> >>>>>>>>> Server was >> >>>>>>>>>>>>>> forked >> >>>>>>>>>>>>>>> in >> >>>>>>>>>>>>>>>>> 1993.) >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Microsoft later added the WITH keyword to make >> >>>>>> it less >> >>>>>>>>>>> ambiguous, >> >>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>> has now deprecated the syntax that does not use >> >>>>> WITH. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> They are forced to keep the syntax for >> >>>>> backwards >> >>>>>>>>>>> compatibility >> >>>>>>>>>>>>>> but >> >>>>>>>>>>>>>>>>> that doesn’t mean that we should shoulder their >> >>>>>> burden. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> I think formatted comments are the right >> >>>>>> container for >> >>>>>>>>>>> hints >> >>>>>>>>>>>>>>> because >> >>>>>>>>>>>>>>>>> it allows us to change the hint syntax without >> >>>>>> changing >> >>>>>>>>> the SQL >> >>>>>>>>>>>>>> parser, >> >>>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>> makes clear that we are at liberty to ignore hints >> >>>>>>>>> entirely. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Julian >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> [1] https://www.amazon.com/s?k=9781565924017 < >> >>>>>>>>>>>>>>>>> https://www.amazon.com/s?k=9781565924017> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>> Danny Chan >> >>>>>>>>>>>>>>>>>>> 在 2020年3月11日 +0800 PM4:03，Timo Walther < >> >>>>>>>>> twal...@apache.org >> >>>>>>>>>>>> ，写道： >> >>>>>>>>>>>>>>>>>>>> Hi Danny, >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> it is true that our DDL is not standard >> >>>>>> compliant by >> >>>>>>>>>>> using the >> >>>>>>>>>>>>>>> WITH >> >>>>>>>>>>>>>>>>>>>> clause. Nevertheless, we aim for not >> >>>>> diverging >> >>>>>> too >> >>>>>>>>> much >> >>>>>>>>>>> and the >> >>>>>>>>>>>>>>>> LIKE >> >>>>>>>>>>>>>>>>>>>> clause is an example of that. It will solve >> >>>>>> things >> >>>>>>>>> like >> >>>>>>>>>>>>>>> overwriting >> >>>>>>>>>>>>>>>>>>>> WATERMARKs, add additional/modifying >> >>>>>> properties and >> >>>>>>>>>>> inherit >> >>>>>>>>>>>>>>> schema. >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Bowen is right that Flink's DDL is mixing 3 >> >>>>>> types >> >>>>>>>>>>> definition >> >>>>>>>>>>>>>>>>> together. >> >>>>>>>>>>>>>>>>>>>> We are not the first ones that try to solve >> >>>>>> this. >> >>>>>>>>> There >> >>>>>>>>>>> is also >> >>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> SQL >> >>>>>>>>>>>>>>>>>>>> MED standard [1] that tried to tackle this >> >>>>>> problem. I >> >>>>>>>>>>> think it >> >>>>>>>>>>>>>>> was >> >>>>>>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>>>>> considered when designing the current DDL. >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Currently, I see 3 options for handling Kafka >> >>>>>>>>> offsets. I >> >>>>>>>>>>> will >> >>>>>>>>>>>>>>> give >> >>>>>>>>>>>>>>>>> some >> >>>>>>>>>>>>>>>>>>>> examples and look forward to feedback here: >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> *Option 1* Runtime and semantic parms as part >> >>>>>> of the >> >>>>>>>>>>> query >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> `SELECT * FROM MyTable('offset'=123)` >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Pros: >> >>>>>>>>>>>>>>>>>>>> - Easy to add >> >>>>>>>>>>>>>>>>>>>> - Parameters are part of the main query >> >>>>>>>>>>>>>>>>>>>> - No complicated hinting syntax >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Cons: >> >>>>>>>>>>>>>>>>>>>> - Not SQL compliant >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> *Option 2* Use metadata in query >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> `CREATE TABLE MyTable (id INT, offset AS >> >>>>>>>>>>>>>>>> SYSTEM_METADATA('offset'))` >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> `SELECT * FROM MyTable WHERE offset > >> >>>>> TIMESTAMP >> >>>>>>>>>>> '2012-12-12 >> >>>>>>>>>>>>>>>>> 12:34:22'` >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Pros: >> >>>>>>>>>>>>>>>>>>>> - SQL compliant in the query >> >>>>>>>>>>>>>>>>>>>> - Access of metadata in the DDL which is >> >>>>>> required >> >>>>>>>>> anyway >> >>>>>>>>>>>>>>>>>>>> - Regular pushdown rules apply >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Cons: >> >>>>>>>>>>>>>>>>>>>> - Users need to add an additional comlumn in >> >>>>>> the DDL >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> *Option 3*: Use hints for properties >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> ` >> >>>>>>>>>>>>>>>>>>>> SELECT * >> >>>>>>>>>>>>>>>>>>>> FROM MyTable /*+ PROPERTIES('offset'=123) */ >> >>>>>>>>>>>>>>>>>>>> ` >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Pros: >> >>>>>>>>>>>>>>>>>>>> - Easy to add >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Cons: >> >>>>>>>>>>>>>>>>>>>> - Parameters are not part of the main query >> >>>>>>>>>>>>>>>>>>>> - Cryptic syntax for new users >> >>>>>>>>>>>>>>>>>>>> - Not standard compliant. >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> If we go with this option, I would suggest to >> >>>>>> make it >> >>>>>>>>>>> available >> >>>>>>>>>>>>>>> in >> >>>>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>>>>>> separate map and don't mix it with statically >> >>>>>> defined >> >>>>>>>>>>>>>> properties. >> >>>>>>>>>>>>>>>>> Such >> >>>>>>>>>>>>>>>>>>>> that the factory can decide which properties >> >>>>>> have the >> >>>>>>>>>>> right to >> >>>>>>>>>>>>>> be >> >>>>>>>>>>>>>>>>>>>> overwritten by the hints: >> >>>>>>>>>>>>>>>>>>>> TableSourceFactory.Context.getQueryHints(): >> >>>>>>>>>>> ReadableConfig >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Regards, >> >>>>>>>>>>>>>>>>>>>> Timo >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> [1] https://en.wikipedia.org/wiki/SQL/MED >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Currently I see 3 options as a >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> On 11.03.20 07:21, Danny Chan wrote: >> >>>>>>>>>>>>>>>>>>>>> Thanks Bowen ~ >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> I agree we should somehow categorize our >> >>>>>> connector >> >>>>>>>>>>>>>> parameters. >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> For type1, I’m already preparing a solution >> >>>>>> like >> >>>>>>>>> the >> >>>>>>>>>>>>>> Confluent >> >>>>>>>>>>>>>>>>> schema registry + Avro schema inference thing, so >> >>>>>> this may >> >>>>>>>>> not >> >>>>>>>>>>> be a >> >>>>>>>>>>>>>>>> problem >> >>>>>>>>>>>>>>>>> in the near future. >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> For type3, I have some questions: >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> "SELECT * FROM mykafka WHERE offset > >> >>>>> 12pm >> >>>>>>>>> yesterday” >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> Where does the offset column come from, a >> >>>>>> virtual >> >>>>>>>>>>> column from >> >>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> table schema, you said that >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> They change >> >>>>>>>>>>>>>>>>>>>>> almost every time a query starts and have >> >>>>>> nothing >> >>>>>>>>> to >> >>>>>>>>>>> do with >> >>>>>>>>>>>>>>>>> metadata, thus >> >>>>>>>>>>>>>>>>>>>>> should not be part of table definition/DDL >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> But why you can reference it in the query, >> >>>>>> I’m >> >>>>>>>>>>> confused for >> >>>>>>>>>>>>>>> that, >> >>>>>>>>>>>>>>>>> can you elaborate a little ? >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>> Danny Chan >> >>>>>>>>>>>>>>>>>>>>> 在 2020年3月11日 +0800 PM12:52，Bowen Li < >> >>>>>>>>>>> bowenl...@gmail.com >> >>>>>>>>>>>>>>> ，写道： >> >>>>>>>>>>>>>>>>>>>>>> Thanks Danny for kicking off the effort >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> The root cause of too much manual work is >> >>>>>> Flink >> >>>>>>>>> DDL >> >>>>>>>>>>> has >> >>>>>>>>>>>>>>> mixed 3 >> >>>>>>>>>>>>>>>>> types of >> >>>>>>>>>>>>>>>>>>>>>> params together and doesn't handle each >> >>>>> of >> >>>>>> them >> >>>>>>>>> very >> >>>>>>>>>>> well. >> >>>>>>>>>>>>>>>> Below >> >>>>>>>>>>>>>>>>> are how I >> >>>>>>>>>>>>>>>>>>>>>> categorize them and corresponding >> >>>>>> solutions in my >> >>>>>>>>>>> mind: >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> - type 1: Metadata of external data, like >> >>>>>>>>> external >> >>>>>>>>>>>>>>>> endpoint/url, >> >>>>>>>>>>>>>>>>>>>>>> username/pwd, schemas, formats. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Such metadata are mostly already >> >>>>>> accessible in >> >>>>>>>>>>> external >> >>>>>>>>>>>>>>> system >> >>>>>>>>>>>>>>>>> as long as >> >>>>>>>>>>>>>>>>>>>>>> endpoints and credentials are provided. >> >>>>>> Flink can >> >>>>>>>>>>> get it >> >>>>>>>>>>>>>> thru >> >>>>>>>>>>>>>>>>> catalogs, but >> >>>>>>>>>>>>>>>>>>>>>> we haven't had many catalogs yet and thus >> >>>>>> Flink >> >>>>>>>>> just >> >>>>>>>>>>> hasn't >> >>>>>>>>>>>>>>>> been >> >>>>>>>>>>>>>>>>> able to >> >>>>>>>>>>>>>>>>>>>>>> leverage that. So the solution should be >> >>>>>> building >> >>>>>>>>>>> more >> >>>>>>>>>>>>>>>> catalogs. >> >>>>>>>>>>>>>>>>> Such >> >>>>>>>>>>>>>>>>>>>>>> params should be part of a Flink table >> >>>>>>>>>>> DDL/definition, and >> >>>>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>> overridable >> >>>>>>>>>>>>>>>>>>>>>> in any means. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> - type 2: Runtime params, like jdbc >> >>>>>> connector's >> >>>>>>>>>>> fetch size, >> >>>>>>>>>>>>>>>>> elasticsearch >> >>>>>>>>>>>>>>>>>>>>>> connector's bulk flush size. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Such params don't affect query results, >> >>>>> but >> >>>>>>>>> affect >> >>>>>>>>>>> how >> >>>>>>>>>>>>>>> results >> >>>>>>>>>>>>>>>>> are produced >> >>>>>>>>>>>>>>>>>>>>>> (eg. fast or slow, aka performance) - >> >>>>> they >> >>>>>> are >> >>>>>>>>>>> essentially >> >>>>>>>>>>>>>>>>> execution and >> >>>>>>>>>>>>>>>>>>>>>> implementation details. They change often >> >>>>>> in >> >>>>>>>>>>> exploration or >> >>>>>>>>>>>>>>>>> development >> >>>>>>>>>>>>>>>>>>>>>> stages, but not quite frequently in >> >>>>>> well-defined >> >>>>>>>>>>>>>> long-running >> >>>>>>>>>>>>>>>>> pipelines. >> >>>>>>>>>>>>>>>>>>>>>> They should always have default values >> >>>>> and >> >>>>>> can be >> >>>>>>>>>>> missing >> >>>>>>>>>>>>>> in >> >>>>>>>>>>>>>>>>> query. They >> >>>>>>>>>>>>>>>>>>>>>> can be part of a table DDL/definition, >> >>>>> but >> >>>>>> should >> >>>>>>>>>>> also be >> >>>>>>>>>>>>>>>>> replaceable in a >> >>>>>>>>>>>>>>>>>>>>>> query - *this is what table "hints" in >> >>>>>> FLIP-113 >> >>>>>>>>>>> should >> >>>>>>>>>>>>>>> cover*. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> - type 3: Semantic params, like kafka >> >>>>>> connector's >> >>>>>>>>>>> start >> >>>>>>>>>>>>>>> offset. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Such params affect query results - the >> >>>>>> semantics. >> >>>>>>>>>>> They'd >> >>>>>>>>>>>>>>> better >> >>>>>>>>>>>>>>>>> be as >> >>>>>>>>>>>>>>>>>>>>>> filter conditions in WHERE clause that >> >>>>> can >> >>>>>> be >> >>>>>>>>> pushed >> >>>>>>>>>>> down. >> >>>>>>>>>>>>>>> They >> >>>>>>>>>>>>>>>>> change >> >>>>>>>>>>>>>>>>>>>>>> almost every time a query starts and have >> >>>>>>>>> nothing to >> >>>>>>>>>>> do >> >>>>>>>>>>>>>> with >> >>>>>>>>>>>>>>>>> metadata, thus >> >>>>>>>>>>>>>>>>>>>>>> should not be part of table >> >>>>>> definition/DDL, nor >> >>>>>>>>> be >> >>>>>>>>>>>>>> persisted >> >>>>>>>>>>>>>>> in >> >>>>>>>>>>>>>>>>> catalogs. >> >>>>>>>>>>>>>>>>>>>>>> If they will, users should create views >> >>>>> to >> >>>>>> keep >> >>>>>>>>> such >> >>>>>>>>>>> params >> >>>>>>>>>>>>>>>>> around (note >> >>>>>>>>>>>>>>>>>>>>>> this is different from variable >> >>>>>> substitution). >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Take Flink-Kafka as an example. Once we >> >>>>>> get these >> >>>>>>>>>>> params >> >>>>>>>>>>>>>>> right, >> >>>>>>>>>>>>>>>>> here're the >> >>>>>>>>>>>>>>>>>>>>>> steps users need to do to develop and run >> >>>>>> a Flink >> >>>>>>>>>>> job: >> >>>>>>>>>>>>>>>>>>>>>> - configure a Flink >> >>>>>> ConfluentSchemaRegistry with >> >>>>>>>>> url, >> >>>>>>>>>>>>>>> username, >> >>>>>>>>>>>>>>>>> and password >> >>>>>>>>>>>>>>>>>>>>>> - run "SELECT * FROM mykafka WHERE offset >> >>>>>>> 12pm >> >>>>>>>>>>> yesterday" >> >>>>>>>>>>>>>>>>> (simplified >> >>>>>>>>>>>>>>>>>>>>>> timestamp) in SQL CLI, Flink >> >>>>> automatically >> >>>>>>>>> retrieves >> >>>>>>>>>>> all >> >>>>>>>>>>>>>>>>> metadata of >> >>>>>>>>>>>>>>>>>>>>>> schema, file format, etc and start the >> >>>>> job >> >>>>>>>>>>>>>>>>>>>>>> - users want to make the job read Kafka >> >>>>>> topic >> >>>>>>>>>>> faster, so it >> >>>>>>>>>>>>>>>> goes >> >>>>>>>>>>>>>>>>> as "SELECT >> >>>>>>>>>>>>>>>>>>>>>> * FROM mykafka /* faster_read_key=value*/ >> >>>>>> WHERE >> >>>>>>>>>>> offset > >> >>>>>>>>>>>>>> 12pm >> >>>>>>>>>>>>>>>>> yesterday" >> >>>>>>>>>>>>>>>>>>>>>> - done and satisfied, users submit it to >> >>>>>>>>> production >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Regarding "CREATE TABLE t LIKE with >> >>>>> (k1=v1, >> >>>>>>>>> k2=v2), >> >>>>>>>>>>> I think >> >>>>>>>>>>>>>>>> it's >> >>>>>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>>>>>>>> nice-to-have feature, but not a >> >>>>>> strategically >> >>>>>>>>>>> critical, >> >>>>>>>>>>>>>>>>> long-term solution, >> >>>>>>>>>>>>>>>>>>>>>> because >> >>>>>>>>>>>>>>>>>>>>>> 1) It may seem promising at the current >> >>>>>> stage to >> >>>>>>>>>>> solve the >> >>>>>>>>>>>>>>>>>>>>>> too-much-manual-work problem, but that's >> >>>>>> only >> >>>>>>>>>>> because Flink >> >>>>>>>>>>>>>>>>> hasn't >> >>>>>>>>>>>>>>>>>>>>>> leveraged catalogs well and handled the 3 >> >>>>>> types >> >>>>>>>>> of >> >>>>>>>>>>> params >> >>>>>>>>>>>>>>> above >> >>>>>>>>>>>>>>>>> properly. >> >>>>>>>>>>>>>>>>>>>>>> Once we get the params types right, the >> >>>>>> LIKE >> >>>>>>>>> syntax >> >>>>>>>>>>> won't >> >>>>>>>>>>>>>> be >> >>>>>>>>>>>>>>>> that >> >>>>>>>>>>>>>>>>>>>>>> important, and will be just an easier way >> >>>>>> to >> >>>>>>>>> create >> >>>>>>>>>>> tables >> >>>>>>>>>>>>>>>>> without retyping >> >>>>>>>>>>>>>>>>>>>>>> long fields like username and pwd. >> >>>>>>>>>>>>>>>>>>>>>> 2) Note that only some rare type of >> >>>>>> catalog can >> >>>>>>>>>>> store k-v >> >>>>>>>>>>>>>>>>> property pair, so >> >>>>>>>>>>>>>>>>>>>>>> table created this way often cannot be >> >>>>>>>>> persisted. In >> >>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> foreseeable >> >>>>>>>>>>>>>>>>>>>>>> future, such catalog will only be >> >>>>>> HiveCatalog, >> >>>>>>>>> and >> >>>>>>>>>>> not >> >>>>>>>>>>>>>>> everyone >> >>>>>>>>>>>>>>>>> has a Hive >> >>>>>>>>>>>>>>>>>>>>>> metastore. To be honest, without >> >>>>>> persistence, >> >>>>>>>>>>> recreating >> >>>>>>>>>>>>>>> tables >> >>>>>>>>>>>>>>>>> every time >> >>>>>>>>>>>>>>>>>>>>>> this way is still a lot of keyboard >> >>>>> typing. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Cheers, >> >>>>>>>>>>>>>>>>>>>>>> Bowen >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> On Tue, Mar 10, 2020 at 8:07 PM Kurt >> >>>>> Young >> >>>>>> < >> >>>>>>>>>>>>>> ykt...@gmail.com >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> If a specific connector want to have >> >>>>> such >> >>>>>>>>>>> parameter and >> >>>>>>>>>>>>>>> read >> >>>>>>>>>>>>>>>>> if out of >> >>>>>>>>>>>>>>>>>>>>>>> configuration, then that's fine. >> >>>>>>>>>>>>>>>>>>>>>>> If we are talking about a configuration >> >>>>>> for all >> >>>>>>>>>>> kinds of >> >>>>>>>>>>>>>>>>> sources, I would >> >>>>>>>>>>>>>>>>>>>>>>> be super careful about that. >> >>>>>>>>>>>>>>>>>>>>>>> It's true it can solve maybe 80% cases, >> >>>>>> but it >> >>>>>>>>>>> will also >> >>>>>>>>>>>>>>> make >> >>>>>>>>>>>>>>>>> the left 20% >> >>>>>>>>>>>>>>>>>>>>>>> feels weird. >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>>>> Kurt >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 11, 2020 at 11:00 AM Jark >> >>>>> Wu >> >>>>>> < >> >>>>>>>>>>>>>> imj...@gmail.com >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> Hi Kurt, >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> #3 Regarding to global offset: >> >>>>>>>>>>>>>>>>>>>>>>>> I'm not saying to use the global >> >>>>>>>>> configuration to >> >>>>>>>>>>>>>>> override >> >>>>>>>>>>>>>>>>> connector >> >>>>>>>>>>>>>>>>>>>>>>>> properties by the planner. >> >>>>>>>>>>>>>>>>>>>>>>>> But the connector should take this >> >>>>>>>>> configuration >> >>>>>>>>>>> and >> >>>>>>>>>>>>>>>>> translate into their >> >>>>>>>>>>>>>>>>>>>>>>>> client API. >> >>>>>>>>>>>>>>>>>>>>>>>> AFAIK, almost all the message queues >> >>>>>> support >> >>>>>>>>>>> eariliest >> >>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>> latest and a >> >>>>>>>>>>>>>>>>>>>>>>>> timestamp value as start point. >> >>>>>>>>>>>>>>>>>>>>>>>> So we can support 3 options for this >> >>>>>>>>>>> configuration: >> >>>>>>>>>>>>>>>>> "eariliest", "latest" >> >>>>>>>>>>>>>>>>>>>>>>>> and a timestamp string value. >> >>>>>>>>>>>>>>>>>>>>>>>> Of course, this can't solve 100% >> >>>>>> cases, but I >> >>>>>>>>>>> guess can >> >>>>>>>>>>>>>>>>> sovle 80% or 90% >> >>>>>>>>>>>>>>>>>>>>>>>> cases. >> >>>>>>>>>>>>>>>>>>>>>>>> And the remaining cases can be >> >>>>>> resolved by >> >>>>>>>>> LIKE >> >>>>>>>>>>> syntax >> >>>>>>>>>>>>>>>> which >> >>>>>>>>>>>>>>>>> I guess is >> >>>>>>>>>>>>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>>>>>>>>> very common cases. >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>>>>> Jark >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> On Wed, 11 Mar 2020 at 10:33, Kurt >> >>>>>> Young < >> >>>>>>>>>>>>>>> ykt...@gmail.com >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> Good to have such lovely >> >>>>>> discussions. I >> >>>>>>>>> also >> >>>>>>>>>>> want to >> >>>>>>>>>>>>>>>> share >> >>>>>>>>>>>>>>>>> some of my >> >>>>>>>>>>>>>>>>>>>>>>>>> opinions. >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> #1 Regarding to error handling: I >> >>>>>> also >> >>>>>>>>> think >> >>>>>>>>>>> ignore >> >>>>>>>>>>>>>>>>> invalid hints would >> >>>>>>>>>>>>>>>>>>>>>>>> be >> >>>>>>>>>>>>>>>>>>>>>>>>> dangerous, maybe >> >>>>>>>>>>>>>>>>>>>>>>>>> the simplest solution is just throw >> >>>>>> an >> >>>>>>>>>>> exception. >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> #2 Regarding to property >> >>>>>> replacement: I >> >>>>>>>>> don't >> >>>>>>>>>>> think >> >>>>>>>>>>>>>> we >> >>>>>>>>>>>>>>>>> should >> >>>>>>>>>>>>>>>>>>>>>>> constraint >> >>>>>>>>>>>>>>>>>>>>>>>>> ourself to >> >>>>>>>>>>>>>>>>>>>>>>>>> the meaning of the word "hint", and >> >>>>>>>>> forbidden >> >>>>>>>>>>> it >> >>>>>>>>>>>>>>>> modifying >> >>>>>>>>>>>>>>>>> any >> >>>>>>>>>>>>>>>>>>>>>>> properties >> >>>>>>>>>>>>>>>>>>>>>>>>> which can effect >> >>>>>>>>>>>>>>>>>>>>>>>>> query results. IMO `PROPERTIES` is >> >>>>>> one of >> >>>>>>>>> the >> >>>>>>>>>>> table >> >>>>>>>>>>>>>>>> hints, >> >>>>>>>>>>>>>>>>> and a >> >>>>>>>>>>>>>>>>>>>>>>> powerful >> >>>>>>>>>>>>>>>>>>>>>>>>> one. It can >> >>>>>>>>>>>>>>>>>>>>>>>>> modify properties located in DDL's >> >>>>>> WITH >> >>>>>>>>> block. >> >>>>>>>>>>> But I >> >>>>>>>>>>>>>>> also >> >>>>>>>>>>>>>>>>> see the harm >> >>>>>>>>>>>>>>>>>>>>>>>> that >> >>>>>>>>>>>>>>>>>>>>>>>>> if we make it >> >>>>>>>>>>>>>>>>>>>>>>>>> too flexible like change the kafka >> >>>>>> topic >> >>>>>>>>> name >> >>>>>>>>>>> with a >> >>>>>>>>>>>>>>>> hint. >> >>>>>>>>>>>>>>>>> Such use >> >>>>>>>>>>>>>>>>>>>>>>> case >> >>>>>>>>>>>>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>>>>>>>>>>> not common and >> >>>>>>>>>>>>>>>>>>>>>>>>> sounds very dangerous to me. I >> >>>>> would >> >>>>>>>>> propose >> >>>>>>>>>>> we have >> >>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>> map >> >>>>>>>>>>>>>>>>> of hintable >> >>>>>>>>>>>>>>>>>>>>>>>>> properties for each >> >>>>>>>>>>>>>>>>>>>>>>>>> connector, and should validate all >> >>>>>> passed >> >>>>>>>>> in >> >>>>>>>>>>>>>> properties >> >>>>>>>>>>>>>>>>> are actually >> >>>>>>>>>>>>>>>>>>>>>>>>> hintable. And combining with >> >>>>>>>>>>>>>>>>>>>>>>>>> #1 error handling, we can throw an >> >>>>>>>>> exception >> >>>>>>>>>>> once >> >>>>>>>>>>>>>>>> received >> >>>>>>>>>>>>>>>>> invalid >> >>>>>>>>>>>>>>>>>>>>>>>>> property. >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> #3 Regarding to global offset: I'm >> >>>>>> not sure >> >>>>>>>>>>> it's >> >>>>>>>>>>>>>>>> feasible. >> >>>>>>>>>>>>>>>>> Different >> >>>>>>>>>>>>>>>>>>>>>>>>> connectors will have totally >> >>>>>>>>>>>>>>>>>>>>>>>>> different properties to represent >> >>>>>> offset, >> >>>>>>>>> some >> >>>>>>>>>>> might >> >>>>>>>>>>>>>> be >> >>>>>>>>>>>>>>>>> timestamps, >> >>>>>>>>>>>>>>>>>>>>>>> some >> >>>>>>>>>>>>>>>>>>>>>>>>> might be string literals >> >>>>>>>>>>>>>>>>>>>>>>>>> like "earliest", and others might >> >>>>> be >> >>>>>> just >> >>>>>>>>>>> integers. >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>>>>>> Kurt >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Mar 10, 2020 at 11:46 PM >> >>>>>> Jark Wu < >> >>>>>>>>>>>>>>>> imj...@gmail.com> >> >>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone, >> >>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>> I want to jump in the discussion >> >>>>>> about >> >>>>>>>>> the >> >>>>>>>>>>> "dynamic >> >>>>>>>>>>>>>>>>> start offset" >> >>>>>>>>>>>>>>>>>>>>>>>>> problem. >> >>>>>>>>>>>>>>>>>>>>>>>>>> First of all, I share the same >> >>>>>> concern >> >>>>>>>>> with >> >>>>>>>>>>> Timo >> >>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>> Fabian, that the >> >>>>>>>>>>>>>>>>>>>>>>>>>> "start offset" affects the query >> >>>>>>>>> semantics, >> >>>>>>>>>>> i.e. >> >>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> query result. >> >>>>>>>>>>>>>>>>>>>>>>>>>> But "hints" is just used for >> >>>>>> optimization >> >>>>>>>>>>> which >> >>>>>>>>>>>>>>> should >> >>>>>>>>>>>>>>>>> affect the >> >>>>>>>>>>>>>>>>>>>>>>>> result? >> >>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>> I think the "dynamic start >> >>>>> offset" >> >>>>>> is an >> >>>>>>>>> very >> >>>>>>>>>>>>>>> important >> >>>>>>>>>>>>>>>>> usability >> >>>>>>>>>>>>>>>>>>>>>>>> problem >> >>>>>>>>>>>>>>>>>>>>>>>>>> which will be faced by many >> >>>>>> streaming >> >>>>>>>>>>> platforms. >> >>>>>>>>>>>>>>>>>>>>>>>>>> I also agree "CREATE TEMPORARY >> >>>>>> TABLE Temp >> >>>>>>>>>>> (LIKE t) >> >>>>>>>>>>>>>>> WITH >> >>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>> ('connector.startup-timestamp-millis' = >> >>>>>>>>>>>>>>>>> '1578538374471')" is verbose, >> >>>>>>>>>>>>>>>>>>>>>>>>> what >> >>>>>>>>>>>>>>>>>>>>>>>>>> if we have 10 tables to join? >> >>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>> However, what I want to propose >> >>>>>> (should >> >>>>>>>>> be >> >>>>>>>>>>> another >> >>>>>>>>>>>>>>>>> thread) is a >> >>>>>>>>>>>>>>>>>>>>>>> global >> >>>>>>>>>>>>>>>>>>>>>>>>>> configuration to reset start >> >>>>>> offsets of >> >>>>>>>>> all >> >>>>>>>>>>> the >> >>>>>>>>>>>>>>> source >> >>>>>>>>>>>>>>>>> connectors >> >>>>>>>>>>>>>>>>>>>>>>>>>> in the query session, e.g. >> >>>>>>>>>>>>>>>> "table.sources.start-offset". >> >>>>>>>>>>>>>>>>> This is >> >>>>>>>>>>>>>>>>>>>>>>>> possible >> >>>>>>>>>>>>>>>>>>>>>>>>>> now because >> >>>>>> `TableSourceFactory.Context` >> >>>>>>>>> has >> >>>>>>>>>>>>>>>>> `getConfiguration` >> >>>>>>>>>>>>>>>>>>>>>>>>>> method to get the session >> >>>>>> configuration, >> >>>>>>>>> and >> >>>>>>>>>>> use it >> >>>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>> create an >> >>>>>>>>>>>>>>>>>>>>>>>> adapted >> >>>>>>>>>>>>>>>>>>>>>>>>>> TableSource. >> >>>>>>>>>>>>>>>>>>>>>>>>>> Then we can also expose to SQL >> >>>>> CLI >> >>>>>> via >> >>>>>>>>> SET >> >>>>>>>>>>> command, >> >>>>>>>>>>>>>>>> e.g. >> >>>>>>>>>>>>>>>>> `SET >> >>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>> 'table.sources.start-offset'='earliest';`, >> >>>>>>>>>>> which is >> >>>>>>>>>>>>>>>>> pretty simple and >> >>>>>>>>>>>>>>>>>>>>>>>>>> straightforward. >> >>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>> This is very similar to KSQL's >> >>>>> `SET >> >>>>>>>>>>>>>>>>> 'auto.offset.reset'='earliest'` >> >>>>>>>>>>>>>>>>>>>>>>>> which >> >>>>>>>>>>>>>>>>>>>>>>>>>> is very helpful IMO. >> >>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>>>>>>> Jark >> >>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, 10 Mar 2020 at 22:29, >> >>>>> Timo >> >>>>>>>>> Walther < >> >>>>>>>>>>>>>>>>> twal...@apache.org> >> >>>>>>>>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Danny, >> >>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> compared to the hints, FLIP-110 >> >>>>>> is >> >>>>>>>>> fully >> >>>>>>>>>>>>>> compliant >> >>>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>> the SQL >> >>>>>>>>>>>>>>>>>>>>>>>> standard. >> >>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't think that `CREATE >> >>>>>> TEMPORARY >> >>>>>>>>> TABLE >> >>>>>>>>>>> Temp >> >>>>>>>>>>>>>>> (LIKE >> >>>>>>>>>>>>>>>>> t) WITH >> >>>>>>>>>>>>>>>>>>>>>>> (k=v)` >> >>>>>>>>>>>>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>>>>>>>>>>>>> too verbose or awkward for the >> >>>>>> power of >> >>>>>>>>>>> basically >> >>>>>>>>>>>>>>>>> changing the >> >>>>>>>>>>>>>>>>>>>>>>> entire >> >>>>>>>>>>>>>>>>>>>>>>>>>>> connector. Usually, this >> >>>>>> statement >> >>>>>>>>> would >> >>>>>>>>>>> just >> >>>>>>>>>>>>>>> precede >> >>>>>>>>>>>>>>>>> the query in >> >>>>>>>>>>>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>>>>>>>>>>>>> multiline file. So it can be >> >>>>>> change >> >>>>>>>>>>> "in-place" >> >>>>>>>>>>>>>> like >> >>>>>>>>>>>>>>>>> the hints you >> >>>>>>>>>>>>>>>>>>>>>>>>>> proposed. >> >>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> Many companies have a >> >>>>>> well-defined set >> >>>>>>>>> of >> >>>>>>>>>>> tables >> >>>>>>>>>>>>>>> that >> >>>>>>>>>>>>>>>>> should be >> >>>>>>>>>>>>>>>>>>>>>>> used. >> >>>>>>>>>>>>>>>>>>>>>>>>> It >> >>>>>>>>>>>>>>>>>>>>>>>>>>> would be dangerous if users can >> >>>>>> change >> >>>>>>>>> the >> >>>>>>>>>>> path >> >>>>>>>>>>>>>> or >> >>>>>>>>>>>>>>>>> topic in a hint. >> >>>>>>>>>>>>>>>>>>>>>>>> The >> >>>>>>>>>>>>>>>>>>>>>>>>>>> catalog/catalog manager should >> >>>>>> be the >> >>>>>>>>>>> entity that >> >>>>>>>>>>>>>>>>> controls which >> >>>>>>>>>>>>>>>>>>>>>>>> tables >> >>>>>>>>>>>>>>>>>>>>>>>>>>> exist and how they can be >> >>>>>> accessed. >> >>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> what’s the problem there if >> >>>>> we >> >>>>>> user >> >>>>>>>>> the >> >>>>>>>>>>> table >> >>>>>>>>>>>>>>> hints >> >>>>>>>>>>>>>>>>> to support >> >>>>>>>>>>>>>>>>>>>>>>>>> “start >> >>>>>>>>>>>>>>>>>>>>>>>>>>> offset”? >> >>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> IMHO it violates the meaning of >> >>>>>> a hint. >> >>>>>>>>>>> According >> >>>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>>> dictionary, >> >>>>>>>>>>>>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>>>>>>>>>>>>> hint is "a statement that >> >>>>>> expresses >> >>>>>>>>>>> indirectly >> >>>>>>>>>>>>>> what >> >>>>>>>>>>>>>>>>> one prefers not >> >>>>>>>>>>>>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>>>>>>>>>>>> say explicitly". But offsets >> >>>>> are >> >>>>>> a >> >>>>>>>>>>> property that >> >>>>>>>>>>>>>>> are >> >>>>>>>>>>>>>>>>> very explicit. >> >>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> If we go with the hint >> >>>>> approach, >> >>>>>> it >> >>>>>>>>> should >> >>>>>>>>>>> be >> >>>>>>>>>>>>>>>>> expressible in the >> >>>>>>>>>>>>>>>>>>>>>>>>>>> TableSourceFactory which >> >>>>>> properties are >> >>>>>>>>>>> supported >> >>>>>>>>>>>>>>> for >> >>>>>>>>>>>>>>>>> hinting. Or >> >>>>>>>>>>>>>>>>>>>>>>> do >> >>>>>>>>>>>>>>>>>>>>>>>>> you >> >>>>>>>>>>>>>>>>>>>>>>>>>>> plan to offer those hints in a >> >>>>>> separate >> >>>>>>>>>>>>>> Map<String, >> >>>>>>>>>>>>>>>>> String> that >> >>>>>>>>>>>>>>>>>>>>>>>> cannot >> >>>>>>>>>>>>>>>>>>>>>>>>>>> overwrite existing properties? >> >>>>> I >> >>>>>> think >> >>>>>>>>>>> this would >> >>>>>>>>>>>>>>> be >> >>>>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>>> different >> >>>>>>>>>>>>>>>>>>>>>>>>> story... >> >>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> Regards, >> >>>>>>>>>>>>>>>>>>>>>>>>>>> Timo >> >>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> On 10.03.20 10:34, Danny Chan >> >>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Timo ~ >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Personally I would say that >> >>>>>> offset > >> >>>>>>>>> 0 >> >>>>>>>>>>> and >> >>>>>>>>>>>>>> start >> >>>>>>>>>>>>>>>>> offset = 10 does >> >>>>>>>>>>>>>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>>>>>>>>>>>> have the same semantic, so from >> >>>>>> the SQL >> >>>>>>>>>>> aspect, >> >>>>>>>>>>>>>> we >> >>>>>>>>>>>>>>>> can >> >>>>>>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>>>>>>>> implement >> >>>>>>>>>>>>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>>>>>>>>>>>>> “starting offset” hint for >> >>>>> query >> >>>>>> with >> >>>>>>>>> such >> >>>>>>>>>>> a >> >>>>>>>>>>>>>>> syntax. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> And the CREATE TABLE LIKE >> >>>>>> syntax is a >> >>>>>>>>>>> DDL which >> >>>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>>> just verbose >> >>>>>>>>>>>>>>>>>>>>>>> for >> >>>>>>>>>>>>>>>>>>>>>>>>>>> defining such dynamic >> >>>>> parameters >> >>>>>> even >> >>>>>>>>> if >> >>>>>>>>>>> it could >> >>>>>>>>>>>>>>> do >> >>>>>>>>>>>>>>>>> that, shall we >> >>>>>>>>>>>>>>>>>>>>>>>>> force >> >>>>>>>>>>>>>>>>>>>>>>>>>>> users to define a temporal >> >>>>> table >> >>>>>> for >> >>>>>>>>> each >> >>>>>>>>>>> query >> >>>>>>>>>>>>>>> with >> >>>>>>>>>>>>>>>>> dynamic >> >>>>>>>>>>>>>>>>>>>>>>> params, >> >>>>>>>>>>>>>>>>>>>>>>>> I >> >>>>>>>>>>>>>>>>>>>>>>>>>>> would say it’s an awkward >> >>>>>> solution. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "Hints should give "hints" >> >>>>> but >> >>>>>> not >> >>>>>>>>>>> affect the >> >>>>>>>>>>>>>>>> actual >> >>>>>>>>>>>>>>>>> produced >> >>>>>>>>>>>>>>>>>>>>>>>>> result.” >> >>>>>>>>>>>>>>>>>>>>>>>>>>> You mentioned that multiple >> >>>>>> times and >> >>>>>>>>>>> could we >> >>>>>>>>>>>>>>> give a >> >>>>>>>>>>>>>>>>> reason, >> >>>>>>>>>>>>>>>>>>>>>>> what’s >> >>>>>>>>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>>>>>>> problem there if we user the >> >>>>>> table >> >>>>>>>>> hints to >> >>>>>>>>>>>>>> support >> >>>>>>>>>>>>>>>>> “start offset” >> >>>>>>>>>>>>>>>>>>>>>>> ? >> >>>>>>>>>>>>>>>>>>>>>>>>> From >> >>>>>>>>>>>>>>>>>>>>>>>>>>> my side I saw some benefits for >> >>>>>> that: >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> • It’s very convent to set up >> >>>>>> these >> >>>>>>>>>>> parameters, >> >>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> syntax is >> >>>>>>>>>>>>>>>>>>>>>>> very >> >>>>>>>>>>>>>>>>>>>>>>>>> much >> >>>>>>>>>>>>>>>>>>>>>>>>>>> like the DDL definition >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> • It’s scope is very clear, >> >>>>>> right on >> >>>>>>>>> the >> >>>>>>>>>>> table >> >>>>>>>>>>>>>> it >> >>>>>>>>>>>>>>>>> attathed >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> • It does not affect the >> >>>>> table >> >>>>>>>>> schema, >> >>>>>>>>>>> which >> >>>>>>>>>>>>>>> means >> >>>>>>>>>>>>>>>>> in order to >> >>>>>>>>>>>>>>>>>>>>>>>>> specify >> >>>>>>>>>>>>>>>>>>>>>>>>>>> the offset, there is no need to >> >>>>>> define >> >>>>>>>>> an >> >>>>>>>>>>> offset >> >>>>>>>>>>>>>>>>> column which is >> >>>>>>>>>>>>>>>>>>>>>>>> weird >> >>>>>>>>>>>>>>>>>>>>>>>>>>> actually, offset should never >> >>>>> be >> >>>>>> a >> >>>>>>>>> column, >> >>>>>>>>>>> it’s >> >>>>>>>>>>>>>>> more >> >>>>>>>>>>>>>>>>> like a >> >>>>>>>>>>>>>>>>>>>>>>> metadata >> >>>>>>>>>>>>>>>>>>>>>>>>> or a >> >>>>>>>>>>>>>>>>>>>>>>>>>>> start option. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> So in total, FLIP-110 uses >> >>>>> the >> >>>>>> offset >> >>>>>>>>>>> more >> >>>>>>>>>>>>>> like a >> >>>>>>>>>>>>>>>>> Hive partition >> >>>>>>>>>>>>>>>>>>>>>>>>> prune, >> >>>>>>>>>>>>>>>>>>>>>>>>>>> we can do that if we have an >> >>>>>> offset >> >>>>>>>>>>> column, but >> >>>>>>>>>>>>>>> most >> >>>>>>>>>>>>>>>>> of the case we >> >>>>>>>>>>>>>>>>>>>>>>>> do >> >>>>>>>>>>>>>>>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>>>>>>>>>>>> define that, so there is >> >>>>>> actually no >> >>>>>>>>>>> conflict or >> >>>>>>>>>>>>>>>>> overlap. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Danny Chan >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 在 2020年3月10日 +0800 >> >>>>> PM4:28，Timo >> >>>>>>>>> Walther < >> >>>>>>>>>>>>>>>>> twal...@apache.org>，写道： >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Danny, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't FLIP-110[1] solve >> >>>>>> most >> >>>>>>>>> of the >> >>>>>>>>>>>>>>> problems >> >>>>>>>>>>>>>>>>> we have around >> >>>>>>>>>>>>>>>>>>>>>>>>>> defining >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> table properties more >> >>>>>> dynamically >> >>>>>>>>>>> without >> >>>>>>>>>>>>>>> manual >> >>>>>>>>>>>>>>>>> schema work? >> >>>>>>>>>>>>>>>>>>>>>>> Also >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> offset definition is easier >> >>>>>> with >> >>>>>>>>> such a >> >>>>>>>>>>>>>> syntax. >> >>>>>>>>>>>>>>>>> They must not be >> >>>>>>>>>>>>>>>>>>>>>>>>>> defined >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> in catalog but could be >> >>>>>> temporary >> >>>>>>>>>>> tables that >> >>>>>>>>>>>>>>>>> extend from the >> >>>>>>>>>>>>>>>>>>>>>>>>> original >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> table. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> In general, we should aim >> >>>>> to >> >>>>>> keep >> >>>>>>>>> the >> >>>>>>>>>>> syntax >> >>>>>>>>>>>>>>>>> concise and don't >> >>>>>>>>>>>>>>>>>>>>>>>>> provide >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> too many ways of doing the >> >>>>>> same >> >>>>>>>>> thing. >> >>>>>>>>>>> Hints >> >>>>>>>>>>>>>>>>> should give "hints" >> >>>>>>>>>>>>>>>>>>>>>>>> but >> >>>>>>>>>>>>>>>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> affect the actual produced >> >>>>>> result. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Some connector properties >> >>>>>> might >> >>>>>>>>> also >> >>>>>>>>>>> change >> >>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> plan or schema >> >>>>>>>>>>>>>>>>>>>>>>> in >> >>>>>>>>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> future. E.g. they might >> >>>>> also >> >>>>>> define >> >>>>>>>>>>> whether a >> >>>>>>>>>>>>>>>>> table source >> >>>>>>>>>>>>>>>>>>>>>>>> supports >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> certain push-downs (e.g. >> >>>>>> predicate >> >>>>>>>>>>>>>> push-down). >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dawid is currently working >> >>>>> a >> >>>>>> draft >> >>>>>>>>>>> that might >> >>>>>>>>>>>>>>>>> makes it possible >> >>>>>>>>>>>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> expose a Kafka offset via >> >>>>> the >> >>>>>>>>> schema >> >>>>>>>>>>> such >> >>>>>>>>>>>>>> that >> >>>>>>>>>>>>>>>>> `SELECT * FROM >> >>>>>>>>>>>>>>>>>>>>>>>> Topic >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> WHERE offset > 10` would >> >>>>>> become >> >>>>>>>>>>> possible and >> >>>>>>>>>>>>>>>> could >> >>>>>>>>>>>>>>>>> be pushed >> >>>>>>>>>>>>>>>>>>>>>>> down. >> >>>>>>>>>>>>>>>>>>>>>>>>> But >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this is of course, not >> >>>>>> planned >> >>>>>>>>>>> initially. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Timo >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>> >> >>>>> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-110%3A+Support+LIKE+clause+in+CREATE+TABLE >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 10.03.20 08:34, Danny >> >>>>> Chan >> >>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Wenlong ~ >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For PROPERTIES Hint Error >> >>>>>>>>> handling >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Actually we have no way >> >>>>> to >> >>>>>>>>> figure out >> >>>>>>>>>>>>>>> whether a >> >>>>>>>>>>>>>>>>> error prone >> >>>>>>>>>>>>>>>>>>>>>>> hint >> >>>>>>>>>>>>>>>>>>>>>>>>> is a >> >>>>>>>>>>>>>>>>>>>>>>>>>>> PROPERTIES hint, for example, >> >>>>> if >> >>>>>> use >> >>>>>>>>>>> writes a >> >>>>>>>>>>>>>> hint >> >>>>>>>>>>>>>>>> like >> >>>>>>>>>>>>>>>>>>>>>>> ‘PROPERTIAS’, >> >>>>>>>>>>>>>>>>>>>>>>>>> we >> >>>>>>>>>>>>>>>>>>>>>>>>>> do >> >>>>>>>>>>>>>>>>>>>>>>>>>>> not know if this hint is a >> >>>>>> PROPERTIES >> >>>>>>>>>>> hint, what >> >>>>>>>>>>>>>> we >> >>>>>>>>>>>>>>>>> know is that >> >>>>>>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>>>>> hint >> >>>>>>>>>>>>>>>>>>>>>>>>>>> name was not registered in our >> >>>>>> Flink. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If the user writes the >> >>>>>> hint name >> >>>>>>>>>>> correctly >> >>>>>>>>>>>>>>>> (i.e. >> >>>>>>>>>>>>>>>>> PROPERTIES), >> >>>>>>>>>>>>>>>>>>>>>>> we >> >>>>>>>>>>>>>>>>>>>>>>>>> did >> >>>>>>>>>>>>>>>>>>>>>>>>>>> can enforce the validation of >> >>>>>> the hint >> >>>>>>>>>>> options >> >>>>>>>>>>>>>>> though >> >>>>>>>>>>>>>>>>> the pluggable >> >>>>>>>>>>>>>>>>>>>>>>>>>>> HintOptionChecker. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For PROPERTIES Hint >> >>>>> Option >> >>>>>> Format >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For a key value style >> >>>>> hint >> >>>>>>>>> option, >> >>>>>>>>>>> the key >> >>>>>>>>>>>>>>> can >> >>>>>>>>>>>>>>>>> be either a >> >>>>>>>>>>>>>>>>>>>>>>> simple >> >>>>>>>>>>>>>>>>>>>>>>>>>>> identifier or a string literal, >> >>>>>> which >> >>>>>>>>>>> means that >> >>>>>>>>>>>>>>> it’s >> >>>>>>>>>>>>>>>>> compatible >> >>>>>>>>>>>>>>>>>>>>>>> with >> >>>>>>>>>>>>>>>>>>>>>>>>> our >> >>>>>>>>>>>>>>>>>>>>>>>>>>> DDL syntax. We support simple >> >>>>>>>>> identifier >> >>>>>>>>>>> because >> >>>>>>>>>>>>>>> many >> >>>>>>>>>>>>>>>>> other hints >> >>>>>>>>>>>>>>>>>>>>>>> do >> >>>>>>>>>>>>>>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>>>>>>>>>>>> have the component complex keys >> >>>>>> like >> >>>>>>>>> the >> >>>>>>>>>>> table >> >>>>>>>>>>>>>>>>> properties, and we >> >>>>>>>>>>>>>>>>>>>>>>>> want >> >>>>>>>>>>>>>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>>>>>>>>>>>> unify the parse block. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Danny Chan >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 在 2020年3月10日 +0800 >> >>>>>>>>>>> PM3:19，wenlong.lwl < >> >>>>>>>>>>>>>>>>> wenlong88....@gmail.com >> >>>>>>>>>>>>>>>>>>>>>>>>>> ，写道： >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Danny, thanks for >> >>>>> the >> >>>>>>>>> proposal. >> >>>>>>>>>>> +1 for >> >>>>>>>>>>>>>>>>> adding table hints, >> >>>>>>>>>>>>>>>>>>>>>>> it >> >>>>>>>>>>>>>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>>>>>>>>>>>>> really >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a necessary feature for >> >>>>>> flink >> >>>>>>>>> sql >> >>>>>>>>>>> to >> >>>>>>>>>>>>>>>> integrate >> >>>>>>>>>>>>>>>>> with a catalog. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For error handling, I >> >>>>>> think it >> >>>>>>>>>>> would be >> >>>>>>>>>>>>>>> more >> >>>>>>>>>>>>>>>>> natural to throw >> >>>>>>>>>>>>>>>>>>>>>>> an >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception when error >> >>>>>> table hint >> >>>>>>>>>>> provided, >> >>>>>>>>>>>>>>>>> because the >> >>>>>>>>>>>>>>>>>>>>>>> properties >> >>>>>>>>>>>>>>>>>>>>>>>>> in >> >>>>>>>>>>>>>>>>>>>>>>>>>>> hint >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be merged and used >> >>>>>> to find >> >>>>>>>>>>> the table >> >>>>>>>>>>>>>>>>> factory which would >> >>>>>>>>>>>>>>>>>>>>>>>>> cause >> >>>>>>>>>>>>>>>>>>>>>>>>>> an >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception when error >> >>>>>> properties >> >>>>>>>>>>> provided, >> >>>>>>>>>>>>>>>>> right? On the other >> >>>>>>>>>>>>>>>>>>>>>>>>> hand, >> >>>>>>>>>>>>>>>>>>>>>>>>>>> unlike >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other hints which just >> >>>>>> affect >> >>>>>>>>> the >> >>>>>>>>>>> way to >> >>>>>>>>>>>>>>>>> execute the query, >> >>>>>>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>>>>>>> property >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table hint actually >> >>>>>> affects the >> >>>>>>>>>>> result of >> >>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> query, we should >> >>>>>>>>>>>>>>>>>>>>>>>>> never >> >>>>>>>>>>>>>>>>>>>>>>>>>>> ignore >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the given property >> >>>>> hints. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the format of >> >>>>>> property >> >>>>>>>>> hints, >> >>>>>>>>>>>>>>> currently, >> >>>>>>>>>>>>>>>>> in sql client, we >> >>>>>>>>>>>>>>>>>>>>>>>>>> accept >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> properties in format of >> >>>>>> string >> >>>>>>>>>>> only in >> >>>>>>>>>>>>>> DDL: >> >>>>>>>>>>>>>>>>>>>>>>>>>> 'connector.type'='kafka', >> >>>>>>>>>>>>>>>>>>>>>>>>>>> I >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> think the format of >> >>>>>> properties >> >>>>>>>>> in >> >>>>>>>>>>> hint >> >>>>>>>>>>>>>>> should >> >>>>>>>>>>>>>>>>> be the same as >> >>>>>>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>>>>>>> format we >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defined in ddl. What do >> >>>>>> you >> >>>>>>>>> think? >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bests, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wenlong Lyu >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, 10 Mar 2020 at >> >>>>>> 14:22, >> >>>>>>>>>>> Danny Chan >> >>>>>>>>>>>>>> < >> >>>>>>>>>>>>>>>>>>>>>>> yuzhao....@gmail.com> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To Weike: About the >> >>>>>> Error >> >>>>>>>>> Handing >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To be consistent with >> >>>>>> other >> >>>>>>>>> SQL >> >>>>>>>>>>>>>> vendors, >> >>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> default is to >> >>>>>>>>>>>>>>>>>>>>>>> log >> >>>>>>>>>>>>>>>>>>>>>>>>>>> warnings >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and if there is any >> >>>>>> error >> >>>>>>>>>>> (invalid hint >> >>>>>>>>>>>>>>>> name >> >>>>>>>>>>>>>>>>> or options), the >> >>>>>>>>>>>>>>>>>>>>>>>>> hint >> >>>>>>>>>>>>>>>>>>>>>>>>>>> is just >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ignored. I have >> >>>>> already >> >>>>>>>>>>> addressed in >> >>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> wiki. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To Timo: About the >> >>>>>> PROPERTIES >> >>>>>>>>>>> Table >> >>>>>>>>>>>>>> Hint >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> • The properties >> >>>>> hints >> >>>>>> is >> >>>>>>>>> also >> >>>>>>>>>>>>>> optional, >> >>>>>>>>>>>>>>>>> user can pass in an >> >>>>>>>>>>>>>>>>>>>>>>>>> option >> >>>>>>>>>>>>>>>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> override the table >> >>>>>> properties >> >>>>>>>>>>> but this >> >>>>>>>>>>>>>>> does >> >>>>>>>>>>>>>>>>> not mean it is >> >>>>>>>>>>>>>>>>>>>>>>>>>> required. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> • They should not >> >>>>>> include >> >>>>>>>>>>> semantics: >> >>>>>>>>>>>>>> does >> >>>>>>>>>>>>>>>>> the properties >> >>>>>>>>>>>>>>>>>>>>>>> belong >> >>>>>>>>>>>>>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> semantic ? I don't >> >>>>>> think so, >> >>>>>>>>> the >> >>>>>>>>>>> plan >> >>>>>>>>>>>>>>> does >> >>>>>>>>>>>>>>>>> not change right ? >> >>>>>>>>>>>>>>>>>>>>>>>> The >> >>>>>>>>>>>>>>>>>>>>>>>>>>> result >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> set may be affected, >> >>>>>> but >> >>>>>>>>> there >> >>>>>>>>>>> are >> >>>>>>>>>>>>>>> already >> >>>>>>>>>>>>>>>>> some hints do so, >> >>>>>>>>>>>>>>>>>>>>>>>> for >> >>>>>>>>>>>>>>>>>>>>>>>>>>> example, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MS-SQL MAXRECURSION >> >>>>> and >> >>>>>>>>> SNAPSHOT >> >>>>>>>>>>> hint >> >>>>>>>>>>>>>> [1] >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> • `SELECT * FROM >> >>>>> t(k=v, >> >>>>>>>>> k=v)`: >> >>>>>>>>>>> this >> >>>>>>>>>>>>>>> grammar >> >>>>>>>>>>>>>>>>> breaks the SQL >> >>>>>>>>>>>>>>>>>>>>>>>>> standard >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compared to the hints >> >>>>>>>>> way(which >> >>>>>>>>>>> is >> >>>>>>>>>>>>>>> included >> >>>>>>>>>>>>>>>>> in comments) >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> • I actually didn't >> >>>>>> found any >> >>>>>>>>>>> vendors >> >>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>> support such >> >>>>>>>>>>>>>>>>>>>>>>> grammar, >> >>>>>>>>>>>>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>>>>>>>>>>>> there >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is no way to override >> >>>>>> table >> >>>>>>>>> level >> >>>>>>>>>>>>>>>> properties >> >>>>>>>>>>>>>>>>> dynamically. For >> >>>>>>>>>>>>>>>>>>>>>>>>>> normal >> >>>>>>>>>>>>>>>>>>>>>>>>>>> RDBMS, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think there are no >> >>>>>> requests >> >>>>>>>>>>> for such >> >>>>>>>>>>>>>>>>> dynamic parameters >> >>>>>>>>>>>>>>>>>>>>>>>> because >> >>>>>>>>>>>>>>>>>>>>>>>>>>> all the >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table have the same >> >>>>>> storage >> >>>>>>>>> and >> >>>>>>>>>>>>>>> computation >> >>>>>>>>>>>>>>>>> and they are >> >>>>>>>>>>>>>>>>>>>>>>> almost >> >>>>>>>>>>>>>>>>>>>>>>>>> all >> >>>>>>>>>>>>>>>>>>>>>>>>>>> batch >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tables. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> • While Flink as a >> >>>>>>>>> computation >> >>>>>>>>>>> engine >> >>>>>>>>>>>>>> has >> >>>>>>>>>>>>>>>>> many connectors, >> >>>>>>>>>>>>>>>>>>>>>>>>>>> especially for >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some message queue >> >>>>> like >> >>>>>>>>> Kafka, >> >>>>>>>>>>> we would >> >>>>>>>>>>>>>>>> have >> >>>>>>>>>>>>>>>>> a start_offset >> >>>>>>>>>>>>>>>>>>>>>>>> which >> >>>>>>>>>>>>>>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different each time >> >>>>> we >> >>>>>> start >> >>>>>>>>> the >> >>>>>>>>>>> query, >> >>>>>>>>>>>>>>>> such >> >>>>>>>>>>>>>>>>> parameters can >> >>>>>>>>>>>>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>>>>>>>>>> be >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> persisted to catalog, >> >>>>>> because >> >>>>>>>>>>> it’s not >> >>>>>>>>>>>>>>>>> static, this is >> >>>>>>>>>>>>>>>>>>>>>>> actually >> >>>>>>>>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> background we propose >> >>>>>> the >> >>>>>>>>> table >> >>>>>>>>>>> hints >> >>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>> indicate such >> >>>>>>>>>>>>>>>>>>>>>>>> properties >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dynamically. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To Jark and Jinsong: >> >>>>> I >> >>>>>> have >> >>>>>>>>>>> removed the >> >>>>>>>>>>>>>>>>> query hints part and >> >>>>>>>>>>>>>>>>>>>>>>>>> change >> >>>>>>>>>>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> title. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>> >> >>>>> >> >>> >> >> >> https://docs.microsoft.com/en-us/sql/t-sql/queries/hints-transact-sql-query?view=sql-server-ver15 >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Danny Chan >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 在 2020年3月9日 +0800 >> >>>>>> PM5:46，Timo >> >>>>>>>>>>> Walther < >> >>>>>>>>>>>>>>>>> twal...@apache.org >> >>>>>>>>>>>>>>>>>>>>>>>> ，写道： >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Danny, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the >> >>>>>> proposal. I >> >>>>>>>>>>> agree with >> >>>>>>>>>>>>>>>> Jark >> >>>>>>>>>>>>>>>>> and Jingsong. >> >>>>>>>>>>>>>>>>>>>>>>>> Planner >> >>>>>>>>>>>>>>>>>>>>>>>>>>> hints >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and table hints are >> >>>>>>>>> orthogonal >> >>>>>>>>>>> topics >> >>>>>>>>>>>>>>>> that >> >>>>>>>>>>>>>>>>> should be >> >>>>>>>>>>>>>>>>>>>>>>> discussed >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> separately. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I share Jingsong's >> >>>>>> opinion >> >>>>>>>>>>> that we >> >>>>>>>>>>>>>>> should >> >>>>>>>>>>>>>>>>> not use planner >> >>>>>>>>>>>>>>>>>>>>>>>> hints >> >>>>>>>>>>>>>>>>>>>>>>>>>> for >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> passing connector >> >>>>>>>>> properties. >> >>>>>>>>>>> Planner >> >>>>>>>>>>>>>>>>> hints should be >> >>>>>>>>>>>>>>>>>>>>>>> optional >> >>>>>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>>>>>>> any >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time. They should >> >>>>> not >> >>>>>>>>> include >> >>>>>>>>>>>>>> semantics >> >>>>>>>>>>>>>>>>> but only affect >> >>>>>>>>>>>>>>>>>>>>>>>>> execution >> >>>>>>>>>>>>>>>>>>>>>>>>>>> time. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Connector >> >>>>> properties >> >>>>>> are an >> >>>>>>>>>>> important >> >>>>>>>>>>>>>>>> part >> >>>>>>>>>>>>>>>>> of the query >> >>>>>>>>>>>>>>>>>>>>>>>> itself. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Have you thought >> >>>>>> about >> >>>>>>>>> options >> >>>>>>>>>>> such >> >>>>>>>>>>>>>> as >> >>>>>>>>>>>>>>>>> `SELECT * FROM t(k=v, >> >>>>>>>>>>>>>>>>>>>>>>>>>> k=v)`? >> >>>>>>>>>>>>>>>>>>>>>>>>>>> How >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are other vendors >> >>>>>> deal with >> >>>>>>>>>>> this >> >>>>>>>>>>>>>>> problem? >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Timo >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 09.03.20 10:37, >> >>>>>>>>> Jingsong Li >> >>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Danny, +1 for >> >>>>>> table >> >>>>>>>>> hints, >> >>>>>>>>>>>>>> thanks >> >>>>>>>>>>>>>>>> for >> >>>>>>>>>>>>>>>>> driving. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I took a look to >> >>>>>> FLIP, >> >>>>>>>>> most >> >>>>>>>>>>> of >> >>>>>>>>>>>>>>> content >> >>>>>>>>>>>>>>>>> are talking about >> >>>>>>>>>>>>>>>>>>>>>>>> query >> >>>>>>>>>>>>>>>>>>>>>>>>>>> hints. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It is >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hard to >> >>>>> discussion >> >>>>>> and >> >>>>>>>>>>> voting. So >> >>>>>>>>>>>>>> +1 >> >>>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>> split it as Jark >> >>>>>>>>>>>>>>>>>>>>>>>> said. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Another thing is >> >>>>>>>>>>> configuration that >> >>>>>>>>>>>>>>>>> suitable to config with >> >>>>>>>>>>>>>>>>>>>>>>>>> table >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hints: >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "connector.path" >> >>>>>> and >> >>>>>>>>>>>>>>> "connector.topic", >> >>>>>>>>>>>>>>>>> Are they really >> >>>>>>>>>>>>>>>>>>>>>>>>> suitable >> >>>>>>>>>>>>>>>>>>>>>>>>>>> for >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hints? Looks >> >>>>> weird >> >>>>>> to me. >> >>>>>>>>>>> Because I >> >>>>>>>>>>>>>>>>> think these properties >> >>>>>>>>>>>>>>>>>>>>>>>> are >> >>>>>>>>>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> core of >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jingsong Lee >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 9, >> >>>>>> 2020 at >> >>>>>>>>> 5:30 >> >>>>>>>>>>> PM Jark >> >>>>>>>>>>>>>>> Wu >> >>>>>>>>>>>>>>>> < >> >>>>>>>>>>>>>>>>> imj...@gmail.com> >> >>>>>>>>>>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Danny >> >>>>> for >> >>>>>>>>> starting >> >>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> discussion. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 for this >> >>>>>> feature. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we just >> >>>>> focus >> >>>>>> on the >> >>>>>>>>>>> table >> >>>>>>>>>>>>>> hints >> >>>>>>>>>>>>>>>>> not the query hints in >> >>>>>>>>>>>>>>>>>>>>>>>>> this >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> release, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> could you split >> >>>>>> the >> >>>>>>>>> FLIP >> >>>>>>>>>>> into two >> >>>>>>>>>>>>>>>>> FLIPs? >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Because it's >> >>>>>> hard to >> >>>>>>>>> vote >> >>>>>>>>>>> on >> >>>>>>>>>>>>>>> partial >> >>>>>>>>>>>>>>>>> part of a FLIP. You >> >>>>>>>>>>>>>>>>>>>>>>> can >> >>>>>>>>>>>>>>>>>>>>>>>>>> keep >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the table >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hints proposal >> >>>>> in >> >>>>>>>>> FLIP-113 >> >>>>>>>>>>> and >> >>>>>>>>>>>>>> move >> >>>>>>>>>>>>>>>>> query hints into >> >>>>>>>>>>>>>>>>>>>>>>> another >> >>>>>>>>>>>>>>>>>>>>>>>>>> FLIP. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So that we can >> >>>>>> focuse >> >>>>>>>>> on >> >>>>>>>>>>> the >> >>>>>>>>>>>>>> table >> >>>>>>>>>>>>>>>>> hints in the FLIP. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jark >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 9 Mar >> >>>>>> 2020 at >> >>>>>>>>>>> 17:14, >> >>>>>>>>>>>>>> DONG, >> >>>>>>>>>>>>>>>>> Weike < >> >>>>>>>>>>>>>>>>>>>>>>>>>> kyled...@connect.hku.hk >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Danny, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is a >> >>>>> nice >> >>>>>>>>> feature, >> >>>>>>>>>>> +1. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> One thing I >> >>>>> am >> >>>>>>>>>>> interested in >> >>>>>>>>>>>>>> but >> >>>>>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>> mentioned in the >> >>>>>>>>>>>>>>>>>>>>>>>>> proposal >> >>>>>>>>>>>>>>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> error >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handling, as >> >>>>>> it is >> >>>>>>>>> quite >> >>>>>>>>>>> common >> >>>>>>>>>>>>>>> for >> >>>>>>>>>>>>>>>>> users to write >> >>>>>>>>>>>>>>>>>>>>>>>>>> inappropriate >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hints in >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL code, if >> >>>>>> illegal >> >>>>>>>>> or >> >>>>>>>>>>> "bad" >> >>>>>>>>>>>>>>> hints >> >>>>>>>>>>>>>>>>> are given, would the >> >>>>>>>>>>>>>>>>>>>>>>>>> system >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> simply >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ignore them >> >>>>> or >> >>>>>> throw >> >>>>>>>>>>>>>> exceptions? >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks : ) >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Weike >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar >> >>>>> 9, >> >>>>>> 2020 >> >>>>>>>>> at >> >>>>>>>>>>> 5:02 PM >> >>>>>>>>>>>>>>>> Danny >> >>>>>>>>>>>>>>>>> Chan < >> >>>>>>>>>>>>>>>>>>>>>>>>>> yuzhao....@gmail.com> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Note: >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we only >> >>>>> plan >> >>>>>> to >> >>>>>>>>>>> support table >> >>>>>>>>>>>>>>>>> hints in Flink release >> >>>>>>>>>>>>>>>>>>>>>>> 1.11, >> >>>>>>>>>>>>>>>>>>>>>>>>> so >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> please >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> focus >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mainly on >> >>>>>> the table >> >>>>>>>>>>> hints >> >>>>>>>>>>>>>> part >> >>>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>> just ignore the >> >>>>>>>>>>>>>>>>>>>>>>> planner >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hints, sorry >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that >> >>>>> mistake >> >>>>>> ~ >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Danny Chan >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 在 2020年3月9日 >> >>>>>> +0800 >> >>>>>>>>>>>>>> PM4:36，Danny >> >>>>>>>>>>>>>>>>> Chan < >> >>>>>>>>>>>>>>>>>>>>>>> yuzhao....@gmail.com >> >>>>>>>>>>>>>>>>>>>>>>>>>>> ，写道： >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >> >>>>>> fellows ~ >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would >> >>>>>> like to >> >>>>>>>>>>> propose the >> >>>>>>>>>>>>>>>>> supports for SQL hints for >> >>>>>>>>>>>>>>>>>>>>>>>> our >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Flink SQL. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We would >> >>>>>> support >> >>>>>>>>>>> hints >> >>>>>>>>>>>>>> syntax >> >>>>>>>>>>>>>>>> as >> >>>>>>>>>>>>>>>>> following: >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> select >> >>>>> /*+ >> >>>>>>>>>>> NO_HASH_JOIN, >> >>>>>>>>>>>>>>>>> RESOURCE(mem='128mb', >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parallelism='24') */ >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> emp /*+ >> >>>>>>>>> INDEX(idx1, >> >>>>>>>>>>> idx2) >> >>>>>>>>>>>>>> */ >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> join >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dept /*+ >> >>>>>>>>>>>>>> PROPERTIES(k1='v1', >> >>>>>>>>>>>>>>>>> k2='v2') */ >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>> emp.deptno >> >>>>>> = >> >>>>>>>>>>> dept.deptno >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Basically >> >>>>>> we >> >>>>>>>>> would >> >>>>>>>>>>> support >> >>>>>>>>>>>>>>> both >> >>>>>>>>>>>>>>>>> query hints(after the >> >>>>>>>>>>>>>>>>>>>>>>>>> SELECT >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> keyword) >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and table >> >>>>>>>>> hints(after >> >>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> referenced table name), for >> >>>>>>>>>>>>>>>>>>>>>>>> 1.11, >> >>>>>>>>>>>>>>>>>>>>>>>>> we >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> plan to >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support >> >>>>>> table hints >> >>>>>>>>>>> with a >> >>>>>>>>>>>>>> hint >> >>>>>>>>>>>>>>>>> probably named >> >>>>>>>>>>>>>>>>>>>>>>> PROPERTIES: >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>> table_name >> >>>>>> /*+ >> >>>>>>>>>>>>>>>>> PROPERTIES(k1='v1', k2='v2') *+/ >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I am >> >>>>>> looking >> >>>>>>>>> forward >> >>>>>>>>>>> to >> >>>>>>>>>>>>>> your >> >>>>>>>>>>>>>>>>> comments. >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> You can >> >>>>>> access >> >>>>>>>>> the >> >>>>>>>>>>> FLIP >> >>>>>>>>>>>>>> here: >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>> >> >>>>> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+SQL+and+Planner+Hints >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Danny >> >>>>> Chan >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >>> >> >> >> > >> >>
