Thanks Lincoln a lot for your quick response and clear explanation. It's clear now. So regarding "*For the retry support in FLIP-234, it only provide a 'lookup_miss' retry predicate which is to retry if can not find it*" , try to double confirm again: 1) if the dim record doesn't exist in dim table after retry finished, the output will be +[order1, id1, null], if the dim record coming in dim table before last retry, the output will be +[order1, id1, xxx], right? (assume left join here) 2) And if I expect the latest event (+[order1, id1, 12]) update the previous event (+[order1, id1, 10]), I should use regular join instead of lookup join, right?
Lincoln Lee <lincoln.8...@gmail.com> 于2022年8月11日周四 12:01写道： > @Rascal Thanks for looking at this new feature! There is a lot of content > in these two flips and we will prepare a more detailed user documentation > before the 1.16 release. > > First of all, unlike regular join, the lookup join only triggers the lookup > action (access the dimension table) through the records of the stream > table, and the updates of the dimension table itself do not actively > trigger the updating results to > downstream. > > For your example, whether retry is enabled or not, only the first > "+[order1, id1, 10]" will sent to downstream (because only one order record > has come), the new updates of dimension table "10:02 -> (id1, 12)" will not > trigger updates. > > For the retry support in FLIP-234, it only provide a 'lookup_miss' retry > predicate which is to retry if can not find it (not always equivalent to > the complete join condition [1]), and not to trigger the retry if can find > a non-null value from the dimension table. If more complex value check > logic required, a viable way is to implement a custom `AsyncRetryPredicate` > in DataStream API (as FLIP-232 introduced). > > [1]: for different connector, the index-lookup capability maybe different, > e.g., HBase can lookup on rowkey only (by default, without secondary > index), while RDBMS can provided more powerful index-lookup capabilities > let's see the lookup join example from flink document( > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#lookup-join > ) > > > if the query with a join condition "ON o.customer_id = c.id and c.country > = > 'US'" > > -- enrich each order with customer informationSELECT o.order_id, > o.total, c.country, c.zipFROM Orders AS o JOIN Customers FOR > SYSTEM_TIME AS OF o.proc_time AS c ON o.customer_id = c.id and > c.country = 'US'; > > for the dimension table in mysql, all of the columns (id & country) can be > used as index-lookup condition > > -- Customers is backed by the JDBC connector and can be used for > lookup joinsCREATE TEMPORARY TABLE Customers ( id INT, name STRING, > country STRING, zip STRING) WITH ( 'connector' = 'jdbc', 'url' = > 'jdbc:mysql://mysqlhost:3306/customerdb', 'table-name' = > 'customers'); > > > > while if the same table stored in HBase (with no secondary index), only the > 'id' column (rowkey in HBase) can be the index-lookup condition > > -- Customers is backed by the JDBC connector and can be used for > lookup joinsCREATE TEMPORARY TABLE Customers ( id INT, name STRING, > country STRING, zip STRING, > PRIMARY KEY (id) NOT ENFORCED) WITH ( 'connector' = 'hbase-xxx', > ...); > > > so, the 'lookup_miss' retry predicte may result differently in different > connectors. > > wish this can helps. > > > Best, > Lincoln Lee > > > Rascal Wu <stayras...@gmail.com> 于2022年8月11日周四 10:32写道： > > > Hi here, > > > > Sorry for digging up this old email. > > > > May I consult one question about what's the final behavior we expected > > after we enabled retryable mechanism. I viewed the FLIP-232 & FLIP-234, > but > > I didn't get a clear answer as lacking of some knowledge. > > > > Let me take a example there, let's assume we have a dim table(id, price), > > the record(id1) existed in dim table before 10:00, and then it was > updated > > in business system at 10:01, but delay updated in dim table at 10:02. the > > record timeline as follow: > > 10:00 -> (id1, 10) > > 10:00 -> the id1 records was been updated, but delayed persistency or > sync > > to dim table. > > 10:02 -> (id1, 12) > > > > And a Flink application processes an order record that will join the dim > > table at 10:01, so it will output an event +[order1, id1, 10]. > > And if we enable retry mechanism no matter it's sync or async, does it > mean > > there will be two events sink to downstream: > > 1. retract event: -[order1, id1, 10] > > 2. new event: +[order1, id1, 12] > > > > does the above all behavior is what we expected to solve the delay update > > for dimension table? > > > > > > Lincoln Lee <lincoln.8...@gmail.com> 于2022年6月7日周二 12:19写道： > > > > > Hi everyone, > > > > > > I started a vote for this FLIP [1], please vote there or ask additional > > > questions here. [2] > > > > > > [1] https://lists.apache.org/thread/bb0kqjs8co3hhmtklmwptws4fc4rz810 > > > [2] https://lists.apache.org/thread/9k1sl2519kh2n3yttwqc00p07xdfns3h > > > > > > > > > Best, > > > Lincoln Lee > > > > > > > > > Lincoln Lee <lincoln.8...@gmail.com> 于2022年6月2日周四 15:51写道： > > > > > > > Hi everyone, > > > > > > > > When reviewing the name of the hint option > 'miss-retry'='true|false', I > > > > feel the current name is not precise enough, it might be easier to > > > > understand by using the retry-predicate directly from flip-232, > > > > e.g. 'retry-predicate'='lookup-miss', which has the additional > benefit > > of > > > > extensibility(maybe more retry condition in the future). > > > > > > > > Jark & Jingsong, do you have any suggestions? If we agree with the > name > > > > 'retry-predicate' or other better candidate, I'll update the FLIP. > > > > > > > > Best, > > > > Lincoln Lee > > > > > > > > > > > > Lincoln Lee <lincoln.8...@gmail.com> 于2022年6月2日周四 11:23写道： > > > > > > > >> Hi everyone, > > > >> > > > >> I've updated the FLIP[1] based on this discussion thread that we > agree > > > to > > > >> have a single unified 'LOOKUP' hint and also a related part in > > > FLIP-221[2] > > > >> which is mainly for the necessity of the common table option > > > >> 'lookup.async'. > > > >> > > > >> The main updates are: > > > >> 1. the new unified 'LOOKUP' hint, make retry support both on sync > and > > > >> async lookup > > > >> 2. clarify the default choice of the planner for those connectors > > which > > > >> have both sync and async lookup capabilities, and how to deal with > the > > > >> query hint > > > >> 3. will add a followup issue to discuss whether to remove the > > > >> 'lookup.async' option in HBase connector. > > > >> > > > >> [1] > > > >> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems > > > >> [2] > https://lists.apache.org/thread/1vokqdnnt01yycl7y1p74g556cc8yvtq > > > >> > > > >> Best, > > > >> Lincoln Lee > > > >> > > > >> > > > >> Lincoln Lee <lincoln.8...@gmail.com> 于2022年6月1日周三 16:03写道： > > > >> > > > >>> Hi Jingsong, > > > >>> > > > >>> There will be no change for connectors with only one capability > (sync > > > or > > > >>> async). > > > >>> > > > >>> Query hint works in a best effort manner, so if users specifies a > > hint > > > >>> with invalid option, the query plan keeps unchanged, e.g., use > > > >>> LOOKUP('table'='customer', 'async'='true'), but backend lookup > source > > > only > > > >>> implemented the sync lookup function, then the async lookup hint > > takes > > > no > > > >>> effect. > > > >>> > > > >>> For these connectors which can have both capabilities of async and > > sync > > > >>> lookup, our advice for the connector developer is implementing both > > > sync > > > >>> and async interfaces if both capabilities have suitable use cases, > > and > > > the > > > >>> planner can decide which capability is the preferable one based on > > cost > > > >>> model or maybe other mechanism (another use case is exactly what > > we're > > > >>> discussing here, users can give the query hint), otherwise choose > one > > > >>> interface to implement. > > > >>> > > > >>> Also, this should be clarified for the lookup function related > APIs. > > > >>> > > > >>> Best, > > > >>> Lincoln Lee > > > >>> > > > >>> > > > >>> Jingsong Li <jingsongl...@gmail.com> 于2022年6月1日周三 15:18写道： > > > >>> > > > >>>> Hi Lincoln, > > > >>>> > > > >>>> > It's better making decisions at the query level when a connector > > has > > > >>>> both > > > >>>> capabilities. > > > >>>> > > > >>>> Can you clarify the mechanism? > > > >>>> - only sync connector: What connector developers should do > > > >>>> - only async connector: What connector developers should do > > > >>>> - both async and sync connector: What connector developers should > do > > > >>>> > > > >>>> Best, > > > >>>> Jingsong > > > >>>> > > > >>>> On Wed, Jun 1, 2022 at 2:29 PM Lincoln Lee < > lincoln.8...@gmail.com> > > > >>>> wrote: > > > >>>> > > > >>>> > Hi Jingsong, > > > >>>> > > > > >>>> > Thanks for your feedback! > > > >>>> > > > > >>>> > Yes, the existing HBase connector use an option 'lookup.async' > to > > > >>>> control > > > >>>> > its lookup source implementations that exposed to the planner, > > > >>>> however it's > > > >>>> > a private option for the HBase connector only, so it will not > > affect > > > >>>> the > > > >>>> > common API. > > > >>>> > > > > >>>> > And as discussed in the mailing thread of FLIP-221[1], we got a > > > >>>> consensus > > > >>>> > that do not make it as a common option. It's better making > > decisions > > > >>>> at the > > > >>>> > query level when a connector has both capabilities. > > > >>>> > > > > >>>> > So if everything goes well, we should discuss it whether to > > > deprecate > > > >>>> > the 'lookup.async' > > > >>>> > or not for HBase connector after the hint been done. > > > >>>> > > > > >>>> > This will be mentioned in the Compatibility part of this > FLIP[2]. > > > >>>> > > > > >>>> > WDYT? > > > >>>> > > > > >>>> > [1]: > > > https://lists.apache.org/thread/v76g8v1o9sjdho9kbzlgjyv38l2oynox > > > >>>> > [2]: > > > >>>> > > > > >>>> > > > > >>>> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems?src=contextnavpagetreemode > > > >>>> > > > > >>>> > Best, > > > >>>> > Lincoln Lee > > > >>>> > > > > >>>> > > > > >>>> > Jingsong Li <jingsongl...@gmail.com> 于2022年6月1日周三 14:11写道： > > > >>>> > > > > >>>> > > Hi Lincoln, > > > >>>> > > > > > >>>> > > The unified lookup hint is what I want. > > > >>>> > > > > > >>>> > > And I like 'async'='true|false' option. > > > >>>> > > > > > >>>> > > But there is a compatibility issue, as I remember if async is > > > >>>> currently > > > >>>> > > controlled by connector, and this may also require some API > > > changes? > > > >>>> > > > > > >>>> > > We need to have a clear story for the connector combined with > > this > > > >>>> > option: > > > >>>> > > - only sync connector > > > >>>> > > - only async connector > > > >>>> > > - both async and sync connector > > > >>>> > > > > > >>>> > > Best, > > > >>>> > > Jingsong > > > >>>> > > > > > >>>> > > On Mon, May 30, 2022 at 3:55 PM Lincoln Lee < > > > lincoln.8...@gmail.com > > > >>>> > > > > >>>> > > wrote: > > > >>>> > > > > > >>>> > > > Thanks Jark for your quick response and the consensus! > > > >>>> > > > > > > >>>> > > > And I will update the FLIP after Jingsong or other > developers > > > >>>> confirm > > > >>>> > > that > > > >>>> > > > there is no problem. > > > >>>> > > > > > > >>>> > > > Best, > > > >>>> > > > Lincoln Lee > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > Jark Wu <imj...@gmail.com> 于2022年5月30日周一 15:49写道： > > > >>>> > > > > > > >>>> > > > > Thanks for the update. > > > >>>> > > > > > > > >>>> > > > > The unified lookup hint looks good to me. > > > >>>> > > > > And thanks for explaining ALLOW_UNORDERED. > > > >>>> > > > > > > > >>>> > > > > Best, > > > >>>> > > > > Jark > > > >>>> > > > > > > > >>>> > > > > On Mon, 30 May 2022 at 15:31, Lincoln Lee < > > > >>>> lincoln.8...@gmail.com> > > > >>>> > > > wrote: > > > >>>> > > > > > > > >>>> > > > > > Hi Jark & Jingsong, > > > >>>> > > > > > > > > >>>> > > > > > Thanks for your feedback! > > > >>>> > > > > > > > > >>>> > > > > > 1.) support retry on sync lookup > > > >>>> > > > > > I also agree with supporting it, this will be useful for > > > >>>> connectors > > > >>>> > > > that > > > >>>> > > > > > don't have asynchronous lookup implementations and can > > also > > > >>>> solve > > > >>>> > the > > > >>>> > > > > ASYNC > > > >>>> > > > > > non-target problem to some extent(because the retrying > is > > > >>>> blocking > > > >>>> > > for > > > >>>> > > > > sync > > > >>>> > > > > > lookup, and may accumulate delay, but it maybe > acceptable > > > for > > > >>>> the > > > >>>> > > case > > > >>>> > > > > that > > > >>>> > > > > > most or all data want to do a delayed lookup). > > > >>>> > > > > > > > > >>>> > > > > > For the api perspective, we can do some unification. > Let's > > > >>>> think of > > > >>>> > > the > > > >>>> > > > > > whole user story for lookup join: > > > >>>> > > > > > > > > >>>> > > > > > ASYNC_LOOKUP vs SYNC_LOOKUP can share a common one: > > LOOKUP > > > by > > > >>>> > > > different > > > >>>> > > > > > hint option values: 'async'='true|false' > > > >>>> > > > > > > > > >>>> > > > > > ASYNC_LOOKUP_MISS_RETRY vs SYNC_LOOKUP_MISS_RETRY can > > share > > > >>>> the > > > >>>> > > > > > LOOKUP_MISS_RETRY with hint option: > > > 'miss-retry'='true|false' > > > >>>> > > > > > > > > >>>> > > > > > we can use one single hint LOOKUP with different hint > > > options > > > >>>> > > > > > ('async'='true|false', 'miss-retry'='true|false') to > cover > > > all > > > >>>> > > related > > > >>>> > > > > > functionalities. > > > >>>> > > > > > Compared to multiple hints with different subsets of > > > >>>> > functionality, a > > > >>>> > > > > > single hint may be easier for users to understand and > use, > > > and > > > >>>> > > specific > > > >>>> > > > > > parameters can be quickly found through documentation > > > >>>> > > > > > > > > >>>> > > > > > the support matrix will be: > > > >>>> > > > > > lookup support async retry > > > >>>> > > > > > sync w/o retry N N > > > >>>> > > > > > sync w/ retry N Y > > > >>>> > > > > > async w/o retry Y N > > > >>>> > > > > > async w/ retry Y Y > > > >>>> > > > > > > > > >>>> > > > > > and the available hint options for each mode: > > > >>>> > > > > > mode support hint options > > > >>>> > > > > > async async'='true' > > > >>>> > > > > > 'output-mode'='ordere|allow-unordered' > > > >>>> > > > > > 'capacity'='100' > > > >>>> > > > > > 'timeout'='180s' > > > >>>> > > > > > retry miss-retry'='true' > > > >>>> > > > > > 'retry-strategy'='fixed-delay' > > > >>>> > > > > > 'delay'='10s' > > > >>>> > > > > > 'max-attempts'='3' > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > 2.) 'allow-unordered' vs 'unordered' for > > > >>>> > > > > > 'table.exec.async-lookup.output-mode' > > > >>>> > > > > > Yes, make it align with DataStream Api maybe more > > intuitive, > > > >>>> but > > > >>>> > > > there's > > > >>>> > > > > > some difference in table layer that makes the > > > >>>> 'allow-unordered' > > > >>>> > > > > meaningful: > > > >>>> > > > > > updates in the pipeline need to be processed in order, > > > >>>> > > ALLOW_UNORDERED > > > >>>> > > > > > means if users allow unordered result, it will attempt > to > > > use > > > >>>> > > > > > AsyncDataStream.OutputMode.UNORDERED when it does not > > affect > > > >>>> the > > > >>>> > > > > > correctness of the result, otherwise ORDERED will be > still > > > >>>> used. > > > >>>> > > > > > > > > >>>> > > > > > Another choice is that when the user specifies unordered > > > mode, > > > >>>> > > planner > > > >>>> > > > > > throws an error when it finds that it may affect > > > correctness. > > > >>>> But > > > >>>> > > this > > > >>>> > > > is > > > >>>> > > > > > not user-friendly and is not consistent with the > customary > > > >>>> > treatment > > > >>>> > > of > > > >>>> > > > > > invalid query hints(best effort). > > > >>>> > > > > > > > > >>>> > > > > > I opened a pr > https://github.com/apache/flink/pull/19759 > > > for > > > >>>> the > > > >>>> > new > > > >>>> > > > > > option > > > >>>> > > > > > 'table.exec.async-lookup.output-mode' and also a > > discussion > > > on > > > >>>> > > > > FLINK-27625: > > > >>>> > > > > > add query hint 'ASYNC_LOOKUP' for async lookup > join(Since > > > the > > > >>>> > changes > > > >>>> > > > > were > > > >>>> > > > > > relatively minor, no new flip was created) > > > >>>> > > > > > > > > >>>> > > > > > If we can reach a consensus on the single unified hint, > > > e.g., > > > >>>> > LOOKUP, > > > >>>> > > > > then > > > >>>> > > > > > FLINK-27625 can be covered. > > > >>>> > > > > > > > > >>>> > > > > > WDYT? > > > >>>> > > > > > > > > >>>> > > > > > Best, > > > >>>> > > > > > Lincoln Lee > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > Jark Wu <imj...@gmail.com> 于2022年5月27日周五 21:04写道： > > > >>>> > > > > > > > > >>>> > > > > > > Hi Lincoln, > > > >>>> > > > > > > > > > >>>> > > > > > > Delayed Dim Join is a frequently requested feature, > it's > > > >>>> exciting > > > >>>> > > to > > > >>>> > > > > see > > > >>>> > > > > > > this feature is on the road. > > > >>>> > > > > > > The FLIP looks good to me in general. I only left some > > > minor > > > >>>> > > > comments. > > > >>>> > > > > > > > > > >>>> > > > > > > 1) support retry for sync lookup > > > >>>> > > > > > > I'm also fine with the idea proposed by Jingsong. But > > this > > > >>>> > doesn't > > > >>>> > > > > > conflict > > > >>>> > > > > > > with the FLIP and can > > > >>>> > > > > > > be future work. It would be great if we can determine > > the > > > >>>> APIs > > > >>>> > > first. > > > >>>> > > > > > > > > > >>>> > > > > > > 1) "allow-unordered" => "unordered" > > > >>>> > > > > > > I would prefer the "unordered" output mode rather than > > > >>>> > > > > "allow-unordered". > > > >>>> > > > > > > Because this fully aligns with the DataStream > behaviors > > > and > > > >>>> > avoids > > > >>>> > > > > > > confusion on the differences. > > > >>>> > > > > > > I understand the purpose that adding a "allow" prefix > > > here, > > > >>>> but I > > > >>>> > > > think > > > >>>> > > > > > the > > > >>>> > > > > > > semantic is fine to just > > > >>>> > > > > > > use "unordered" here. We didn't see any users confused > > > about > > > >>>> > > > > > > OutputMode#UNORDERED. > > > >>>> > > > > > > > > > >>>> > > > > > > Best, > > > >>>> > > > > > > Jark > > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > On Fri, 27 May 2022 at 12:58, Jingsong Li < > > > >>>> > jingsongl...@gmail.com> > > > >>>> > > > > > wrote: > > > >>>> > > > > > > > > > >>>> > > > > > > > Thanks Lincoln for your proposal. > > > >>>> > > > > > > > > > > >>>> > > > > > > > Take a look at `strategy: fixed-delay delay: > duration, > > > >>>> e.g., > > > >>>> > 10s > > > >>>> > > > > > > > max-attempts: integer, e.g., 3`. > > > >>>> > > > > > > > > > > >>>> > > > > > > > Are these options only for async? It looks like > normal > > > >>>> lookups > > > >>>> > > work > > > >>>> > > > > > too? > > > >>>> > > > > > > > > > > >>>> > > > > > > > One thing is: most of the lookup functions seem to > be > > > >>>> > synchronous > > > >>>> > > > > now? > > > >>>> > > > > > > > There are not so many asynchronous ones? > > > >>>> > > > > > > > > > > >>>> > > > > > > > Best, > > > >>>> > > > > > > > Jingsong > > > >>>> > > > > > > > > > > >>>> > > > > > > > On Tue, May 24, 2022 at 11:48 AM Lincoln Lee < > > > >>>> > > > lincoln.8...@gmail.com > > > >>>> > > > > > > > > >>>> > > > > > > > wrote: > > > >>>> > > > > > > > > > > >>>> > > > > > > > > Hi all, > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > Considering the new common table option > > > >>>> 'lookup.max-retries' > > > >>>> > > > > proposed > > > >>>> > > > > > > in > > > >>>> > > > > > > > > FLIP-221[1] which is commonly used for exception > > > >>>> handling in > > > >>>> > > > > > connector > > > >>>> > > > > > > > > implementation, we should clearly distinguish > > > >>>> > > ASYNC_LOOKUP_RETRY > > > >>>> > > > > from > > > >>>> > > > > > > it > > > >>>> > > > > > > > to > > > >>>> > > > > > > > > avoid confusing users. > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > To do so, the name ASYNC_LOOKUP_RETRY can change > to > > > >>>> > > > > > > > > ASYNC_LOOKUP_MISS_RETRY, and as the name implies, > > > >>>> restrict > > > >>>> > it > > > >>>> > > to > > > >>>> > > > > > > support > > > >>>> > > > > > > > > retries only for lookup misses and no longer > include > > > >>>> > exceptions > > > >>>> > > > > (for > > > >>>> > > > > > > sql > > > >>>> > > > > > > > > connectors, let the connector implementer decide > how > > > to > > > >>>> > handle > > > >>>> > > > > > > exceptions > > > >>>> > > > > > > > > since there are various kinds of retryable > > exceptions > > > >>>> and can > > > >>>> > > not > > > >>>> > > > > > retry > > > >>>> > > > > > > > > ones). > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > The FLIP[2] has been updated. > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > [1] > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-221%3A+Abstraction+for+lookup+source+cache+and+metric > > > >>>> > > > > > > > > [2] > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > Best, > > > >>>> > > > > > > > > Lincoln Lee > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > Lincoln Lee <lincoln.8...@gmail.com> > 于2022年5月19日周四 > > > >>>> 18:24写道： > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > Dear Flink developers, > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > I would like to open a discussion on FLIP 234 > [1] > > to > > > >>>> > support > > > >>>> > > > > > > retryable > > > >>>> > > > > > > > > > lookup join to solve delayed updates issue, as a > > > >>>> pre-work > > > >>>> > for > > > >>>> > > > > this > > > >>>> > > > > > > > > > solution, we proposed FLIP-232[2] which adds a > > > generic > > > >>>> > retry > > > >>>> > > > > > support > > > >>>> > > > > > > > for > > > >>>> > > > > > > > > > Async I/O. > > > >>>> > > > > > > > > > We prefer to offer this retry capability via > query > > > >>>> hints, > > > >>>> > > > similar > > > >>>> > > > > > to > > > >>>> > > > > > > > new > > > >>>> > > > > > > > > > join hints proposed in FLINK-27625[3] & > > FLIP-204[4]. > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > This feature is backwards compatible and > > > >>>> transparently to > > > >>>> > > > > > connectors. > > > >>>> > > > > > > > For > > > >>>> > > > > > > > > > existing connectors which implements > > > >>>> AsyncTableFunction, > > > >>>> > can > > > >>>> > > > > easily > > > >>>> > > > > > > > > enable > > > >>>> > > > > > > > > > async retry via the new join hint. > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > [1] > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems > > > >>>> > > > > > > > > > [2] > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883963 > > > >>>> > > > > > > > > > [3] > > > >>>> > > > > > > > > >>>> https://lists.apache.org/thread/jm9kg33wk9z2bvo2b0g5bp3n5kfj6qv8 > > > >>>> > > > > > > > > > [4] > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-204:+Introduce+Hash+Lookup+Join > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > Best, > > > >>>> > > > > > > > > > Lincoln Lee > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>> > > > > > >
