Hi guys,
Since there are no further comments, Kindly ping for the vote thread
[1] :D
Thanks,
Aitozi.
[1]: https://lists.apache.org/thread/7g5n2vshosom2dj9bp7x4n01okrnx4xx
Aitozi <gjying1...@gmail.com> 于2023年6月26日周一 10:31写道：
> Hi Lincoln,
> Thanks for your confirmation. I have updated the consensus to the FLIP
> doc.
> If there are no other comments, I'd like to restart the vote process in
> [1] today.
>
> https://lists.apache.org/thread/7g5n2vshosom2dj9bp7x4n01okrnx4xx
>
> Thanks,
> Aitozi.
>
> Lincoln Lee <lincoln.8...@gmail.com> 于2023年6月21日周三 22:29写道：
>
>> Hi Aitozi,
>>
>> Thanks for your updates!
>>
>> By the design of hints, the hints after select clause belong to the query
>> hints category, and this new hint is also a kind of join hints[1].
>> Join table function is one of the join type defined by flink sql joins[2],
>> all existing join hints[1] omit the 'join' keyword,
>> so I would prefer the 'ASYNC_TABLE_FUNC' (which is actually the one for
>> 'ASYNC_TABLE_FUNC_JOIN').
>>
>> Since a short Chinese holiday is coming, I suggest waiting for other
>> people's responses before continuing to vote (next monday?)
>>
>> Btw, I discussed with @fudian offline about pyflink support, there should
>> be no known issues, so you can create a subtask with pyflink support after
>> the vote passed.
>>
>> [1]
>>
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/hints/#join-hints
>> [2]
>>
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/
>>
>> Best,
>> Lincoln Lee
>>
>>
>> Aitozi <gjying1...@gmail.com> 于2023年6月18日周日 21:18写道：
>>
>> > Hi all,
>> > Sorry for the late reply, I have a discussion with Lincoln offline,
>> > mainly about
>> > the naming of the hints option. Thanks Lincoln for the valuable
>> > suggestions.
>> >
>> > Let me answer the last email inline.
>> >
>> > >For `JavaAsyncTableFunc0` in flip, can you use a scenario like RPC
>> call as
>> > an example?
>> >
>> > Sure, will give an example when adding the doc of async udtf and will
>> > update the FLIP simultaneously
>> >
>> > >For the name of this query hint, 'LATERAL' (include its internal
>> options)
>> > don't show any relevance to async, but I haven't thought of a suitable
>> name
>> > at the moment,
>> >
>> > After some discussion with Lincoln, We prefer to choose one of the
>> > `ASYNC_TABLE_FUNC` and `ASYNC_LATERAL`.
>> > Besides, In my opinion the keyword `lateral`'s use scenario is wider
>> than
>> > the table function join, but in this case we only want to config
>> > the async table function, So I'm a bit more lean to the
>> `ASYNC_TABLE_FUNC`.
>> > Looking forward to some inputs if you guys have
>> > some better suggestion on the naming.
>> >
>> > For the usage of the hints config option, I have updated the section
>> > of ConfigOption, you can refer to the FLIP
>> > for more details.
>> >
>> > >Also, the terms 'correlate join' and 'lateral join' are not the same
>> as in
>> > the current joins page[1], so maybe it would be better if we unified
>> them
>> > into 'join table function'
>> >
>> > Yes, we should unified to the 'join table function', updated.
>> >
>> > Best,
>> > Aitozi
>> >
>> > Lincoln Lee <lincoln.8...@gmail.com> 于2023年6月15日周四 09:15写道：
>> >
>> > > Hi Aitozi,
>> > >
>> > > Thanks for your reply! Gives sql users more flexibility to get
>> > > asynchronous processing capabilities via lateral join table function
>> +1
>> > for
>> > > this
>> > >
>> > > For `JavaAsyncTableFunc0` in flip, can you use a scenario like RPC
>> call
>> > as
>> > > an example?
>> > >
>> > > For the name of this query hint, 'LATERAL' (include its internal
>> options)
>> > > don't show any relevance to async, but I haven't thought of a suitable
>> > name
>> > > at the moment,
>> > > maybe we need to highlight the async keyword directly, we can also
>> see if
>> > > others have better candidates
>> > >
>> > > For the hint option "timeout = '180s'" should be "'timeout' = '180s'",
>> > > seems a typo in the flip. And use upper case for all keywords in sql
>> > > examples.
>> > > Also, the terms 'correlate join' and 'lateral join' are not the same
>> as
>> > in
>> > > the current joins page[1], so maybe it would be better if we unified
>> them
>> > > into 'join table function'
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#table-function
>> > >
>> > > Best,
>> > > Lincoln Lee
>> > >
>> > >
>> > > Aitozi <gjying1...@gmail.com> 于2023年6月14日周三 16:11写道：
>> > >
>> > > > Hi Lincoln
>> > > >
>> > > > Very thanks for your valuable question. I will try to answer
>> your
>> > > > questions inline.
>> > > >
>> > > > >Does the async udtf bring any additional benefits besides a
>> > > > lighter implementation？
>> > > >
>> > > > IMO, async udtf is more than a lighter implementation. It can act
>> as a
>> > > > general way for sql users to use the async operator. And they don't
>> > have
>> > > to
>> > > > bind the async function with a table (a LookupTable), and they are
>> not
>> > > > forced to join on an equality join condition, and they can use it
>> to do
>> > > > more than enrich data.
>> > > >
>> > > > The async lookup join is more like a subset/specific usage of async
>> > udtf.
>> > > > The specific version has more opportunity to be optimized (like push
>> > > down)
>> > > > is acceptable. Async table function should be categorized to
>> > used-defined
>> > > > function.
>> > > >
>> > > > >Should users
>> > > >
>> > > > migrate to the lookup source when they encounter similar
>> requirements
>> > or
>> > > >
>> > > > problems, or should we develop an additional set of similar
>> mechanisms？
>> > > >
>> > > > As I clarified above, the lookup join is a specific usage of async
>> > udtf.
>> > > So
>> > > > it deserves more refined optimization like caching / retryable. But
>> it
>> > > may
>> > > > not all
>> > > >
>> > > > suitable for the async udtf. As function, it can be deterministic/or
>> > > > non-deterministic. So caching is not suitable, and we also do not
>> have
>> > a
>> > > > common cache for the udf now. So I think optimization like
>> > caching/retry
>> > > > should be handed over to the function implementor.
>> > > >
>> > > > > the newly added query hint need a different name that
>> > > > can be easier related to the lateral operation as the current join
>> > > hints[5]
>> > > > do.
>> > > >
>> > > >
>> > > > What about using LATERAL?
>> > > >
>> > > > as below
>> > > >
>> > > > SELECT /*+ LATERAL('output-mode' = 'ordered', 'capacity' = '200',
>> > > timeout =
>> > > > '180s') */ a, c1, c2
>> > > >
>> > > > FROM T1
>> > > >
>> > > > LEFT JOIN lateral TABLE (async_split(b)) AS T(c1, c2) ON true
>> > > >
>> > > > >For the async func example, since the target scenario is an
>> external
>> > io
>> > > > operation, it's better to add the `close` method to actively release
>> > > > resources as a good example for users
>> > > >
>> > > >
>> > > > Make sense to me, will update the FLIP
>> > > >
>> > > > Best,
>> > > >
>> > > > Aitozi.
>> > > >
>> > > > Lincoln Lee <lincoln.8...@gmail.com> 于2023年6月14日周三 14:24写道：
>> > > >
>> > > > > Hi Aitozi,
>> > > > >
>> > > > > Sorry for the lately reply here! Supports async
>> > > > udtf(`AsyncTableFunction`)
>> > > > > directly in sql seems like an attractive feature, but there're two
>> > > issues
>> > > > > that need to be addressed before we can be sure to add it:
>> > > > > 1. As mentioned in the flip[1], the current lookup function can
>> > already
>> > > > > implement the requirements, but it requires implementing an extra
>> > > > > `LookupTableSource` and explicitly declaring the table schema
>> (which
>> > > can
>> > > > > help implementers the various push-down optimizations supported by
>> > the
>> > > > > planner). Does the async udtf bring any additional benefits
>> besides a
>> > > > > lighter implementation?
>> > > > > 2. FLIP-221[2] abstracts a reusable cache and metric
>> infrastructure
>> > for
>> > > > > lookup sources, which are important to improve performance and
>> > > > > observability for high overhead external io scenarios, how do we
>> > > > integrate
>> > > > > and reuse these capabilities after introducing async udtf? Should
>> > users
>> > > > > migrate to the lookup source when they encounter similar
>> requirements
>> > > or
>> > > > > problems, or should we develop an additional set of similar
>> > mechanisms?
>> > > > (a
>> > > > > similarly case: FLIP-234[3] introduced the retryable capability
>> for
>> > > > lookup
>> > > > > join)
>> > > > >
>> > > > > For the flip itself,
>> > > > > 1. Considering the 'options' is already used as the dynamic table
>> > > > > options[4] in flink, the newly added query hint need a different
>> name
>> > > > that
>> > > > > can be easier related to the lateral operation as the current join
>> > > > hints[5]
>> > > > > do.
>> > > > > 2. For the async func example, since the target scenario is an
>> > external
>> > > > io
>> > > > > operation, it's better to add the `close` method to actively
>> release
>> > > > > resources as a good example for users. Also in terms of the
>> > determinism
>> > > > of
>> > > > > a function, it is important to remind users that unless the
>> behavior
>> > of
>> > > > the
>> > > > > function is deterministic, it needs to be explicitly declared as
>> > > > > non-deterministic.
>> > > > >
>> > > > > [1].
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-313%3A+Add+support+of+User+Defined+AsyncTableFunction?src=contextnavpagetreemode
>> > > > > [2].
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-221%3A+Abstraction+for+lookup+source+cache+and+metric?src=contextnavpagetreemode
>> > > > > [3].
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems?src=contextnavpagetreemode
>> > > > > [4].
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL?src=contextnavpagetreemode
>> > > > > [5].
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job?src=contextnavpagetreemode
>> > > > >
>> > > > > Best,
>> > > > > Lincoln Lee
>> > > > >
>> > > > >
>> > > > > Aitozi <gjying1...@gmail.com> 于2023年6月13日周二 11:30写道：
>> > > > >
>> > > > > > Get your meaning now, thanks :)
>> > > > > >
>> > > > > > Best,
>> > > > > > Aitozi.
>> > > > > >
>> > > > > > Feng Jin <jinfeng1...@gmail.com> 于2023年6月13日周二 11:16写道：
>> > > > > >
>> > > > > > > Hi Aitozi,
>> > > > > > >
>> > > > > > > Sorry for the confusing description.
>> > > > > > >
>> > > > > > > What I meant was that if we need to remind users about tire
>> > safety
>> > > > > > issues,
>> > > > > > > we should introduce the new UDTF interface instead of
>> executing
>> > the
>> > > > > > > original UDTF asynchronously. Therefore, I agree with
>> introducing
>> > > the
>> > > > > > > AsyncTableFunction.
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Feng
>> > > > > > >
>> > > > > > > On Tue, Jun 13, 2023 at 10:42 AM Aitozi <gjying1...@gmail.com
>> >
>> > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Feng,
>> > > > > > > > Thanks for your question. We do not provide a way to
>> switch
>> > > the
>> > > > > > UDTF
>> > > > > > > > between sync and async way,
>> > > > > > > > So there should be no thread safety problem here.
>> > > > > > > >
>> > > > > > > > Best,
>> > > > > > > > Aitozi
>> > > > > > > >
>> > > > > > > > Feng Jin <jinfeng1...@gmail.com> 于2023年6月13日周二 10:31写道：
>> > > > > > > >
>> > > > > > > > > Hi Aitozi, We do need to remind users about thread safety
>> > > issues.
>> > > > > > Thank
>> > > > > > > > you
>> > > > > > > > > for your efforts on this FLIP. I have no further
>> questions.
>> > > > > > > > > Best, Feng
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Tue, Jun 13, 2023 at 6:05 AM Jing Ge
>> > > > <j...@ververica.com.invalid
>> > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi Aitozi,
>> > > > > > > > > >
>> > > > > > > > > > Thanks for taking care of that part. I have no other
>> > concern.
>> > > > > > > > > >
>> > > > > > > > > > Best regards,
>> > > > > > > > > > Jing
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Mon, Jun 12, 2023 at 5:38 PM Aitozi <
>> > gjying1...@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > BTW, If there are no other more blocking issue /
>> > comments,
>> > > I
>> > > > > > would
>> > > > > > > > like
>> > > > > > > > > > to
>> > > > > > > > > > > start a VOTE in another thread this wednesday 6.14
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks,
>> > > > > > > > > > > Aitozi.
>> > > > > > > > > > >
>> > > > > > > > > > > Aitozi <gjying1...@gmail.com> 于2023年6月12日周一 23:34写道：
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi, Jing,
>> > > > > > > > > > > > Thanks for your explanation. I get your point
>> now.
>> > > > > > > > > > > >
>> > > > > > > > > > > > For the performance part, I think it's a good idea
>> to
>> > run
>> > > > > with
>> > > > > > > > > > returning
>> > > > > > > > > > > a
>> > > > > > > > > > > > big table case, the memory consumption
>> > > > > > > > > > > > should be a point to be taken care about. Because in
>> > the
>> > > > > > ordered
>> > > > > > > > > mode,
>> > > > > > > > > > > the
>> > > > > > > > > > > > head element in buffer may affect the
>> > > > > > > > > > > > total memory consumption.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > Aitozi.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Jing Ge <j...@ververica.com.invalid> 于2023年6月12日周一
>> > > > 20:28写道：
>> > > > > > > > > > > >
>> > > > > > > > > > > >> Hi Aitozi,
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Which key will be used for lookup is not an issue,
>> > only
>> > > > one
>> > > > > > row
>> > > > > > > > will
>> > > > > > > > > > be
>> > > > > > > > > > > >> required for each key in order to enrich it. True,
>> it
>> > > > > depends
>> > > > > > on
>> > > > > > > > the
>> > > > > > > > > > > >> implementation whether multiple rows or single row
>> for
>> > > > each
>> > > > > > key
>> > > > > > > > will
>> > > > > > > > > > be
>> > > > > > > > > > > >> returned. However, for the lookup & enrichment
>> > scenario,
>> > > > one
>> > > > > > > > row/key
>> > > > > > > > > > is
>> > > > > > > > > > > >> recommended, otherwise, like I mentioned
>> previously,
>> > > > > > enrichment
>> > > > > > > > > won't
>> > > > > > > > > > > >> work.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> I am a little bit concerned about returning a big
>> > table
>> > > > for
>> > > > > > each
>> > > > > > > > > key,
>> > > > > > > > > > > >> since
>> > > > > > > > > > > >> it will take the async call longer to return and
>> need
>> > > more
>> > > > > > > memory.
>> > > > > > > > > The
>> > > > > > > > > > > >> performance tests should cover this scenario. This
>> is
>> > > not
>> > > > a
>> > > > > > > > blocking
>> > > > > > > > > > > issue
>> > > > > > > > > > > >> for this FLIP.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Best regards,
>> > > > > > > > > > > >> Jing
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <
>> > > > > gjying1...@gmail.com>
>> > > > > > > > > wrote:
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> > Hi Jing,
>> > > > > > > > > > > >> > I means the join key is not necessary to be
>> the
>> > > > > primary
>> > > > > > > key
>> > > > > > > > or
>> > > > > > > > > > > >> unique
>> > > > > > > > > > > >> > index of the database.
>> > > > > > > > > > > >> > In this situation, we may queried out multi rows
>> for
>> > > one
>> > > > > > join
>> > > > > > > > > key. I
>> > > > > > > > > > > >> think
>> > > > > > > > > > > >> > that's why the
>> > > > > > > > > > > >> > LookupFunction#lookup will return a collection of
>> > > > RowData.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > BTW, I think the behavior of lookup join will not
>> > > affect
>> > > > > the
>> > > > > > > > > > semantic
>> > > > > > > > > > > of
>> > > > > > > > > > > >> > the async udtf.
>> > > > > > > > > > > >> > We use the Async TableFunction here and the table
>> > > > function
>> > > > > > can
>> > > > > > > > > > collect
>> > > > > > > > > > > >> > multiple rows.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Thanks,
>> > > > > > > > > > > >> > Atiozi.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Jing Ge <j...@ververica.com.invalid>
>> 于2023年6月10日周六
>> > > > > 00:15写道：
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > > Hi Aitozi,
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > The keyRow used in this case contains all
>> keys[1].
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > Best regards,
>> > > > > > > > > > > >> > > Jing
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > [1]
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <
>> > > > > > gjying1...@gmail.com
>> > > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > Hi Jing,
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > The performance test is added to the
>> FLIP.
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > As I know, The lookup join can return
>> multi
>> > > > rows,
>> > > > > > it
>> > > > > > > > > > depends
>> > > > > > > > > > > on
>> > > > > > > > > > > >> > > > whether the join key
>> > > > > > > > > > > >> > > > is the primary key of the external database
>> or
>> > > not.
>> > > > > The
>> > > > > > > > > `lookup`
>> > > > > > > > > > > [1]
>> > > > > > > > > > > >> > will
>> > > > > > > > > > > >> > > > return a collection of
>> > > > > > > > > > > >> > > > joined result, and each of them will be
>> > collected
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > [1]:
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > Thanks,
>> > > > > > > > > > > >> > > > Aitozi.
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > Jing Ge <j...@ververica.com.invalid>
>> > 于2023年6月9日周五
>> > > > > > > 17:05写道：
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > Hi Aitozi,
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > Thanks for the feedback. Looking forward to
>> > the
>> > > > > > > > performance
>> > > > > > > > > > > tests.
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > Afaik, lookup returns one row for each key
>> [1]
>> > > > [2].
>> > > > > > > > > > > Conceptually,
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> > > > > lookup function is used to enrich column(s)
>> > from
>> > > > the
>> > > > > > > > > dimension
>> > > > > > > > > > > >> table.
>> > > > > > > > > > > >> > > If,
>> > > > > > > > > > > >> > > > > for the given key, there will be more than
>> one
>> > > > row,
>> > > > > > > there
>> > > > > > > > > will
>> > > > > > > > > > > be
>> > > > > > > > > > > >> no
>> > > > > > > > > > > >> > > way
>> > > > > > > > > > > >> > > > to
>> > > > > > > > > > > >> > > > > know which row will be used to enrich the
>> key.
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > [1]
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
>> > > > > > > > > > > >> > > > > [2]
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > Best regards,
>> > > > > > > > > > > >> > > > > Jing
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <
>> > > > > > > > gjying1...@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > > >> wrote:
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > Hi Jing
>> > > > > > > > > > > >> > > > > > Thanks for your good questions. I
>> have
>> > > > updated
>> > > > > > the
>> > > > > > > > > > example
>> > > > > > > > > > > >> to
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> > > > > FLIP.
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > Only one row for each lookup
>> > > > > > > > > > > >> > > > > > lookup can also return multi rows, based
>> on
>> > > the
>> > > > > > query
>> > > > > > > > > > result.
>> > > > > > > > > > > >> [1]
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > [1]:
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > If we use async calls with lateral
>> join,
>> > my
>> > > > gut
>> > > > > > > > feeling
>> > > > > > > > > is
>> > > > > > > > > > > >> > > > > > that we might have many more async calls
>> > than
>> > > > > lookup
>> > > > > > > > > join. I
>> > > > > > > > > > > am
>> > > > > > > > > > > >> not
>> > > > > > > > > > > >> > > > > really
>> > > > > > > > > > > >> > > > > > sure if we will be facing potential
>> issues
>> > in
>> > > > this
>> > > > > > > case
>> > > > > > > > or
>> > > > > > > > > > > not.
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > IMO, the work pattern is similar to the
>> > lookup
>> > > > > > > function,
>> > > > > > > > > for
>> > > > > > > > > > > >> each
>> > > > > > > > > > > >> > row
>> > > > > > > > > > > >> > > > > from
>> > > > > > > > > > > >> > > > > > the left table,
>> > > > > > > > > > > >> > > > > > it will evaluate the eval method once, so
>> > the
>> > > > > async
>> > > > > > > call
>> > > > > > > > > > > numbers
>> > > > > > > > > > > >> > will
>> > > > > > > > > > > >> > > > not
>> > > > > > > > > > > >> > > > > > change.
>> > > > > > > > > > > >> > > > > > and the maximum calls in flight is
>> limited
>> > by
>> > > > the
>> > > > > > > Async
>> > > > > > > > > > > >> operators
>> > > > > > > > > > > >> > > > buffer
>> > > > > > > > > > > >> > > > > > capacity
>> > > > > > > > > > > >> > > > > > which will be controlled by the option.
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > BTW, for the naming of these option, I
>> > updated
>> > > > the
>> > > > > > > FLIP
>> > > > > > > > > > about
>> > > > > > > > > > > >> this
>> > > > > > > > > > > >> > > you
>> > > > > > > > > > > >> > > > > can
>> > > > > > > > > > > >> > > > > > refer to
>> > > > > > > > > > > >> > > > > > the section of "ConfigOption" and
>> "Rejected
>> > > > > > > > Alternatives"
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > In the end, for the performance
>> evaluation,
>> > > I'd
>> > > > > like
>> > > > > > > to
>> > > > > > > > do
>> > > > > > > > > > > some
>> > > > > > > > > > > >> > tests
>> > > > > > > > > > > >> > > > and
>> > > > > > > > > > > >> > > > > > will update it to the FLIP doc
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > Thanks,
>> > > > > > > > > > > >> > > > > > Aitozi.
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > Jing Ge <j...@ververica.com.invalid>
>> > > > 于2023年6月9日周五
>> > > > > > > > > 07:23写道：
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > Hi Aitozi,
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > > > Thanks for the clarification. The code
>> > > example
>> > > > > > looks
>> > > > > > > > > > > >> > interesting. I
>> > > > > > > > > > > >> > > > > would
>> > > > > > > > > > > >> > > > > > > suggest adding them into the FLIP. The
>> > > > > description
>> > > > > > > > with
>> > > > > > > > > > code
>> > > > > > > > > > > >> > > examples
>> > > > > > > > > > > >> > > > > > will
>> > > > > > > > > > > >> > > > > > > help readers understand the motivation
>> and
>> > > how
>> > > > > to
>> > > > > > > use
>> > > > > > > > > it.
>> > > > > > > > > > > >> Afaiac,
>> > > > > > > > > > > >> > > it
>> > > > > > > > > > > >> > > > > is a
>> > > > > > > > > > > >> > > > > > > valid feature for Flink users.
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > > > As we knew, lookup join is based on
>> > temporal
>> > > > > join,
>> > > > > > > > i.e.
>> > > > > > > > > > FOR
>> > > > > > > > > > > >> > > > SYSTEM_TIME
>> > > > > > > > > > > >> > > > > > AS
>> > > > > > > > > > > >> > > > > > > OF which is also used in your code
>> > example.
>> > > > > > Temporal
>> > > > > > > > > join
>> > > > > > > > > > > >> > performs
>> > > > > > > > > > > >> > > > the
>> > > > > > > > > > > >> > > > > > > lookup based on the processing time
>> match.
>> > > > Only
>> > > > > > one
>> > > > > > > > row
>> > > > > > > > > > for
>> > > > > > > > > > > >> each
>> > > > > > > > > > > >> > > > > > > lookup(afaiu, I need to check the
>> source
>> > > code
>> > > > to
>> > > > > > > > double
>> > > > > > > > > > > >> confirm)
>> > > > > > > > > > > >> > > will
>> > > > > > > > > > > >> > > > > > > return for further enrichment. One the
>> > other
>> > > > > hand,
>> > > > > > > > > lateral
>> > > > > > > > > > > >> join
>> > > > > > > > > > > >> > > will
>> > > > > > > > > > > >> > > > > have
>> > > > > > > > > > > >> > > > > > > sub-queries correlated with every
>> > individual
>> > > > > value
>> > > > > > > of
>> > > > > > > > > the
>> > > > > > > > > > > >> > reference
>> > > > > > > > > > > >> > > > > table
>> > > > > > > > > > > >> > > > > > > from the preceding part of the query
>> and
>> > > each
>> > > > > sub
>> > > > > > > > query
>> > > > > > > > > > will
>> > > > > > > > > > > >> > return
>> > > > > > > > > > > >> > > > > > > multiple rows. If we use async calls
>> with
>> > > > > lateral
>> > > > > > > > join,
>> > > > > > > > > my
>> > > > > > > > > > > gut
>> > > > > > > > > > > >> > > > feeling
>> > > > > > > > > > > >> > > > > is
>> > > > > > > > > > > >> > > > > > > that we might have many more async
>> calls
>> > > than
>> > > > > > lookup
>> > > > > > > > > > join. I
>> > > > > > > > > > > >> am
>> > > > > > > > > > > >> > not
>> > > > > > > > > > > >> > > > > > really
>> > > > > > > > > > > >> > > > > > > sure if we will be facing potential
>> issues
>> > > in
>> > > > > this
>> > > > > > > > case
>> > > > > > > > > or
>> > > > > > > > > > > >> not.
>> > > > > > > > > > > >> > > > > Possible
>> > > > > > > > > > > >> > > > > > > issues I can think of now e.g. too many
>> > PRC
>> > > > > calls,
>> > > > > > > too
>> > > > > > > > > > many
>> > > > > > > > > > > >> async
>> > > > > > > > > > > >> > > > calls
>> > > > > > > > > > > >> > > > > > > processing, the sub query will return a
>> > > table
>> > > > > > which
>> > > > > > > > > might
>> > > > > > > > > > be
>> > > > > > > > > > > >> > (too)
>> > > > > > > > > > > >> > > > big,
>> > > > > > > > > > > >> > > > > > and
>> > > > > > > > > > > >> > > > > > > might cause performance issues. I would
>> > > > suggest
>> > > > > > > > > preparing
>> > > > > > > > > > > some
>> > > > > > > > > > > >> > use
>> > > > > > > > > > > >> > > > > cases
>> > > > > > > > > > > >> > > > > > > and running some performance tests to
>> > check
>> > > > it.
>> > > > > > > These
>> > > > > > > > > are
>> > > > > > > > > > my
>> > > > > > > > > > > >> > > concerns
>> > > > > > > > > > > >> > > > > > about
>> > > > > > > > > > > >> > > > > > > using async calls with lateral join and
>> > I'd
>> > > > like
>> > > > > > to
>> > > > > > > > > share
>> > > > > > > > > > > with
>> > > > > > > > > > > >> > you,
>> > > > > > > > > > > >> > > > > happy
>> > > > > > > > > > > >> > > > > > > to discuss with you and hear different
>> > > > opinions,
>> > > > > > > > > hopefully
>> > > > > > > > > > > the
>> > > > > > > > > > > >> > > > > > > discussion could help me understand it
>> > more
>> > > > > > deeply.
>> > > > > > > > > Please
>> > > > > > > > > > > >> > correct
>> > > > > > > > > > > >> > > me
>> > > > > > > > > > > >> > > > > if
>> > > > > > > > > > > >> > > > > > I
>> > > > > > > > > > > >> > > > > > > am wrong.
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > > > Best regards,
>> > > > > > > > > > > >> > > > > > > Jing
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <
>> > > > > > > > > > gjying1...@gmail.com
>> > > > > > > > > > > >
>> > > > > > > > > > > >> > > wrote:
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > > > > Hi Mason,
>> > > > > > > > > > > >> > > > > > > > Thanks for your input. I think
>> if we
>> > > > > support
>> > > > > > > the
>> > > > > > > > > > user
>> > > > > > > > > > > >> > defined
>> > > > > > > > > > > >> > > > > async
>> > > > > > > > > > > >> > > > > > > > table function,
>> > > > > > > > > > > >> > > > > > > > user will be able to use it to hold a
>> > > batch
>> > > > > data
>> > > > > > > > then
>> > > > > > > > > > > >> handle it
>> > > > > > > > > > > >> > > at
>> > > > > > > > > > > >> > > > > one
>> > > > > > > > > > > >> > > > > > > time
>> > > > > > > > > > > >> > > > > > > > in the customized function.
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > > > AsyncSink is meant for the sink
>> > operator.
>> > > I
>> > > > > have
>> > > > > > > not
>> > > > > > > > > > > figure
>> > > > > > > > > > > >> out
>> > > > > > > > > > > >> > > how
>> > > > > > > > > > > >> > > > > to
>> > > > > > > > > > > >> > > > > > > > integrate in this case.
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > > > Thanks,
>> > > > > > > > > > > >> > > > > > > > Atiozi.
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > > > Mason Chen <mas.chen6...@gmail.com>
>> > > > > > 于2023年6月8日周四
>> > > > > > > > > > 12:40写道：
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > Hi Aitozi,
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > I think it makes sense to make it
>> > easier
>> > > > for
>> > > > > > SQL
>> > > > > > > > > users
>> > > > > > > > > > > to
>> > > > > > > > > > > >> > make
>> > > > > > > > > > > >> > > > > RPCs.
>> > > > > > > > > > > >> > > > > > Do
>> > > > > > > > > > > >> > > > > > > > you
>> > > > > > > > > > > >> > > > > > > > > think your proposal can extend to
>> the
>> > > > > ability
>> > > > > > to
>> > > > > > > > > batch
>> > > > > > > > > > > >> data
>> > > > > > > > > > > >> > for
>> > > > > > > > > > > >> > > > the
>> > > > > > > > > > > >> > > > > > > RPC?
>> > > > > > > > > > > >> > > > > > > > > This is also another common
>> strategy
>> > to
>> > > > > > increase
>> > > > > > > > > > > >> throughput.
>> > > > > > > > > > > >> > > > Also,
>> > > > > > > > > > > >> > > > > > have
>> > > > > > > > > > > >> > > > > > > > you
>> > > > > > > > > > > >> > > > > > > > > considered solving this a bit
>> > > differently
>> > > > by
>> > > > > > > > > > leveraging
>> > > > > > > > > > > >> > Flink's
>> > > > > > > > > > > >> > > > > > > > AsyncSink?
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > Best,
>> > > > > > > > > > > >> > > > > > > > > Mason
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM
>> Aitozi
>> > <
>> > > > > > > > > > > >> gjying1...@gmail.com>
>> > > > > > > > > > > >> > > > > wrote:
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > One more thing for discussion:
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > In our internal implementation,
>> we
>> > > reuse
>> > > > > the
>> > > > > > > > > option
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > `table.exec.async-lookup.buffer-capacity`
>> > > > > > and
>> > > > > > > > > > > >> > > > > > > > > >
>> `table.exec.async-lookup.timeout` to
>> > > > > config
>> > > > > > > > > > > >> > > > > > > > > > the async udtf. Do you think we
>> > should
>> > > > add
>> > > > > > two
>> > > > > > > > > extra
>> > > > > > > > > > > >> option
>> > > > > > > > > > > >> > > to
>> > > > > > > > > > > >> > > > > > > > > distinguish
>> > > > > > > > > > > >> > > > > > > > > > from the lookup option such as
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > >
>> > > `table.exec.async-udtf.buffer-capacity`
>> > > > > > > > > > > >> > > > > > > > > > `table.exec.async-udtf.timeout`
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > Best,
>> > > > > > > > > > > >> > > > > > > > > > Aitozi.
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > Aitozi <gjying1...@gmail.com>
>> > > > > 于2023年6月5日周一
>> > > > > > > > > 12:20写道：
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > Hi Jing,
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > > what is the difference
>> > between
>> > > > the
>> > > > > > RPC
>> > > > > > > > > call
>> > > > > > > > > > or
>> > > > > > > > > > > >> > query
>> > > > > > > > > > > >> > > > you
>> > > > > > > > > > > >> > > > > > > > > mentioned
>> > > > > > > > > > > >> > > > > > > > > > > and the lookup in a very
>> > > > > > > > > > > >> > > > > > > > > > > general way
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > I think the RPC call or query
>> > > service
>> > > > is
>> > > > > > > quite
>> > > > > > > > > > > >> similar to
>> > > > > > > > > > > >> > > the
>> > > > > > > > > > > >> > > > > > > lookup
>> > > > > > > > > > > >> > > > > > > > > > join.
>> > > > > > > > > > > >> > > > > > > > > > > But lookup join should work
>> > > > > > > > > > > >> > > > > > > > > > > with `LookupTableSource`.
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > Let's see how we can perform an
>> > > async
>> > > > > RPC
>> > > > > > > call
>> > > > > > > > > > with
>> > > > > > > > > > > >> > lookup
>> > > > > > > > > > > >> > > > > join:
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > (1) Implement an
>> > AsyncTableFunction
>> > > > with
>> > > > > > RPC
>> > > > > > > > > call
>> > > > > > > > > > > >> logic.
>> > > > > > > > > > > >> > > > > > > > > > > (2) Implement a
>> > `LookupTableSource`
>> > > > > > > connector
>> > > > > > > > > run
>> > > > > > > > > > > with
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> > > > > async
>> > > > > > > > > > > >> > > > > > > udtf
>> > > > > > > > > > > >> > > > > > > > > > > defined in (1).
>> > > > > > > > > > > >> > > > > > > > > > > (3) Then define a DDL of this
>> look
>> > > up
>> > > > > > table
>> > > > > > > in
>> > > > > > > > > SQL
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > CREATE TEMPORARY TABLE
>> Customers (
>> > > > > > > > > > > >> > > > > > > > > > > id INT,
>> > > > > > > > > > > >> > > > > > > > > > > name STRING,
>> > > > > > > > > > > >> > > > > > > > > > > country STRING,
>> > > > > > > > > > > >> > > > > > > > > > > zip STRING
>> > > > > > > > > > > >> > > > > > > > > > > ) WITH (
>> > > > > > > > > > > >> > > > > > > > > > > 'connector' = 'custom'
>> > > > > > > > > > > >> > > > > > > > > > > );
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > (4) Run with the query as
>> below:
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total,
>> > > c.country,
>> > > > > > c.zip
>> > > > > > > > > > > >> > > > > > > > > > > FROM Orders AS o
>> > > > > > > > > > > >> > > > > > > > > > > JOIN Customers FOR
>> SYSTEM_TIME
>> > AS
>> > > OF
>> > > > > > > > > o.proc_time
>> > > > > > > > > > > AS
>> > > > > > > > > > > >> c
>> > > > > > > > > > > >> > > > > > > > > > > ON o.customer_id = c.id;
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > This example is from doc
>> > > > > > > > > > > >> > > > > > > > > > > <
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
>> > > > > > > > > > > >> > > > > > > > > > >.You
>> > > > > > > > > > > >> > > > > > > > > > > can image the look up process
>> as
>> > an
>> > > > > async
>> > > > > > > RPC
>> > > > > > > > > call
>> > > > > > > > > > > >> > process.
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > Let's see how we can perform an
>> > > async
>> > > > > RPC
>> > > > > > > call
>> > > > > > > > > > with
>> > > > > > > > > > > >> > lateral
>> > > > > > > > > > > >> > > > > join:
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > (1) Implement an
>> > AsyncTableFunction
>> > > > with
>> > > > > > RPC
>> > > > > > > > > call
>> > > > > > > > > > > >> logic.
>> > > > > > > > > > > >> > > > > > > > > > > (2) Run query with
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > Create function f1 as '...' ;
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total,
>> > > c.country,
>> > > > > > c.zip
>> > > > > > > > > FROM
>> > > > > > > > > > > >> Orders
>> > > > > > > > > > > >> > > > > lateral
>> > > > > > > > > > > >> > > > > > > > table
>> > > > > > > > > > > >> > > > > > > > > > > (f1(order_id)) as T(...);
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > As you can see, the lateral
>> join
>> > > > version
>> > > > > > is
>> > > > > > > > more
>> > > > > > > > > > > >> simple
>> > > > > > > > > > > >> > and
>> > > > > > > > > > > >> > > > > > > intuitive
>> > > > > > > > > > > >> > > > > > > > > to
>> > > > > > > > > > > >> > > > > > > > > > > users. Users do not have to
>> wrap a
>> > > > > > > > > > > >> > > > > > > > > > > LookupTableSource for the
>> purpose
>> > of
>> > > > > using
>> > > > > > > > async
>> > > > > > > > > > > udtf.
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > In the end, We can also see the
>> > user
>> > > > > > defined
>> > > > > > > > > async
>> > > > > > > > > > > >> table
>> > > > > > > > > > > >> > > > > function
>> > > > > > > > > > > >> > > > > > > is
>> > > > > > > > > > > >> > > > > > > > an
>> > > > > > > > > > > >> > > > > > > > > > > enhancement of the current
>> lateral
>> > > > table
>> > > > > > > join
>> > > > > > > > > > > >> > > > > > > > > > > which only supports sync
>> lateral
>> > > join
>> > > > > now.
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > Best,
>> > > > > > > > > > > >> > > > > > > > > > > Aitozi.
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > Jing Ge
>> > <j...@ververica.com.invalid
>> > > >
>> > > > > > > > > 于2023年6月2日周五
>> > > > > > > > > > > >> > 19:37写道：
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> Hi Aitozi,
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> Thanks for the update. Just
>> out
>> > of
>> > > > > > > curiosity,
>> > > > > > > > > > what
>> > > > > > > > > > > is
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> > > > > > > difference
>> > > > > > > > > > > >> > > > > > > > > > >> between the RPC call or query
>> you
>> > > > > > mentioned
>> > > > > > > > and
>> > > > > > > > > > the
>> > > > > > > > > > > >> > lookup
>> > > > > > > > > > > >> > > > in
>> > > > > > > > > > > >> > > > > a
>> > > > > > > > > > > >> > > > > > > very
>> > > > > > > > > > > >> > > > > > > > > > >> general way? Since Lateral
>> join
>> > is
>> > > > used
>> > > > > > in
>> > > > > > > > the
>> > > > > > > > > > > FLIP.
>> > > > > > > > > > > >> Is
>> > > > > > > > > > > >> > > > there
>> > > > > > > > > > > >> > > > > > any
>> > > > > > > > > > > >> > > > > > > > > > special
>> > > > > > > > > > > >> > > > > > > > > > >> thought for that? Sorry for
>> > asking
>> > > so
>> > > > > > many
>> > > > > > > > > > > questions.
>> > > > > > > > > > > >> > The
>> > > > > > > > > > > >> > > > FLIP
>> > > > > > > > > > > >> > > > > > > > > contains
>> > > > > > > > > > > >> > > > > > > > > > >> limited information to
>> understand
>> > > the
>> > > > > > > > > motivation.
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> Best regards,
>> > > > > > > > > > > >> > > > > > > > > > >> Jing
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM
>> > > > Aitozi <
>> > > > > > > > > > > >> > > gjying1...@gmail.com
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > wrote:
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > Hi Jing,
>> > > > > > > > > > > >> > > > > > > > > > >> > I have updated the
>> proposed
>> > > > > changes
>> > > > > > > to
>> > > > > > > > > the
>> > > > > > > > > > > >> FLIP.
>> > > > > > > > > > > >> > > IMO,
>> > > > > > > > > > > >> > > > > > lookup
>> > > > > > > > > > > >> > > > > > > > has
>> > > > > > > > > > > >> > > > > > > > > > its
>> > > > > > > > > > > >> > > > > > > > > > >> > clear
>> > > > > > > > > > > >> > > > > > > > > > >> > async call requirement is
>> due
>> > to
>> > > > its
>> > > > > IO
>> > > > > > > > heavy
>> > > > > > > > > > > >> > operator.
>> > > > > > > > > > > >> > > In
>> > > > > > > > > > > >> > > > > our
>> > > > > > > > > > > >> > > > > > > > > usage,
>> > > > > > > > > > > >> > > > > > > > > > >> sql
>> > > > > > > > > > > >> > > > > > > > > > >> > users have
>> > > > > > > > > > > >> > > > > > > > > > >> > logic to do some RPC call or
>> > > query
>> > > > > the
>> > > > > > > > > > > third-party
>> > > > > > > > > > > >> > > service
>> > > > > > > > > > > >> > > > > > which
>> > > > > > > > > > > >> > > > > > > > is
>> > > > > > > > > > > >> > > > > > > > > > >> also IO
>> > > > > > > > > > > >> > > > > > > > > > >> > intensive.
>> > > > > > > > > > > >> > > > > > > > > > >> > In these case, we'd like to
>> > > > leverage
>> > > > > > the
>> > > > > > > > > async
>> > > > > > > > > > > >> > function
>> > > > > > > > > > > >> > > to
>> > > > > > > > > > > >> > > > > > > improve
>> > > > > > > > > > > >> > > > > > > > > the
>> > > > > > > > > > > >> > > > > > > > > > >> > throughput.
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > Thanks,
>> > > > > > > > > > > >> > > > > > > > > > >> > Aitozi.
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > Jing Ge
>> > > <j...@ververica.com.invalid
>> > > > >
>> > > > > > > > > > > 于2023年6月1日周四
>> > > > > > > > > > > >> > > > 22:55写道：
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > Hi Aitozi,
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > Sorry for the late reply.
>> > Would
>> > > > you
>> > > > > > > like
>> > > > > > > > to
>> > > > > > > > > > > >> update
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> > > > > > > proposed
>> > > > > > > > > > > >> > > > > > > > > > >> changes
>> > > > > > > > > > > >> > > > > > > > > > >> > > with more details into the
>> > FLIP
>> > > > > too?
>> > > > > > > > > > > >> > > > > > > > > > >> > > I got your point. It looks
>> > > like a
>> > > > > > > > rational
>> > > > > > > > > > > idea.
>> > > > > > > > > > > >> > > > However,
>> > > > > > > > > > > >> > > > > > > since
>> > > > > > > > > > > >> > > > > > > > > > lookup
>> > > > > > > > > > > >> > > > > > > > > > >> > has
>> > > > > > > > > > > >> > > > > > > > > > >> > > its clear async call
>> > > requirement,
>> > > > > are
>> > > > > > > > there
>> > > > > > > > > > any
>> > > > > > > > > > > >> real
>> > > > > > > > > > > >> > > use
>> > > > > > > > > > > >> > > > > > cases
>> > > > > > > > > > > >> > > > > > > > > that
>> > > > > > > > > > > >> > > > > > > > > > >> > > need this change? This
>> will
>> > > help
>> > > > us
>> > > > > > > > > > understand
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> > > > > > motivation.
>> > > > > > > > > > > >> > > > > > > > > After
>> > > > > > > > > > > >> > > > > > > > > > >> all,
>> > > > > > > > > > > >> > > > > > > > > > >> > > lateral join and temporal
>> > > lookup
>> > > > > > > join[1]
>> > > > > > > > > are
>> > > > > > > > > > > >> quite
>> > > > > > > > > > > >> > > > > > different.
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > Best regards,
>> > > > > > > > > > > >> > > > > > > > > > >> > > Jing
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > [1]
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > On Wed, May 31, 2023 at
>> > 8:53 AM
>> > > > > > Aitozi
>> > > > > > > <
>> > > > > > > > > > > >> > > > > > gjying1...@gmail.com>
>> > > > > > > > > > > >> > > > > > > > > > wrote:
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > Hi Jing,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > What do you think
>> about
>> > > it?
>> > > > > Can
>> > > > > > > we
>> > > > > > > > > move
>> > > > > > > > > > > >> > forward
>> > > > > > > > > > > >> > > > this
>> > > > > > > > > > > >> > > > > > > > > feature?
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > Thanks,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > Aitozi.
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > Aitozi <
>> > gjying1...@gmail.com
>> > > >
>> > > > > > > > > > 于2023年5月29日周一
>> > > > > > > > > > > >> > > 09:56写道：
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > Hi Jing,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > > "Do you mean to
>> > > support
>> > > > > the
>> > > > > > > > > > > >> > > AyncTableFunction
>> > > > > > > > > > > >> > > > > > beyond
>> > > > > > > > > > > >> > > > > > > > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > LookupTableSource?"
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > Yes, I mean to support
>> > the
>> > > > > > > > > > > AyncTableFunction
>> > > > > > > > > > > >> > > beyond
>> > > > > > > > > > > >> > > > > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > LookupTableSource.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > The
>> "AsyncTableFunction"
>> > is
>> > > > the
>> > > > > > > > > function
>> > > > > > > > > > > with
>> > > > > > > > > > > >> > > > ability
>> > > > > > > > > > > >> > > > > to
>> > > > > > > > > > > >> > > > > > > be
>> > > > > > > > > > > >> > > > > > > > > > >> executed
>> > > > > > > > > > > >> > > > > > > > > > >> > > > async
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > (with
>> AsyncWaitOperator).
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > The async lookup join
>> is
>> > a
>> > > > one
>> > > > > of
>> > > > > > > > usage
>> > > > > > > > > > of
>> > > > > > > > > > > >> this.
>> > > > > > > > > > > >> > > So,
>> > > > > > > > > > > >> > > > > we
>> > > > > > > > > > > >> > > > > > > > don't
>> > > > > > > > > > > >> > > > > > > > > > >> have to
>> > > > > > > > > > > >> > > > > > > > > > >> > > > bind
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > the AyncTableFunction
>> > with
>> > > > > > > > > > > LookupTableSource.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > If User-defined
>> > > > > > AsyncTableFunction
>> > > > > > > is
>> > > > > > > > > > > >> supported,
>> > > > > > > > > > > >> > > > user
>> > > > > > > > > > > >> > > > > > can
>> > > > > > > > > > > >> > > > > > > > > > directly
>> > > > > > > > > > > >> > > > > > > > > > >> > use
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > lateral table syntax
>> to
>> > > > perform
>> > > > > > > async
>> > > > > > > > > > > >> operation.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > > "It would be better
>> if
>> > > you
>> > > > > > could
>> > > > > > > > > > > elaborate
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> > > > > > proposed
>> > > > > > > > > > > >> > > > > > > > > > changes
>> > > > > > > > > > > >> > > > > > > > > > >> wrt
>> > > > > > > > > > > >> > > > > > > > > > >> > > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> CorrelatedCodeGenerator
>> > > with
>> > > > > more
>> > > > > > > > > > details"
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > In the proposal, we
>> use
>> > > > lateral
>> > > > > > > table
>> > > > > > > > > > > syntax
>> > > > > > > > > > > >> to
>> > > > > > > > > > > >> > > > > support
>> > > > > > > > > > > >> > > > > > > the
>> > > > > > > > > > > >> > > > > > > > > > async
>> > > > > > > > > > > >> > > > > > > > > > >> > table
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > function. So the
>> planner
>> > > will
>> > > > > > also
>> > > > > > > > > treat
>> > > > > > > > > > > this
>> > > > > > > > > > > >> > > > > statement
>> > > > > > > > > > > >> > > > > > > to a
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > CommonExecCorrelate
>> node.
>> > > So
>> > > > > the
>> > > > > > > > > runtime
>> > > > > > > > > > > code
>> > > > > > > > > > > >> > > should
>> > > > > > > > > > > >> > > > > be
>> > > > > > > > > > > >> > > > > > > > > > generated
>> > > > > > > > > > > >> > > > > > > > > > >> in
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> CorrelatedCodeGenerator.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > In
>> > CorrelatedCodeGenerator,
>> > > > we
>> > > > > > will
>> > > > > > > > > know
>> > > > > > > > > > > the
>> > > > > > > > > > > >> > > > > > > TableFunction's
>> > > > > > > > > > > >> > > > > > > > > > Kind
>> > > > > > > > > > > >> > > > > > > > > > >> of
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > `FunctionKind.Table`
>> or
>> > > > > > > > > > > >> > `FunctionKind.ASYNC_TABLE`
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > For
>> > > > `FunctionKind.ASYNC_TABLE`
>> > > > > > we
>> > > > > > > > can
>> > > > > > > > > > > >> generate
>> > > > > > > > > > > >> > a
>> > > > > > > > > > > >> > > > > > > > > > >> AsyncWaitOperator
>> > > > > > > > > > > >> > > > > > > > > > >> > to
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > execute the async
>> table
>> > > > > function.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > Thanks,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > Aitozi.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > Jing Ge
>> > > > > > <j...@ververica.com.invalid
>> > > > > > > >
>> > > > > > > > > > > >> > > 于2023年5月29日周一
>> > > > > > > > > > > >> > > > > > > 03:22写道：
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Hi Aitozi,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Thanks for the
>> > > > clarification.
>> > > > > > The
>> > > > > > > > > naming
>> > > > > > > > > > > >> > "Lookup"
>> > > > > > > > > > > >> > > > > might
>> > > > > > > > > > > >> > > > > > > > > suggest
>> > > > > > > > > > > >> > > > > > > > > > >> > using
>> > > > > > > > > > > >> > > > > > > > > > >> > > it
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> for table look up.
>> But
>> > > > > > > conceptually
>> > > > > > > > > what
>> > > > > > > > > > > the
>> > > > > > > > > > > >> > > eval()
>> > > > > > > > > > > >> > > > > > > method
>> > > > > > > > > > > >> > > > > > > > > will
>> > > > > > > > > > > >> > > > > > > > > > >> do
>> > > > > > > > > > > >> > > > > > > > > > >> > is
>> > > > > > > > > > > >> > > > > > > > > > >> > > to
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> get a collection of
>> > > > > results(Row,
>> > > > > > > > > > RowData)
>> > > > > > > > > > > >> from
>> > > > > > > > > > > >> > > the
>> > > > > > > > > > > >> > > > > > given
>> > > > > > > > > > > >> > > > > > > > > keys.
>> > > > > > > > > > > >> > > > > > > > > > >> How
>> > > > > > > > > > > >> > > > > > > > > > >> > it
>> > > > > > > > > > > >> > > > > > > > > > >> > > > will
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> be done depends on
>> the
>> > > > > > > > implementation,
>> > > > > > > > > > > i.e.
>> > > > > > > > > > > >> you
>> > > > > > > > > > > >> > > can
>> > > > > > > > > > > >> > > > > > > > implement
>> > > > > > > > > > > >> > > > > > > > > > >> your
>> > > > > > > > > > > >> > > > > > > > > > >> > own
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Source[1][2]. The
>> > example
>> > > in
>> > > > > the
>> > > > > > > > FLIP
>> > > > > > > > > > > >> should be
>> > > > > > > > > > > >> > > > able
>> > > > > > > > > > > >> > > > > to
>> > > > > > > > > > > >> > > > > > > be
>> > > > > > > > > > > >> > > > > > > > > > >> handled
>> > > > > > > > > > > >> > > > > > > > > > >> > in
>> > > > > > > > > > > >> > > > > > > > > > >> > > > this
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> way.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Do you mean to
>> support
>> > the
>> > > > > > > > > > > AyncTableFunction
>> > > > > > > > > > > >> > > beyond
>> > > > > > > > > > > >> > > > > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > LookupTableSource?
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> It would be better if
>> > you
>> > > > > could
>> > > > > > > > > > elaborate
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> > > > > proposed
>> > > > > > > > > > > >> > > > > > > > > changes
>> > > > > > > > > > > >> > > > > > > > > > >> wrt
>> > > > > > > > > > > >> > > > > > > > > > >> > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> CorrelatedCodeGenerator
>> > > with
>> > > > > > more
>> > > > > > > > > > details.
>> > > > > > > > > > > >> > > Thanks!
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Best regards,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Jing
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> [1]
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> [2]
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> On Sat, May 27, 2023
>> at
>> > > > > 9:48 AM
>> > > > > > > > > Aitozi <
>> > > > > > > > > > > >> > > > > > > > gjying1...@gmail.com
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > wrote:
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Hi Jing,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Thanks for your
>> > > > > response.
>> > > > > > As
>> > > > > > > > > > stated
>> > > > > > > > > > > in
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> > > > > FLIP,
>> > > > > > > > > > > >> > > > > > > the
>> > > > > > > > > > > >> > > > > > > > > > >> purpose
>> > > > > > > > > > > >> > > > > > > > > > >> > of
>> > > > > > > > > > > >> > > > > > > > > > >> > > > this
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > FLIP is meant to
>> > support
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > user-defined async
>> > table
>> > > > > > > function.
>> > > > > > > > > As
>> > > > > > > > > > > >> > described
>> > > > > > > > > > > >> > > > in
>> > > > > > > > > > > >> > > > > > > flink
>> > > > > > > > > > > >> > > > > > > > > > >> document
>> > > > > > > > > > > >> > > > > > > > > > >> > > [1]
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Async table
>> functions
>> > > are
>> > > > > > > special
>> > > > > > > > > > > >> functions
>> > > > > > > > > > > >> > for
>> > > > > > > > > > > >> > > > > table
>> > > > > > > > > > > >> > > > > > > > > sources
>> > > > > > > > > > > >> > > > > > > > > > >> that
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> perform
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > a lookup.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > So end user can not
>> > > > directly
>> > > > > > > > define
>> > > > > > > > > > and
>> > > > > > > > > > > >> use
>> > > > > > > > > > > >> > > async
>> > > > > > > > > > > >> > > > > > table
>> > > > > > > > > > > >> > > > > > > > > > >> function
>> > > > > > > > > > > >> > > > > > > > > > >> > > now.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > An
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > user case is
>> reported
>> > in
>> > > > [2]
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > So, in conclusion,
>> no
>> > > new
>> > > > > > > > interface
>> > > > > > > > > is
>> > > > > > > > > > > >> > > > introduced,
>> > > > > > > > > > > >> > > > > > but
>> > > > > > > > > > > >> > > > > > > we
>> > > > > > > > > > > >> > > > > > > > > > >> extend
>> > > > > > > > > > > >> > > > > > > > > > >> > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > ability to support
>> > > > > > user-defined
>> > > > > > > > > async
>> > > > > > > > > > > >> table
>> > > > > > > > > > > >> > > > > function.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > [1]:
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > [2]:
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> >
>> > > > > > > >
>> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Thanks.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Aitozi.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Jing Ge
>> > > > > > > > <j...@ververica.com.invalid
>> > > > > > > > > >
>> > > > > > > > > > > >> > > > 于2023年5月27日周六
>> > > > > > > > > > > >> > > > > > > > > 06:40写道：
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Hi Aitozi,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Thanks for your
>> > > > proposal.
>> > > > > I
>> > > > > > am
>> > > > > > > > not
>> > > > > > > > > > > quite
>> > > > > > > > > > > >> > sure
>> > > > > > > > > > > >> > > > if
>> > > > > > > > > > > >> > > > > I
>> > > > > > > > > > > >> > > > > > > > > > understood
>> > > > > > > > > > > >> > > > > > > > > > >> > your
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > thoughts
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > correctly. You
>> > > > described a
>> > > > > > > > special
>> > > > > > > > > > > case
>> > > > > > > > > > > >> > > > > > > implementation
>> > > > > > > > > > > >> > > > > > > > of
>> > > > > > > > > > > >> > > > > > > > > > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> AsyncTableFunction
>> > > with
>> > > > on
>> > > > > > > > public
>> > > > > > > > > > API
>> > > > > > > > > > > >> > > changes.
>> > > > > > > > > > > >> > > > > > Would
>> > > > > > > > > > > >> > > > > > > > you
>> > > > > > > > > > > >> > > > > > > > > > >> please
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> elaborate
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > your purpose of
>> > > writing
>> > > > a
>> > > > > > FLIP
>> > > > > > > > > > > >> according to
>> > > > > > > > > > > >> > > the
>> > > > > > > > > > > >> > > > > > FLIP
>> > > > > > > > > > > >> > > > > > > > > > >> > > > documentation[1]?
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Thanks!
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > [1]
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Best regards,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Jing
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > On Wed, May 24,
>> 2023
>> > > at
>> > > > > > > 1:07 PM
>> > > > > > > > > > > Aitozi <
>> > > > > > > > > > > >> > > > > > > > > > gjying1...@gmail.com
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > wrote:
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > May I ask for
>> some
>> > > > > > feedback
>> > > > > > > > :D
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Thanks,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi <
>> > > > > > > gjying1...@gmail.com>
>> > > > > > > > > > > >> > > 于2023年5月23日周二
>> > > > > > > > > > > >> > > > > > > 19:14写道：
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > Just catch an
>> > user
>> > > > > case
>> > > > > > > > report
>> > > > > > > > > > > from
>> > > > > > > > > > > >> > > Giannis
>> > > > > > > > > > > >> > > > > > > Polyzos
>> > > > > > > > > > > >> > > > > > > > > for
>> > > > > > > > > > > >> > > > > > > > > > >> this
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> usage:
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > >
>> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > Aitozi <
>> > > > > > > > gjying1...@gmail.com>
>> > > > > > > > > > > >> > > > 于2023年5月23日周二
>> > > > > > > > > > > >> > > > > > > > 17:45写道：
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Hi guys,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > I want
>> to
>> > > > bring
>> > > > > > up a
>> > > > > > > > > > > >> discussion
>> > > > > > > > > > > >> > > about
>> > > > > > > > > > > >> > > > > > > adding
>> > > > > > > > > > > >> > > > > > > > > > >> support
>> > > > > > > > > > > >> > > > > > > > > > >> > of
>> > > > > > > > > > > >> > > > > > > > > > >> > > > User
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Defined
>> > > > > > > AsyncTableFunction
>> > > > > > > > > in
>> > > > > > > > > > > >> Flink.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Currently,
>> > async
>> > > > > table
>> > > > > > > > > > function
>> > > > > > > > > > > >> are
>> > > > > > > > > > > >> > > > special
>> > > > > > > > > > > >> > > > > > > > > functions
>> > > > > > > > > > > >> > > > > > > > > > >> for
>> > > > > > > > > > > >> > > > > > > > > > >> > > > table
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > source
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > to perform
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > async
>> lookup.
>> > > > > However,
>> > > > > > > > it's
>> > > > > > > > > > > worth
>> > > > > > > > > > > >> to
>> > > > > > > > > > > >> > > > > support
>> > > > > > > > > > > >> > > > > > > the
>> > > > > > > > > > > >> > > > > > > > > user
>> > > > > > > > > > > >> > > > > > > > > > >> > > defined
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> async
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > table
>> > function.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Because, in
>> > this
>> > > > > way,
>> > > > > > > the
>> > > > > > > > > end
>> > > > > > > > > > > SQL
>> > > > > > > > > > > >> > user
>> > > > > > > > > > > >> > > > can
>> > > > > > > > > > > >> > > > > > > > leverage
>> > > > > > > > > > > >> > > > > > > > > > it
>> > > > > > > > > > > >> > > > > > > > > > >> to
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> perform
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > async
>> > operation
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > which is
>> > useful
>> > > to
>> > > > > > > maximum
>> > > > > > > > > the
>> > > > > > > > > > > >> system
>> > > > > > > > > > > >> > > > > > > throughput
>> > > > > > > > > > > >> > > > > > > > > > >> > especially
>> > > > > > > > > > > >> > > > > > > > > > >> > > > for
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> IO
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > bottleneck
>> > case.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > You can
>> find
>> > > some
>> > > > > more
>> > > > > > > > > detail
>> > > > > > > > > > in
>> > > > > > > > > > > >> [1].
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Looking
>> > forward
>> > > to
>> > > > > > > > feedback
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > [1]:
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Thanks,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Aitozi.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
