Re: [DISUCSS] Deprecate multiple APIs in 1.18

[Jing Ge]

got it, thanks for the clarification.

Best regards,
Jing

On Fri, Jul 21, 2023 at 5:14 AM Jane Chan  wrote:

> Hi, Jing,
>
> I share the same opinion with Xintong. The base class `TimestampExtractor`
> for these two classes was deprecated in release-1.12[1][2], so deprecating
> these two subclasses is relatively straightforward.
>
> [1] https://issues.apache.org/jira/browse/FLINK-19453
> [2]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces
>
>
> Best,
> Jane
>
> On Fri, Jul 21, 2023 at 10:19 AM Xintong Song 
> wrote:
>
> > I personally don't think it's necessary. IIUC, we have already decided to
> > deprecate TableSource / TableSink, and not marking StreamRecordTimestamp
> > and ExistingField deprecated is just an oversight in carrying out that
> > decision.
> >
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Fri, Jul 21, 2023 at 7:48 AM Jing Ge 
> > wrote:
> >
> > > Hi Jane, Hi Xintong,
> > >
> > > The specific discussion of deprecating `StreamRecordTimestamp` &
> > > `ExistingField` was kind of hiding in the discussion of the umbrella
> > topic
> > > that people might miss out. Does it make sense to start an
> > > individual Discussion thread for it like we deprecate any other APIs?
> > >
> > > For all APIs that should be marked as @PublicEvolving, dedicated
> > graduation
> > > threads are recommended.
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > > On Wed, Jul 19, 2023 at 10:09 AM Xintong Song 
> > > wrote:
> > >
> > > > Thank you, Jane.
> > > >
> > > > What you said (preparing a FLIP for ManagedTable related classes, not
> > > > deprecating SourceFunctionProvider, and starting a dedicated
> discussion
> > > for
> > > > the missing annotations) sounds good to me.
> > > >
> > > > In addition, if there's no objections from the community on marking
> > > > `StreamRecordTimestamp` & `ExistingField` as deprecated, you may
> > consider
> > > > creating a Jira ticket for it as a sub-task of FLINK-32557.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Wed, Jul 19, 2023 at 3:48 PM Jane Chan 
> > wrote:
> > > >
> > > > > Hi Xintong,
> > > > >
> > > > > > IIUC, you are suggesting to mark the classes with a red
> background
> > as
> > > > > `@Deprecated` in 1.18?
> > > > >
> > > > > Exactly. They are not marked at the code level but suppose to be
> > > > > deprecated.
> > > > >
> > > > > > - For the ManagedTable related classes, is there any FLIP
> > explicitly
> > > > that
> > > > > decides to deprecate them? If not, I think it would be nice to have
> > > one,
> > > > to
> > > > > formally decide the deprecation with a vote. I'd expect such a FLIP
> > can
> > > > be
> > > > > quite lightweight.
> > > > >
> > > > > Currently, there's no formal FLIP yet. I'd like to prepare one to
> > > > initiate
> > > > > the deprecation process.
> > > > >
> > > > > > - For `SourceFunctionProvider`, please beware there are recently
> > some
> > > > > objections against deprecating `SourceFunction`, and the code
> changes
> > > > > marking it as `@Deprecated` might be reverted. See [1][2] for more
> > > > details.
> > > > > So my question is, if `SourceFunction` will not be deprecated until
> > all
> > > > > sub-tasks in FLINK-28045 are resolved, do you still think we should
> > > > > deprecate `SourceFunctionProvider` now?
> > > > >
> > > > > Thanks for the reminder. If `SourceFunction` is no longer being
> > > > deprecated,
> > > > > then `SourceFunctionProvider` also needs to be retained. I have
> > updated
> > > > the
> > > > > sheet and removed `SourceFunctionProvider`.
> > > > >
> > > > > > I'm in general +1 to add the missing API annotations. However, I
> > > don't
> > > > > have the expertise to comment on the classes and suggestd API
> levels
> > >

Re: [DISCUSS][2.0] FLIP-344: Remove parameter in RichFunction#open

[Jing Ge]

+1

On Fri, Jul 21, 2023 at 10:22 AM Yuxin Tan  wrote:

> +1
>
> Best,
> Yuxin
>
>
> Xintong Song  于2023年7月21日周五 12:04写道：
>
> > +1
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Fri, Jul 21, 2023 at 10:52 AM Wencong Liu 
> wrote:
> >
> > > Hi devs,
> > >
> > > I would like to start a discussion on FLIP-344: Remove parameter in
> > > RichFunction#open [1].
> > >
> > > The open() method in RichFunction requires a Configuration instance as
> an
> > > argument,
> > > which is always passed as a new instance without any configuration
> > > parameters in
> > > AbstractUdfStreamOperator#open. Thus, it is unnecessary to include this
> > > parameter
> > > in the open() method.
> > > As such I propose to remove the Configuration field from
> > > RichFunction#open(Configuration parameters).
> > > Looking forward to your feedback.
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425231
> > > Best regards,
> > >
> > >
> > > Wencong Liu
> >
>

Re: [DISCUSS][2.0] FLIP-347: Remove IOReadableWritable serialization in Path

[Jing Ge]

Hi Wencong,

Just out of curiosity, will the newly introduced
deserializeFromDataInputView() method make the Path mutable again?

What Matthias suggested makes sense, although the extension might make this
FLIP cross multiple modules.

Best regards,
Jing

On Fri, Jul 21, 2023 at 10:23 AM Matthias Pohl
 wrote:

> There's a kind-of-related issue FLINK-4758 [1] that proposes removing the
> IOReadableWritable interface from more classes. It was briefly mentioned in
> the must-have work items discussion [2].
>
> I'm not too sure about the usage of IOReadableWritable: ...whether it would
> go away with the removal of the DataSet API in general (the Jira issue has
> DataSet as a component), anyway.
>
> Otherwise, might it make sense to extend the scope of this FLIP?
>
> [1] https://issues.apache.org/jira/browse/FLINK-4758
> [2] https://lists.apache.org/thread/gf0h4gh3xfsj78cpdsxsnj70nhzcmv9r
>
> On Fri, Jul 21, 2023 at 6:04 AM Xintong Song 
> wrote:
>
> > +1
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Fri, Jul 21, 2023 at 10:54 AM Wencong Liu 
> wrote:
> >
> > > Hi devs,
> > >
> > > I would like to start a discussion on FLIP-347: Remove
> IOReadableWritable
> > > serialization in Path [1].
> > >
> > >
> > > The Path class is currently mutable to support IOReadableWritable
> > > serialization. However, many parts
> > > of the code assume that the Path is immutable. By making the Path class
> > > immutable, we can ensure
> > > that paths are stored correctly without the possibility of mutation and
> > > eliminate the occurrence of subtle errors.
> > > As such I propose to modify the Path class to no longer implement the
> > > IOReadableWritable interface.
> > > Looking forward to your feedback.
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-347%3A+Remove+IOReadableWritable+serialization+in+Path
> > > Best regards,
> > >
> > >
> > > Wencong Liu
> >
>

Re: [DISCUSS][2.0] FLIP-343: Remove parameter in WindowAssigner#getDefaultTrigger()

[Jing Ge]

+1

NIT: the release in the FLIP is still empty, it should be 2.0

Best regards,
Jing

On Fri, Jul 21, 2023 at 6:03 AM Xintong Song  wrote:

> +1
>
> Best,
>
> Xintong
>
>
>
> On Fri, Jul 21, 2023 at 10:53 AM Wencong Liu  wrote:
>
> > Hi devs,
> >
> > I would like to start a discussion on FLIP-343: Remove parameter in
> > WindowAssigner#getDefaultTrigger() [1].
> >
> >
> > The method getDefaultTrigger() in WindowAssigner takes a
> > StreamExecutionEnvironment
> > parameter, but this parameter is not actually used for any subclasses of
> > WindowAssigner.
> > Therefore, it is unnecessary to include this parameter.
> > As such I propose to remove the StreamExecutionEnvironment field from
> > WindowAssigner#getDefaultTrigger(StreamExecutionEnvironment env).
> > Looking forward to your feedback.
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425229
> > Best regards,
> >
> >
> > Wencong Liu
>

Re: [DISUCSS] Deprecate multiple APIs in 1.18

[Jing Ge]

eFunction`, and the code
> changes
> > >marking it as `@Deprecated` might be reverted. See [1][2] for more
> > > details.
> > >So my question is, if `SourceFunction` will not be deprecated until
> > all
> > >sub-tasks in FLINK-28045 are resolved, do you still think we should
> > >deprecate `SourceFunctionProvider` now?
> > >
> > > 2. In addition, during the process of organizing, it was found that
> some
> > > > APIs under the flink-table-api-java and flink-table-common modules do
> > not
> > > > have an explicit API annotation (you can find detailed information in
> > > this
> > > > sheet <
> > > >
> > >
> >
> https://docs.google.com/spreadsheets/d/1e8M0tUtKkZXEd8rCZtZ0C6Ty9QkNaPySsrCgz0vEID4/edit?usp=sharing
> > > >).
> > > > I suggest explicitly marking the level for these APIs.
> > >
> > > I'm in general +1 to add the missing API annotations. However, I don't
> > have
> > > the expertise to comment on the classes and suggestd API levels being
> > > listed.
> > >
> > > 3. As there are still some internal and test code dependencies on these
> > > > APIs,  can we first gradually migrate these dependencies to
> alternative
> > > > APIs to make the deprecation process relatively easy?
> > > >
> > > That makes sense to me. I think the purpose of trying to mark the APIs
> as
> > > deprecated in 1.18 is to send users the signal early that these APIs
> will
> > > be removed. As for the internal and test code dependencies, I don't see
> > any
> > > problem in gradually migrating them.
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > > [1] https://lists.apache.org/thread/734zhkvs59w2o4d1rsnozr1bfqlr6rgm
> > >
> > > [2] https://issues.apache.org/jira/browse/FLINK-28046
> > >
> > >
> > >
> > > On Wed, Jul 19, 2023 at 11:41 AM Jane Chan 
> > wrote:
> > >
> > > > Hi Xintong,
> > > >
> > > > Thanks for driving this topic. Regarding the Table API deprecation, I
> > can
> > > > provide some details to help with the process.
> > > >
> > > >1. This sheet
> > > ><
> > > >
> > >
> >
> https://docs.google.com/spreadsheets/d/1dZBNHLuAHYJt3pFU8ZtfUzrYyf2ZFQ6wybDXGS1bHno/edit?usp=sharing
> > > > >
> > > > summarizes
> > > >the user-facing classes and methods that need to be deprecated
> under
> > > the
> > > >flink-table module, some of which are marked with a red background
> > and
> > > >belong to the APIs that need to be depreciated but are not
> > explicitly
> > > >marked in the code. This mainly includes legacy table source/sink,
> > > > legacy
> > > >table schema, legacy SQL function, and some internal APIs designed
> > for
> > > >Paimon but are now obsolete.
> > > >2. In addition, during the process of organizing, it was found
> that
> > > some
> > > >APIs under the flink-table-api-java and flink-table-common modules
> > do
> > > > not
> > > >have an explicit API annotation (you can find detailed information
> > in
> > > > this
> > > >sheet
> > > ><
> > > >
> > >
> >
> https://docs.google.com/spreadsheets/d/1e8M0tUtKkZXEd8rCZtZ0C6Ty9QkNaPySsrCgz0vEID4/edit?usp=sharing
> > > > >).
> > > >I suggest explicitly marking the level for these APIs.
> > > >3. As there are still some internal and test code dependencies on
> > > these
> > > >APIs,  can we first gradually migrate these dependencies to
> > > alternative
> > > >APIs to make the deprecation process relatively easy?
> > > >
> > > > I hope this helps.
> > > >
> > > > Best,
> > > > Jane
> > > >
> > > > On Tue, Jul 11, 2023 at 12:08 PM Xintong Song  >
> > > > wrote:
> > > >
> > > > > Thanks for bringing this up, Matthias. I've replied to you in the
> > other
> > > > > thread[1].
> > > > >
> > > > > Best,
> > > > >
> > > > > Xintong
> > > > >
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/l3dkdy

Re: [DISCUSS] Connectors, Formats, and even User Code should also be pluggable.

[Jing Ge]

Hi Zhiqiang,

Thanks for your proposal. The idea is very interesting! Since almost all
connectors are or will be externalized, the pluggable design you suggested
could help reduce the development effort.

As you mentioned, the attached doc contains only your preliminary idea. I
would suggest that you could turn it into a FLIP with more details and do
the followings:

1. Describe a real conflict case that will benefit from the pluggable design
2. Describe where you want to modify the code, or provide a POC branch
3. Guideline to tell users how to use it, i.e. where (the plugins dir)
should the connector jar be located, how does it look like with an example,
etc.

WDYT?

Best regards,
Jing


On Fri, Jul 14, 2023 at 3:00 PM zhiqiang li 
wrote:

> Hi devs,
>
> I have observed that in [1], connectors and formats are pluggable,
> allowing user code to be easily integrated. The advantages of having
> pluggable connectors are evident, as it helps avoid conflicts between
> different versions of jar packages. If classloader isolation is not used,
> shading becomes necessary to resolve conflicts, resulting in a significant
> waste of development time. I understand that implementing this change may
> require numerous API modifications, so I would like to discuss in this
> email.
>
> > Plugins cannot access classes from other plugins or from Flink that have
> not been specifically whitelisted.
> > This strict isolation allows plugins to contain conflicting versions of
> the same library without the need to relocate classes or to converge to
> common versions.
> > Currently, file systems and metric reporters are pluggable but, in the
> future, connectors, formats, and even user code should also be pluggable.
>
> [2] It is my preliminary idea.
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/
> [2]
> https://docs.google.com/document/d/1XP2fBpcntK0YIdQ_Ax7JV2MhNdebvkFxSiNJRp6WQ24/edit?usp=sharing
>
>
> Best,
> Zhiqiang
>
>

Re: [DISCUSS] FLIP-346: Deprecate ManagedTable related APIs

[Jing Ge]

+1

On Thu, Jul 20, 2023 at 10:37 AM Lincoln Lee  wrote:

> +1
>
>
> Best,
> Lincoln Lee
>
>
> Leonard Xu  于2023年7月20日周四 16:26写道：
>
> > Thanks Jane for driving this FLIP.
> >
> >
> > +1,
> >
> > Minor: We can remove related document as well, Keeping a external link to
> > Apache Paimon should be enough.
> >
> > Best,
> > Leonard
> >
> >
> >
> > > On Jul 20, 2023, at 3:07 PM, Martijn Visser 
> > wrote:
> > >
> > > +1
> > >
> > > On Thu, Jul 20, 2023 at 7:53 AM Xintong Song 
> > wrote:
> > >
> > >> +1
> > >>
> > >> Best,
> > >>
> > >> Xintong
> > >>
> > >>
> > >>
> > >> On Thu, Jul 20, 2023 at 1:25 PM Jingsong Li 
> > >> wrote:
> > >>
> > >>> +1
> > >>>
> > >>> On Thu, Jul 20, 2023 at 12:31 PM Jane Chan 
> > >> wrote:
> > 
> >  Hi, devs,
> > 
> >  I would like to start a discussion on FLIP-346: Deprecate
> ManagedTable
> >  related APIs[1].
> > 
> >  These APIs were initially designed for Flink Table Store, which has
> >  joined the Apache Incubator as a separate project called Apache
> >  Paimon(incubating).
> > 
> >  Since they are obsolete and not used by Paimon anymore, I propose to
> >  deprecate them in v1.18 and further remove them before v2.0.
> > 
> >  Looking forward to your feedback.
> > 
> >  [1]
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-346%3A+Deprecate+ManagedTable+related+APIs
> > 
> >  Best regards,
> >  Jane
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS][2.0] FLIP-338: Remove terminationMode query parameter from job cancellation REST endpoint

[Jing Ge]

+1

On Mon, Jul 17, 2023 at 5:30 AM Xintong Song  wrote:

> +1
>
> Best,
>
> Xintong
>
>
>
> On Thu, Jul 13, 2023 at 9:41 PM Chesnay Schepler 
> wrote:
>
> > Hello,
> >
> > The job cancellation REST endpoint has a terminationMode query
> > parameter, which in the past could be set to either CANCEL or STOP, but
> > nowadays the job stop endpoint has subsumed the STOP functionality.
> >
> > Since then the cancel endpoint rejected requests that specified STOP.
> >
> > I propose to finally remove this parameter, as it currently serves no
> > function.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-338%3A+Remove+terminationMode+query+parameter+from+job+cancellation+REST+endpoint
> >
> >
> >
> > Regards,
> >
> > Chesnay
> >
>

Re: [DISCUSS][2.0] FLIP-336: Remove "now" timestamp field from REST responses

[Jing Ge]

+1

On Mon, Jul 17, 2023 at 5:29 AM Xintong Song  wrote:

> +1
>
> Best,
>
> Xintong
>
>
>
> On Thu, Jul 13, 2023 at 9:05 PM Chesnay Schepler 
> wrote:
>
> > Hello,
> >
> > Several REST responses contain a timestamp field of the current time
> >
> >   There is no known use-case for said timestamp, it makes caching of
> > responses technically sketchy (since the response differs on each call)
> > and  it complicates testing since the timestamp field can't be easily
> > predicted.
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424789
> >
> >
> > Regards,
> >
> > Chesnay
> >
> >
>

Re: [DISCUSS][2.0] FLIP-341: Remove MetricGroup methods accepting an int as a name

[Jing Ge]

+1

On Tue, Jul 18, 2023 at 1:11 PM Chesnay Schepler  wrote:

> Good catch; i've fixed the list.
>
> On 18/07/2023 12:20, Xintong Song wrote:
> > +1 in general.
> >
> > I think the list of affected public interfaces in the FLIP is not
> accurate.
> >
> > - `#counter(int, Counter)` is missed
> > - `#meter(int)` should be `#meter(int, Meter)`
> > - `#group(int)` should be `#addGroup(int)`
> >
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Tue, Jul 18, 2023 at 4:39 PM Chesnay Schepler 
> wrote:
> >
> >> The MetricGroup interface contains methods to create groups and metrics
> >> using an int as a name. The original intention was to allow pattern like
> >> |group.addGroup("subtaskIndex").addGroup(0)| , but this didn't really
> >> work out, with |addGroup(String, String)|  serving this use case much
> >> better.
> >>
> >> Metric methods accept an int mostly for consistency, but there's no good
> >> use-case for it.
> >>
> >> These methods also offer hardly any convenience since all they do is
> >> save potential users from using |String.valueOf| on one argument. That's
> >> doesn't seem valuable enough for something that doubles the size of the
> >> interface.
> >>
> >> I propose to remove said method.
> >>
> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-341%3A+Remove+MetricGroup+methods+accepting+an+int+as+a+name
> >>
>
>

Re: FLIP-342: Remove brackets around keys returned by MetricGroup#getAllVariables

[Jing Ge]

+1

On Tue, Jul 18, 2023 at 12:24 PM Xintong Song  wrote:

> +1
>
> Best,
>
> Xintong
>
>
>
> On Tue, Jul 18, 2023 at 5:02 PM Chesnay Schepler 
> wrote:
>
> > The FLIP number was changed to 342.
> >
> > On 18/07/2023 10:56, Chesnay Schepler wrote:
> > > MetricGroup#getAllVariables returns all variables associated with the
> > > metric, for example:
> > >
> > > | = abcde|
> > > | = ||0|
> > >
> > > The keys are surrounded by brackets for no particular reason.
> > >
> > > In virtually every use-case for this method the user is stripping the
> > > brackets from keys, as done in:
> > >
> > >  * our datadog reporter:
> > >
> >
> https://github.com/apache/flink/blob/9c3c8afbd9325b5df8291bd831da2d9f8785b30a/flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpReporter.java#L244
> > > <
> >
> https://github.com/apache/flink/blob/9c3c8afbd9325b5df8291bd831da2d9f8785b30a/flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpReporter.java#L244
> > >
> > >  * our prometheus reporter (implicitly via a character filter):
> > >
> >
> https://github.com/apache/flink/blob/9c3c8afbd9325b5df8291bd831da2d9f8785b30a/flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/AbstractPrometheusReporter.java#L236
> > > <
> >
> https://github.com/apache/flink/blob/9c3c8afbd9325b5df8291bd831da2d9f8785b30a/flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/AbstractPrometheusReporter.java#L236
> > >
> > >  * our JMX reporter:
> > >
> >
> https://github.com/apache/flink/blob/9c3c8afbd9325b5df8291bd831da2d9f8785b30a/flink-metrics/flink-metrics-jmx/src/main/java/org/apache/flink/metrics/jmx/JMXReporter.java#L223
> > > <
> >
> https://github.com/apache/flink/blob/9c3c8afbd9325b5df8291bd831da2d9f8785b30a/flink-metrics/flink-metrics-jmx/src/main/java/org/apache/flink/metrics/jmx/JMXReporter.java#L223
> > >
> > >
> > > I propose to change the method spec and implementation to remove the
> > > brackets around keys.
> > >
> > > For migration purposes it may make sense to add a new method with the
> > > new behavior (|getVariables()|) and deprecate the old method.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425202
> > >
> >
> >
>

Re: [DISCUSS][2.0] FLIP-340: Remove rescale REST endpoint

[Jing Ge]

+1

On Tue, Jul 18, 2023 at 1:05 PM Maximilian Michels  wrote:

> +1
>
> On Tue, Jul 18, 2023 at 12:29 PM Gyula Fóra  wrote:
> >
> > +1
> >
> > On Tue, 18 Jul 2023 at 12:12, Xintong Song 
> wrote:
> >
> > > +1
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Tue, Jul 18, 2023 at 4:25 PM Chesnay Schepler 
> > > wrote:
> > >
> > > > The endpoint hasn't been working for years and was only kept to
> inform
> > > > users about it. Let's finally remove it.
> > > >
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-340%3A+Remove+rescale+REST+endpoint
> > > >
> > > >
> > >
>

Re: [DISCUSS][2.0] FLIP-337: Remove JarRequestBody#programArgs

[Jing Ge]

got it thanks!

For @Deprecated, I meant to force using like: @Deprecated(since = "1.18",
forRemoval = true)

Best regards,
Jing

On Tue, Jul 18, 2023 at 11:06 AM Hong Teoh  wrote:

> +1 to this. Nice to simplify the REST API!
>
>
> Regards,
> Hong
>
>
> > On 18 Jul 2023, at 10:00, Chesnay Schepler  wrote:
> >
> > Something to note is that the UI is using this parameter, and would have
> to be changed to the new one.
> >
> > Since we want to avoid having to split arguments ourselves, this may
> imply changes to the UI.
> >
> > On 18/07/2023 10:18, Chesnay Schepler wrote:
> >> We'll log a warn message when it is used and maybe hide it from the
> docs.
> >>
> >> Archunit rule doesn't really work here because it's not annotated with
> stability annotations (as it shouldn't since the classes aren't really
> user-facing).
> >>
> >> On 17/07/2023 21:56, Jing Ge wrote:
> >>> Hi Chesnay,
> >>>
> >>> I am trying to understand what is the right removal process with this
> >>> concrete example. Given all things about the programArgs are private or
> >>> package private except the constructor. Will you just mark it as
> deprecated
> >>> with constructor overloading in 1.18 and remove it in 2.0? Should we
> >>> describe the deprecation work in the FLIP?
> >>>
> >>> Another more general question, maybe offtrack, I don't know which
> thread is
> >>> the right place to ask, since Java 11 has been recommended, should we
> >>> always include "since" and "forRemoval" while adding @Deprecated, i.e.
> >>> ArchUnit rule?
> >>>
> >>> Best regards,
> >>> Jing
> >>>
> >>> On Mon, Jul 17, 2023 at 5:33 AM Xintong Song 
> wrote:
> >>>
> >>>> +1
> >>>>
> >>>> Best,
> >>>>
> >>>> Xintong
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Jul 13, 2023 at 9:34 PM Chesnay Schepler 
> >>>> wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> The request body for the jar run/plan REST endpoints accepts program
> >>>>> arguments as a string (programArgs) or a list of strings
> >>>>> (programArgsList). The latter was introduced as kept running into
> issues
> >>>>> with splitting the string into individual arguments./
> >>>>> /
> >>>>>
> >>>>> We ideally force users to use the list argument, and we can simplify
> the
> >>>>> codebase if there'd only be 1 way to pass arguments.
> >>>>>
> >>>>> As such I propose to remove the programArgs field from the request
> body.
> >>>>>
> >>>>>
> >>>>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424796
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Chesnay
> >>>>>
> >>
> >
>
>

Re: [VOTE] FLIP-309: Support using larger checkpointing interval when source is processing backlog

[Jing Ge]

+1(binding)

Best regards,
Jing

On Tue, Jul 18, 2023 at 8:31 AM Rui Fan <1996fan...@gmail.com> wrote:

> +1(binding)
>
> Best,
> Rui Fan
>
>
> On Tue, Jul 18, 2023 at 12:04 PM Dong Lin  wrote:
>
> > Hi all,
> >
> > We would like to start the vote for FLIP-309: Support using larger
> > checkpointing interval when source is processing backlog [1]. This FLIP
> was
> > discussed in this thread [2].
> >
> > The vote will be open until at least July 21st (at least 72 hours),
> > following
> > the consensus voting process.
> >
> > Cheers,
> > Yunfeng and Dong
> >
> > [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-309
> >
> >
> %3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog
> > [2] https://lists.apache.org/thread/l1l7f30h7zldjp6ow97y70dcthx7tl37
> >
>

Re: [DISCUSS][2.0] FLIP-337: Remove JarRequestBody#programArgs

[Jing Ge]

Hi Chesnay,

I am trying to understand what is the right removal process with this
concrete example. Given all things about the programArgs are private or
package private except the constructor. Will you just mark it as deprecated
with constructor overloading in 1.18 and remove it in 2.0?  Should we
describe the deprecation work in the FLIP?

Another more general question, maybe offtrack, I don't know which thread is
the right place to ask, since Java 11 has been recommended, should we
always include "since" and "forRemoval" while adding @Deprecated, i.e.
ArchUnit rule?

Best regards,
Jing

On Mon, Jul 17, 2023 at 5:33 AM Xintong Song  wrote:

> +1
>
> Best,
>
> Xintong
>
>
>
> On Thu, Jul 13, 2023 at 9:34 PM Chesnay Schepler 
> wrote:
>
> > Hello,
> >
> > The request body for the jar run/plan REST endpoints accepts program
> > arguments as a string (programArgs) or a list of strings
> > (programArgsList). The latter was introduced as kept running into issues
> > with splitting the string into individual arguments./
> > /
> >
> > We ideally force users to use the list argument, and we can simplify the
> > codebase if there'd only be 1 way to pass arguments.
> >
> > As such I propose to remove the programArgs field from the request body.
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424796
> >
> >
> > Regards,
> >
> > Chesnay
> >
>

Re: [DISCUSS] FLIP-330: Support specifying record timestamp requirement

[Jing Ge]

Hi Yunfeng,

Thanks for your clarification. It might make sense to add one more
performance test with object-reuse disabled alongside to let us know how
big the improvement will be.

Best regards,
Jing

On Thu, Jul 13, 2023 at 11:51 AM Matt Wang  wrote:

> Hi Yunfeng,
>
> Thanks for the proposal. The POC showed a performance improvement of 20%,
> which is very exciting. But I have some questions:
> 1. Is the performance improvement here mainly due to the reduction of
> serialization, or is it due to the judgment consumption caused by tags?
> 2. Watermark is not needed in some scenarios, but the latency maker is a
> useful function. If the latency maker cannot be used, it will greatly limit
> the usage scenarios. Whether the solution design can retain the capability
> of the latency marker;
> 3. The data of the POC test is of long type. Here I want to see how much
> profit it will have if it is a string with a length of 100B or 1KB.
>
>
> --
>
> Best,
> Matt Wang
>
>
>  Replied Message 
> | From | Yunfeng Zhou |
> | Date | 07/13/2023 14:52 |
> | To |  |
> | Subject | Re: [DISCUSS] FLIP-330: Support specifying record timestamp
> requirement |
> Hi Jing,
>
> Thanks for reviewing this FLIP.
>
> 1. I did change the names of some APIs in the FLIP compared with the
> original version according to which I implemented the POC. As the core
> optimization logic remains the same and the POC's performance can
> still reflect the current FLIP's expected improvement, I have not
> updated the POC code after that. I'll add a note on the benchmark
> section of the FLIP saying that the namings in the POC code might be
> outdated, and FLIP is still the source of truth for our proposed
> design.
>
> 2. This FLIP could bring a fixed reduction on the workload of the
> per-record serialization path in Flink, so if the absolute time cost
> by non-optimized components could be lower, the performance
> improvement of this FLIP would be more obvious. That's why I chose to
> enable object-reuse and to transmit Boolean values in serialization.
> If it would be more widely regarded as acceptable for a benchmark to
> adopt more commonly-applied behavior(for object reuse, I believe
> disable is more common), I would be glad to update the benchmark
> result to disable object reuse.
>
> Best regards,
> Yunfeng
>
>
> On Thu, Jul 13, 2023 at 6:37 AM Jing Ge 
> wrote:
>
> Hi Yunfeng,
>
> Thanks for the proposal. It makes sense to offer the optimization. I got
> some NIT questions.
>
> 1. I guess you changed your thoughts while coding the POC, I found
> pipeline.enable-operator-timestamp in the code but  is
> pipeline.force-timestamp-support defined in the FLIP
> 2. about the benchmark example, why did you enable object reuse? Since It
> is an optimization of serde, will the benchmark be better if it is
> disabled?
>
> Best regards,
> Jing
>
> On Mon, Jul 10, 2023 at 11:54 AM Yunfeng Zhou  >
> wrote:
>
> Hi all,
>
> Dong(cc'ed) and I are opening this thread to discuss our proposal to
> support optimizing StreamRecord's serialization performance.
>
> Currently, a StreamRecord would be converted into a 1-byte tag (+
> 8-byte timestamp) + N-byte serialized value during the serialization
> process. In scenarios where timestamps and watermarks are not needed,
> and latency tracking is enabled, this process would include
> unnecessary information in the serialized byte array. This FLIP aims
> to avoid such overhead and increases Flink job's performance during
> serialization.
>
> Please refer to the FLIP document for more details about the proposed
> design and implementation. We welcome any feedback and opinions on
> this proposal.
>
> Best regards, Dong and Yunfeng
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-330%3A+Support+specifying+record+timestamp+requirement
>
>

Re: [VOTE] Graduate the FileSink to @PublicEvolving

[Jing Ge]

Hi,

After talking with Dong offline, it turns out @PublicEvolving are intended
for public use and should have stable behavior. FileSink is not ready for
that according to the data loss issue raised in the discussion [1].

This vote is cancelled.

Best regards,
Jing

[1] https://lists.apache.org/thread/wxoo7py5pqqlz37l4w8jrq6qdvsdq5wc


On Tue, Jul 11, 2023 at 9:56 PM Jing Ge  wrote:

> Hi,
>
> Sorry for the typo. The title is correct. The VOTE is for graduating the
> "FileSink" to @PublicEvolving.
>
> Best regards,
> Jing
>
>
> On Mon, Jul 10, 2023 at 1:10 PM Jing Ge  wrote:
>
>> Hi,
>>
>> I'd like to start the VOTE for graduating the FlinkSink
>> to @PublicEvolving. The discussion thread can be found at [1]
>>
>> The vote will be open until at least July 13 12pm GMT(72 hours) unless
>> there is an objection or insufficient votes.
>>
>> Thanks,
>>
>> Jing Ge
>>
>> [1] https://lists.apache.org/thread/wxoo7py5pqqlz37l4w8jrq6qdvsdq5wc
>>
>

Re: [DISCUSS] Deprecating Public API for 2.0 requiring FLIPs

[Jing Ge]

Hi,

Thanks Matthias for starting this discussion and thanks Chesnay for the
clarification.

I don't want to hijack this discussion but I would suggest postponing the
1.18 feature freeze over postponing the deprecations to 1.19. If there are
some contributors who think it makes sense, I will raise it in the 1.18
release channel to postpone 1.18 feature freeze again.

Speaking of the FLIP rules, if we follow it exactly like it is defined, we
should also write FLIPs when graduating @PublicEvloving APIs to be @Public,
especially for those APIs who will replace some deprecated APIs.  Doing
that is to guarantee that new public APIs will cover all
functionalities(including the capability that APIs are easy enough to
implement) that deprecated APIs offer, so that migrations can be executed
smoothly. With this in mind, we will avoid the big issue we are facing now
wrt the new Source and Sink API [1].

Best regards,
Jing

[1] https://lists.apache.org/thread/734zhkvs59w2o4d1rsnozr1bfqlr6rgm

On Thu, Jul 13, 2023 at 3:01 PM Chesnay Schepler  wrote:

> The issue isn't avoiding 1.19.
>
> The issue is that if things aren't deprecated in 1.18 then for every
> breaking change we have to start discussing exemptions from the API
> deprecation process, which stipulates that all Public APIs must be
> deprecated for at least 2 minor releases before they can be removed
> (which is now unsurprisingly backfiring on us).
>
> So if something isn't deprecated in 1.18 then either:
> - we delay 2.0 by at 1 release cycle
> - we effectively ignore the newly agreed upon deprecation process for 2.0
> - we circumvent the newly agreed upon deprecation process by creating 2
> minor releases in the same time-frame that we'd usually create 1 release
> in.
>
> None of these options are great.
>
> On 13/07/2023 14:03, Matthias Pohl wrote:
> > Jing brought up a question in the FLIP-335 discussion thread [1] which I
> > want to move into a dedicated discussion thread as it's a bit more
> general:
> > How do we handle the deprecation process of Public APIs for Flink 2.0?
> >> I just have a related question: Do we need to create a FLIP each time
> >> when we want to deprecate any classes?
> >
> > The community defined the requirements of a FLIP [2] in the following
> way:
> > - Any major new feature, subsystem, or piece of functionality
> > - Any change that impacts the public interfaces of the project
> >
> > public interfaces in this sense are defined as:
> > - DataStream and DataSet API, including classes related to that, such as
> > StreamExecutionEnvironment
> > - Classes marked with the @Public annotation
> > - On-disk binary formats, such as checkpoints/savepoints
> > - User-facing scripts/command-line tools, i.e. bin/flink, Yarn scripts,
> > Mesos scripts
> > - Configuration settings
> > - Exposed monitoring information
> >
> > I think that this makes sense. There should be a proper discussion on the
> > best approach to change public APIs. Additionally, the FLIP should be a
> way
> > to document the changes in the discussion process towards the change
> > transparently.
> >
> > In contrast, the impression I have is that we're trying to push all the
> > deprecation work into 1.18 (which has its feature freeze date by the end
> of
> > next week) to avoid having an additional 1.19 minor release. (Correct me
> if
> > I'm wrong here but that's the impression I'm getting from the ML threads
> > I've seen so far.)
> >
> > I have some concerns on the Flink 2.0 development in this regard. Zhu Zhu
> > [4] and Chesnay [5] shared similar concerns in the thread about the 2.0
> > must-have work items.
> >
> > Considering that quite a few (or some; I haven't checked in detail to be
> > honest) of the changes for 2.0 should require a FLIP and that there are
> > still some hanging items [6] (Jira issues which are annotated for Flink
> 2.0
> > but have been properly checked, yet): Shouldn't we avoid pushing
> everything
> > into 1.18? Instead, we should follow the required process properly and
> > might plan for another 1.19 minor release, instead?
> >
> > I'm curious how other contributors feel here and sorry in case I have
> > misinterpreted the ongoing discussions.
> >
> > Best,
> > Matthias
> >
> > [1] https://lists.apache.org/thread/48ysrg1rrtl8s1twg9wmx35l201hnc2w
> > [2]
> >
> https://cwiki.apache.org/confluence/display/Flink/Flink+Improvement+Proposals#FlinkImprovementProposals-Whatisconsidereda%22majorchange%22thatneedsaFLIP
> > ?
> > [3] https://lists.apache.org/thread/r0y9syc6k5nmcxvnd0hj33htdpdj9k6m
> > [4] https://lists.apache.org/thread/45xm348jr8n6s89jldntv5z3t13hdbn8
> > [5] https://lists.apache.org/thread/98wgqrx0sycpskvgpydvkywsoxt0fkc6
> > [6] https://lists.apache.org/thread/77hj39ls3kxvx2qd6o09hq1ndtn6hg4y
> >
>
>

Re: [DISCUSS] FLIP-335: Removing Flink's Time classes as part of Flink 2.0

[Jing Ge]

yeah, saw that thread. Thanks for starting the discussion!

Best regards,
Jing

On Thu, Jul 13, 2023 at 2:15 PM Matthias Pohl
 wrote:

> Thanks Jing. About your question regarding FLIPs: The FLIP is necessary
> when these classes touch user-facing API (which should be the case if we're
> preparing it for 2.0). I created a dedicated discussion thread [1] to
> discuss this topic because it's more general and independent of FLIP-335.
>
> Best,
> Matthias
>
> [1] https://lists.apache.org/thread/1007v4f4ms7ftp1qtkjsq25s5lwmk9wo
>
> On Thu, Jul 13, 2023 at 12:21 PM Jing Ge 
> wrote:
>
> > Hi Matthias,
> >
> > Thanks for raising this up! I saw your conversation with Chesnay in the
> PR
> > and after reading the historical discussions, there should be no doubt to
> > deprecate Time classes. +1
> >
> > I just have a related question: Do we need to create a FLIP each time
> when
> > we want to deprecate any classes? (BTW, FLIP-335 contains useful
> > information. It makes sense to write it)
> >
> > Best regards,
> > Jing
> >
> > On Thu, Jul 13, 2023 at 12:09 PM Matthias Pohl
> >  wrote:
> >
> > > The 2.0 feature list includes the removal of Flink's Time classes in
> > favor
> > > of the JDKs java.time.Duration class. There was already a discussion
> > about
> > > it in [1] and FLINK-14068 [2] was created as a consequence of this
> > > discussion.
> > >
> > > I started working on marking the APIs as deprecated in FLINK-32570 [3]
> > > where Chesnay raised a fair point that there isn't a FLIP, yet, to
> > > formalize this public API change. Therefore, I went ahead and created
> > > FLIP-335 [4] to have this change properly documented.
> > >
> > > I'm not 100% sure whether there are better ways of checking whether
> we're
> > > covering everything Public API-related. There are even classes which I
> > > think might be user-facing but are not labeled accordingly (e.g.
> > > flink-cep). But I don't have the proper knowledge in these parts of the
> > > code. Therefore, I would propose marking these methods as deprecated,
> > > anyway, to be on the safe side.
> > >
> > > I'm open to any suggestions on improving the Test Plan of this change.
> > >
> > > I'm looking forward to feedback on this FLIP.
> > >
> > > Best,
> > > Matthias
> > >
> > > [1] https://lists.apache.org/thread/76yywnwf3lk8qn4dby0vz7yoqx7f7pkj
> > > [2] https://issues.apache.org/jira/browse/FLINK-14068
> > > [3] https://issues.apache.org/jira/browse/FLINK-32570
> > > [4]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-335%3A+Removing+Flink%27s+Time+classes
> > >
> >
>

Re: [DISCUSS] FLIP-335: Removing Flink's Time classes as part of Flink 2.0

[Jing Ge]

Hi Matthias,

Thanks for raising this up! I saw your conversation with Chesnay in the PR
and after reading the historical discussions, there should be no doubt to
deprecate Time classes. +1

I just have a related question: Do we need to create a FLIP each time when
we want to deprecate any classes? (BTW, FLIP-335 contains useful
information. It makes sense to write it)

Best regards,
Jing

On Thu, Jul 13, 2023 at 12:09 PM Matthias Pohl
 wrote:

> The 2.0 feature list includes the removal of Flink's Time classes in favor
> of the JDKs java.time.Duration class. There was already a discussion about
> it in [1] and FLINK-14068 [2] was created as a consequence of this
> discussion.
>
> I started working on marking the APIs as deprecated in FLINK-32570 [3]
> where Chesnay raised a fair point that there isn't a FLIP, yet, to
> formalize this public API change. Therefore, I went ahead and created
> FLIP-335 [4] to have this change properly documented.
>
> I'm not 100% sure whether there are better ways of checking whether we're
> covering everything Public API-related. There are even classes which I
> think might be user-facing but are not labeled accordingly (e.g.
> flink-cep). But I don't have the proper knowledge in these parts of the
> code. Therefore, I would propose marking these methods as deprecated,
> anyway, to be on the safe side.
>
> I'm open to any suggestions on improving the Test Plan of this change.
>
> I'm looking forward to feedback on this FLIP.
>
> Best,
> Matthias
>
> [1] https://lists.apache.org/thread/76yywnwf3lk8qn4dby0vz7yoqx7f7pkj
> [2] https://issues.apache.org/jira/browse/FLINK-14068
> [3] https://issues.apache.org/jira/browse/FLINK-32570
> [4]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-335%3A+Removing+Flink%27s+Time+classes
>

Re: [DISCUSS] FLIP-330: Support specifying record timestamp requirement

[Jing Ge]

Hi Yunfeng,

Thanks for the proposal. It makes sense to offer the optimization. I got
some NIT questions.

1. I guess you changed your thoughts while coding the POC, I found
pipeline.enable-operator-timestamp in the code but  is
pipeline.force-timestamp-support defined in the FLIP
2. about the benchmark example, why did you enable object reuse? Since It
is an optimization of serde, will the benchmark be better if it is
disabled?

Best regards,
Jing

On Mon, Jul 10, 2023 at 11:54 AM Yunfeng Zhou 
wrote:

> Hi all,
>
> Dong(cc'ed) and I are opening this thread to discuss our proposal to
> support optimizing StreamRecord's serialization performance.
>
> Currently, a StreamRecord would be converted into a 1-byte tag (+
> 8-byte timestamp) + N-byte serialized value during the serialization
> process. In scenarios where timestamps and watermarks are not needed,
> and latency tracking is enabled, this process would include
> unnecessary information in the serialized byte array. This FLIP aims
> to avoid such overhead and increases Flink job's performance during
> serialization.
>
> Please refer to the FLIP document for more details about the proposed
> design and implementation. We welcome any feedback and opinions on
> this proposal.
>
> Best regards, Dong and Yunfeng
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-330%3A+Support+specifying+record+timestamp+requirement
>

Re: [DISCUSS] FLIP-327: Support stream-batch unified operator to improve job throughput when processing backlog data

[Jing Ge]

Hi Dong,

Thanks for your reply!

Best regards,
Jing

On Wed, Jul 12, 2023 at 3:25 AM Dong Lin  wrote:

> Hi Jing,
>
> Thanks for the comments. Please see my reply inline.
>
> On Wed, Jul 12, 2023 at 5:04 AM Jing Ge 
> wrote:
>
> > Hi Dong,
> >
> > Thanks for the clarification. Now it is clear for me. I got additional
> noob
> > questions wrt the internal sorter.
> >
> > 1. when to call setter to set the internalSorterSupported to be true?
> >
>
> Developer of the operator class (i.e. those classes which implements
> `StreamOperator`) should override the `#getOperatorAttributes()` API to set
> internalSorterSupported to true, if he/she decides to sort records
> internally in the operator.
>
>
> > 2
> > *"For those operators whose throughput can be considerably improved with
> an
> > internal sorter, update it to take advantage of the internal sorter when
> > its input has isBacklog=true.*
> > *Typically, operators that involve aggregation operation (e.g. join,
> > cogroup, aggregate) on keyed inputs can benefit from using an internal
> > sorter."*
> >
> > *"The operator that performs CoGroup operation will instantiate two
> > internal sorter to sorts records from its two inputs separately. Then it
> > can pull the sorted records from these two sorters. This can be done
> > without wrapping input records with TaggedUnion<...>. In comparison, the
> > existing DataStream#coGroup needs to wrap input records with
> > TaggedUnion<...> before sorting them using one external sorter, which
> > introduces higher overhead."*
> >
> > According to the performance test, it seems that internal sorter has
> better
> > performance than external sorter. Is it possible to make those operators
> > that can benefit from it use internal sorter by default?
> >
>
> Yes, it is possible. After this FLIP is done, users can use
> DataStream#coGroup with EndOfStreamWindows as the window assigner to
> co-group two streams in effectively the batch manner. An operator that uses
> an internal sorter will be used to perform the co-group operation. There is
> no need for users of the DataStream API to explicitly know or set the
> internal sorter in anyway.
>
> In the future, we plan to incrementally optimize other aggregation
> operation (e.g. aggregate) on the DataStream API when EndOfStreamWindows is
> used as the window assigner.
>
> Best,
> Dong
>
>
> >
> > Best regards,
> > Jing
> >
> >
> > On Tue, Jul 11, 2023 at 2:58 PM Dong Lin  wrote:
> >
> > > Hi Jing,
> > >
> > > Thank you for the comments! Please see my reply inline.
> > >
> > > On Tue, Jul 11, 2023 at 5:41 AM Jing Ge 
> > > wrote:
> > >
> > > > Hi Dong,
> > > >
> > > > Thanks for the proposal! The FLIP is already in good shape. I got
> some
> > > NIT
> > > > questions.
> > > >
> > > > 1. It is a little bit weird to write the hint right after the
> > motivation
> > > > that some features have been moved to FLIP-331, because at that time,
> > > > readers don't know the context about what features does it mean. I
> > would
> > > > suggest moving the note to the beginning of "Public interfaces"
> > sections.
> > > >
> > >
> > > Given that the reviewer who commented on this email thread before I
> > > refactored the FLIP (i.e. Piotr) has read FLP-331, I think it is
> simpler
> > to
> > > just remove any mention of FLIP-331. I have updated the FLIP
> accordingly.
> > >
> > >
> > > > 2. It is also a little bit weird to describe all behaviour changes at
> > > first
> > > > but only focus on one single feature, i.e. how to implement
> > > > internalSorterSupported. TBH, I was lost while I was reading the
> Public
> > > > interfaces. Maybe change the FLIP title? Another option could be to
> > > write a
> > > > short summary of all features and point out that this FLIP will only
> > > focus
> > > > on the internalSorterSupported feature. Others could be found in
> > > FLIP-331.
> > > > WDYT?
> > > >
> > >
> > > Conceptually, the purpose of this FLIP is to allow a stream mode job to
> > run
> > > parts of the topology in batch mode so that it can apply
> > > optimizations/computations that can not be used together with
> > checkpointing
> > > (and thus not usable in stream mod

Re: [DISCUSS] FLIP-327: Support stream-batch unified operator to improve job throughput when processing backlog data

[Jing Ge]

Hi Dong,

Thanks for the clarification. Now it is clear for me. I got additional noob
questions wrt the internal sorter.

1. when to call setter to set the internalSorterSupported to be true?

2
*"For those operators whose throughput can be considerably improved with an
internal sorter, update it to take advantage of the internal sorter when
its input has isBacklog=true.*
*Typically, operators that involve aggregation operation (e.g. join,
cogroup, aggregate) on keyed inputs can benefit from using an internal
sorter."*

*"The operator that performs CoGroup operation will instantiate two
internal sorter to sorts records from its two inputs separately. Then it
can pull the sorted records from these two sorters. This can be done
without wrapping input records with TaggedUnion<...>. In comparison, the
existing DataStream#coGroup needs to wrap input records with
TaggedUnion<...> before sorting them using one external sorter, which
introduces higher overhead."*

According to the performance test, it seems that internal sorter has better
performance than external sorter. Is it possible to make those operators
that can benefit from it use internal sorter by default?

Best regards,
Jing


On Tue, Jul 11, 2023 at 2:58 PM Dong Lin  wrote:

> Hi Jing,
>
> Thank you for the comments! Please see my reply inline.
>
> On Tue, Jul 11, 2023 at 5:41 AM Jing Ge 
> wrote:
>
> > Hi Dong,
> >
> > Thanks for the proposal! The FLIP is already in good shape. I got some
> NIT
> > questions.
> >
> > 1. It is a little bit weird to write the hint right after the motivation
> > that some features have been moved to FLIP-331, because at that time,
> > readers don't know the context about what features does it mean. I would
> > suggest moving the note to the beginning of "Public interfaces" sections.
> >
>
> Given that the reviewer who commented on this email thread before I
> refactored the FLIP (i.e. Piotr) has read FLP-331, I think it is simpler to
> just remove any mention of FLIP-331. I have updated the FLIP accordingly.
>
>
> > 2. It is also a little bit weird to describe all behaviour changes at
> first
> > but only focus on one single feature, i.e. how to implement
> > internalSorterSupported. TBH, I was lost while I was reading the Public
> > interfaces. Maybe change the FLIP title? Another option could be to
> write a
> > short summary of all features and point out that this FLIP will only
> focus
> > on the internalSorterSupported feature. Others could be found in
> FLIP-331.
> > WDYT?
> >
>
> Conceptually, the purpose of this FLIP is to allow a stream mode job to run
> parts of the topology in batch mode so that it can apply
> optimizations/computations that can not be used together with checkpointing
> (and thus not usable in stream mode). Although internal sorter is the only
> optimization immediately supported in this FLIP, this FLIP lays the
> foundation to support other optimizations in the future, such as using GPU
> to process a bounded stream of records.
>
> Therefore, I find it better to keep the current title rather than limiting
> the scope to internal sorter. What do you think?
>
>
>
> > 3. There should be a typo at 4) Checkpoint and failover strategy -> Mixed
> > mode ->
> >
> >- If any task fails when isBacklog=false true, this task is restarted
> to
> >re-process its input from the beginning.
> >
> >
> Thank you for catching this issue. It is fixed now.
>
> Best,
> Dong
>
>
> >
> >
> > Best regards
> > Jing
> >
> >
> > On Thu, Jul 6, 2023 at 1:24 PM Dong Lin  wrote:
> >
> > > Hi Piotr,
> > >
> > > Thanks for your comments! Please see my reply inline.
> > >
> > > On Wed, Jul 5, 2023 at 11:44 PM Piotr Nowojski <
> piotr.nowoj...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Dong,
> > > >
> > > > I have a couple of questions.
> > > >
> > > > Could you explain why those properties
> > > >
> > > > @Nullable private Boolean isOutputOnEOF = null;
> > > > @Nullable private Boolean isOutputOnCheckpoint = null;
> > > > @Nullable private Boolean isInternalSorterSupported = null;
> > > >
> > > > must be `@Nullable`, instead of having the default value set to
> > `false`?
> > > >
> > >
> > > By initializing these private variables in OperatorAttributesBuilder as
> > > null, we can implement `OperatorAttributesBuilder#build()` in such a
> way
> > > that it can print DEBUG lev

Re: [VOTE] Graduate the FileSink to @PublicEvolving

[Jing Ge]

Hi,

Sorry for the typo. The title is correct. The VOTE is for graduating the
"FileSink" to @PublicEvolving.

Best regards,
Jing


On Mon, Jul 10, 2023 at 1:10 PM Jing Ge  wrote:

> Hi,
>
> I'd like to start the VOTE for graduating the FlinkSink
> to @PublicEvolving. The discussion thread can be found at [1]
>
> The vote will be open until at least July 13 12pm GMT(72 hours) unless
> there is an objection or insufficient votes.
>
> Thanks,
>
> Jing Ge
>
> [1] https://lists.apache.org/thread/wxoo7py5pqqlz37l4w8jrq6qdvsdq5wc
>

Re: [VOTE] Release 2.0 must-have work items

[Jing Ge]

agree with what Leonard said. There are actually more issues wrt the new
Source and SinkV2[1]

Speaking of must-have vs nice-to-have, I think it depends on the priority.
If removing them has higher priority, we should keep related tasks as
must-have and make sure enough effort will be put to solve those issues and
therefore be able to remove those APIs.

Best regards,
Jing

[1] https://lists.apache.org/thread/90qc9nrlzf0vbvg92klzp9ftxxc43nbk

On Tue, Jul 11, 2023 at 10:26 AM Leonard Xu  wrote:

> Thanks Xintong for driving this great work! But I’ve to give my
> -1(binding) here：
>
> -1 to mark "deprecat SourceFunction/SinkFunction/Sinkv1" item as must to
> have for release 2.0.
>
> I do a lot of connector work in the community, and I have two insights
> from past experience:
>
> 1. Many developers reported that it is very difficult to migrate from
> SourceFunction to new Source [1]. The migration of existing conenctors
> after deprecated SourceFunction is very difficult. Some developers (Flavio
> Pompermaier) reported that they gave up the migration because it was too
> complicated. I believe it's not a few cases. This means that deprecating
> SourceFunction related interfaces require community contributors to reduce
> the migration cost before starting the migration work.
>
> 2. IIRC, the function of SinkV2 cannot currently cover SinkFunction as
> described in FLIP-287[2], it means the migration path after deprecate
> SinkFunction/Sinkv1 does not exist, thus we cannot mark the related
> interfaces of sinkfunction/sinkv1  as deprecated in 1.18.
>
> Based on these two cognitions, I think we should not mark these interfaces
> as must to have in 2.0. Maintaining the two sets of source/sink interfaces
> is not a concern for me, users can choose the interface to implement
> according to their energy and needs.
>
> Btw, some work items in 2.0 are marked as must to have, but no contributor
> has claimed them yet. I think this is a risk and hope the Release Managers
> could pay attention to it.
>
> Thank you all RMs for your work, sorry again for interrupting the vote
>
> Best,
> Leonard
>
> [1] https://lists.apache.org/thread/sqq26s9rorynr4vx4nhxz3fmmxpgtdqp
> [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240880853
>
> > On Jul 11, 2023, at 4:11 PM, Yuan Mei  wrote:
> >
> > As a second thought, I think "Eager State Declaration" is probably not a
> > must-have.
> >
> > I was originally thinking it is a prerequisite for "state querying for
> > disaggregated state management".
> >
> > Since disaggregated state management itself is not a must-have, "Eager
> > State Declaration" is not as well. We can downgrade it to "nice to have"
> if
> > no objection.
> >
> > Best
> >
> > Yuan
> >
> > On Mon, Jul 10, 2023 at 7:02 PM Jing Ge 
> wrote:
> >
> >> +1
> >>
> >> On Mon, Jul 10, 2023 at 12:52 PM Yu Li  wrote:
> >>
> >>> +1 (binding)
> >>>
> >>> Thanks for driving this and great to see us moving forward.
> >>>
> >>> Best Regards,
> >>> Yu
> >>>
> >>>
> >>> On Mon, 10 Jul 2023 at 11:59, Feng Wang  wrote:
> >>>
> >>>> +1
> >>>> Thanks for driving this, looking forward to the next stage of flink.
> >>>>
> >>>> On Fri, Jul 7, 2023 at 5:31 PM Xintong Song 
> >>> wrote:
> >>>>
> >>>>> Hi all,
> >>>>>
> >>>>> I'd like to start the VOTE for the must-have work items for release
> >> 2.0
> >>>>> [1]. The corresponding discussion thread is [2].
> >>>>>
> >>>>> Please note that once the vote is approved, any changes to the
> >>> must-have
> >>>>> items (adding / removing must-have items, changing the priority)
> >>> requires
> >>>>> another vote. Assigning contributors / reviewers, updating
> >>> descriptions /
> >>>>> progress, changes to nice-to-have items do not require another vote.
> >>>>>
> >>>>> The vote will be open until at least July 12, following the consensus
> >>>>> voting process. Votes of PMC members are binding.
> >>>>>
> >>>>> Best,
> >>>>>
> >>>>> Xintong
> >>>>>
> >>>>>
> >>>>> [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> >>>>>
> >>>>> [2] https://lists.apache.org/thread/l3dkdypyrovd3txzodn07lgdwtwvhgk4
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: [VOTE] Release 2.0 must-have work items

[Jing Ge]

Hi Galen,

We were aware of the issue and are working on it. StreamingFileSink is a
SinkFunction that could not be removed yes as mentioned previously. You can
find SinkV1 at [1]

Best regards,
Jing


[1]
https://github.com/apache/flink/blob/4cf2124d71a8dd0595e40f07c2dbcc4c85883b82/flink-core/src/main/java/org/apache/flink/api/connector/sink/Sink.java#L55

On Tue, Jul 11, 2023 at 1:59 PM Galen Warren
 wrote:

> Regarding SinkV1 vs. SinkV2: Is StreamingFileSink a SinkV1-related
> interface that is proposed to be removed? In a separate thread, it was
> discussed how it's important not to remove StreamingFileSink as long as
> this critical issue with SinkV2 is still outstanding --
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/FLINK-30238 --
> because of the prospect of data loss when stopping and restarting jobs with
> savepoints.
>
> Thanks,
> Galen
>
> On Tue, Jul 11, 2023 at 7:47 AM Leonard Xu  wrote:
>
> > Hi, Xintong
> >
> > > Could you please clarify what exact changes you are proposing to make
> on
> > > the existing list?
> > > - Are you suggesting removing the item "Remove deprecated APIs -
> > > SourceFunction / SinkFunction / SinkV1", or are you suggesting
> > downgrading
> > > it as nice-to-have?
> >
> > I prefer to remove the item as we cannot deprecate  SourceFunction /
> > SinkFunction related interfaces in 1.18, thus he 2.0 version would not
> > satisfy two minor versions condition and would not remove them as well.
> >
> > > - You said SinkV2 cannot cover SinkFunction. Then how about SinkV1? Is
> it
> > > covered by SinkV2? Should it be removed or preserved?
> >
> > SinkV2 related interfaces covers SinkV1 related interfaces well, and
> > SinkV1 related interfaces have been deprecated, I think they can be
> removed
> > in 2.0 safely.
> >
> > In a word, my proposal is replace must have item "Remove deprecated APIs
> -
> > SourceFunction / SinkFunction / SinkV1"  with must have item "Remove
> > deprecated APIs  SinkV1" .
> >
> > Best,
> > Leonard
> >
> >
> >
> >
> >
> >
> >
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Tue, Jul 11, 2023 at 4:26 PM Leonard Xu  wrote:
> > >
> > >> Thanks Xintong for driving this great work! But I’ve to give my
> > >> -1(binding) here：
> > >>
> > >> -1 to mark "deprecat SourceFunction/SinkFunction/Sinkv1" item as must
> to
> > >> have for release 2.0.
> > >>
> > >> I do a lot of connector work in the community, and I have two insights
> > >> from past experience:
> > >>
> > >> 1. Many developers reported that it is very difficult to migrate from
> > >> SourceFunction to new Source [1]. The migration of existing conenctors
> > >> after deprecated SourceFunction is very difficult. Some developers
> > (Flavio
> > >> Pompermaier) reported that they gave up the migration because it was
> too
> > >> complicated. I believe it's not a few cases. This means that
> deprecating
> > >> SourceFunction related interfaces require community contributors to
> > reduce
> > >> the migration cost before starting the migration work.
> > >>
> > >> 2. IIRC, the function of SinkV2 cannot currently cover SinkFunction as
> > >> described in FLIP-287[2], it means the migration path after deprecate
> > >> SinkFunction/Sinkv1 does not exist, thus we cannot mark the related
> > >> interfaces of sinkfunction/sinkv1  as deprecated in 1.18.
> > >>
> > >> Based on these two cognitions, I think we should not mark these
> > interfaces
> > >> as must to have in 2.0. Maintaining the two sets of source/sink
> > interfaces
> > >> is not a concern for me, users can choose the interface to implement
> > >> according to their energy and needs.
> > >>
> > >> Btw, some work items in 2.0 are marked as must to have, but no
> > contributor
> > >> has claimed them yet. I think this is a risk and hope the Release
> > Managers
> > >> could pay attention to it.
> > >>
> > >> Thank you all RMs for your work, sorry again for interrupting the vote
> > >>
> > >> Best,
> > >> Leonard
> > >>
> > >> [1] https://lists.apache.org/thread/sqq26s9rorynr4vx4nhxz3fmmxpgtdqp
> > >> [2]
> &g

Re: [DISCUSS] FLIP 333 - Redesign Apache Flink website

[Jing Ge]

Hi,

+1, the UI design looks good!

Commonly speaking, there are two parts of the whole website: Flink Web and
Flink doc. Will the dart mode also cover Flink doc?

Best regards,
Jing

On Tue, Jul 11, 2023 at 12:40 PM Matthias Pohl
 wrote:

> I also like the proposed designs. Considering that you want to touch
> individual subpages, there are also some subpages of Flink's website not
> being mentioned in the FLIP (e.g. roadmap [1]). What is the plan with
> those? Are they covered by the "We recommend modifications only to the
> design of the following pages" part but are not listed there?
>
> Additionally, it would be nice to get a bit more insight into the feedback
> from users as Xintong pointed out. It would be interesting to understand
> how the new design helps solving certain problems (besides having the
> motivation to modernize the look and feel).
>
> I'm also wondering whether it's doable to do a discussion (FLIP?) per
> subpage on the design (as proposed by Chesnay in the first discussion on
> this topic [2] to have smaller changes rather than a single big one). But I
> could imagine this being quite tedious because different people might
> have different opinions on how something should be done.
>
> I don't have any experience with frontend design. I'm wondering how much
> time such a redesign takes. Could it be linked to the 2.0 release?
>
> Thanks for the FLIP (you might want to add [2] to the FLIP's header for the
> sake of transparency).
> Matthias
>
> [1] https://flink.apache.org/roadmap/
> [2] https://lists.apache.org/thread/c3pt00cf77lrtgt242p26lgp9l2z5yc8
>
> On Tue, Jul 11, 2023 at 11:39 AM Xintong Song 
> wrote:
>
> > +1 in general.
> >
> > Thanks for proposing this contribution, Deepthi. The new design looks
> very
> > cool.
> >
> > I have a few questions, which might be entry-level given that I barely
> know
> > anything about the website design.
> > - Do you think it's feasible to maintain two sets of website designs at
> the
> > same time? E.g., adding a button "back to previous version". I'm asking
> > because, while the new UI might be more friendly to newcomers, the
> original
> > website might be more convenient for people who are already familiar with
> > it to find things. It would be nice if we can offer both options to
> users.
> > - For the documentation, I wonder if it makes sense to offer the same
> color
> > theme as the website, to keep the experience consistent. How much effort
> > does it require?
> > - In the FLIP, you mentioned things like "there's a general consensus"
> and
> > "feedback from customers". I'm curious where these come from. Have you
> > conducted some sort of survey? Would you mind sharing a bit more about
> > that?
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Tue, Jul 11, 2023 at 4:57 PM Feifan Wang  wrote:
> >
> > > +1 , the new design looks more attractive and is well organized
> > >
> > > |
> > > |
> > > Feifan Wang
> > > |
> > > |
> > > zoltar9...@163.com
> > > |
> > >
> > >
> > >  Replied Message 
> > > | From | Leonard Xu |
> > > | Date | 07/11/2023 16:34 |
> > > | To | dev |
> > > | Subject | Re: [DISCUSS] FLIP 333 - Redesign Apache Flink website |
> > > +1 for the redesigning, the new website looks cool.
> > >
> > >
> > > Best,
> > > Leonard
> > >
> > > On Jul 11, 2023, at 7:55 AM, Mohan, Deepthi  >
> > > wrote:
> > >
> > > Hi,
> > >
> > > I’m opening this thread to discuss a proposal to redesign the Apache
> > Flink
> > > website: https://flink.apache.org. The approach and a few initial
> > mockups
> > > are included in FLIP 333 - Redesign Apache Flink website.<
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-333%3A+Redesign+Apache+Flink+website
> > > >
> > >
> > > The goal is to modernize the website design to help existing and new
> > users
> > > easily understand Flink’s value proposition and make Flink attractive
> to
> > > new users. As suggested in a previous thread, there are no proposed
> > changes
> > > to Flink documentation.
> > >
> > > I look forward to your feedback and the discussion.
> > >
> > > Thanks,
> > > Deepthi
> > >
> > >
> > >
> >
>

Re: [DISCUSS] FLIP-327: Support stream-batch unified operator to improve job throughput when processing backlog data

[Jing Ge]

Hi Dong,

Thanks for the proposal! The FLIP is already in good shape. I got some NIT
questions.

1. It is a little bit weird to write the hint right after the motivation
that some features have been moved to FLIP-331, because at that time,
readers don't know the context about what features does it mean. I would
suggest moving the note to the beginning of "Public interfaces" sections.
2. It is also a little bit weird to describe all behaviour changes at first
but only focus on one single feature, i.e. how to implement
internalSorterSupported. TBH, I was lost while I was reading the Public
interfaces. Maybe change the FLIP title? Another option could be to write a
short summary of all features and point out that this FLIP will only focus
on the internalSorterSupported feature. Others could be found in FLIP-331.
WDYT?
3. There should be a typo at 4) Checkpoint and failover strategy -> Mixed
mode ->

   - If any task fails when isBacklog=false true, this task is restarted to
   re-process its input from the beginning.



Best regards
Jing


On Thu, Jul 6, 2023 at 1:24 PM Dong Lin  wrote:

> Hi Piotr,
>
> Thanks for your comments! Please see my reply inline.
>
> On Wed, Jul 5, 2023 at 11:44 PM Piotr Nowojski 
> wrote:
>
> > Hi Dong,
> >
> > I have a couple of questions.
> >
> > Could you explain why those properties
> >
> > @Nullable private Boolean isOutputOnEOF = null;
> > @Nullable private Boolean isOutputOnCheckpoint = null;
> > @Nullable private Boolean isInternalSorterSupported = null;
> >
> > must be `@Nullable`, instead of having the default value set to `false`?
> >
>
> By initializing these private variables in OperatorAttributesBuilder as
> null, we can implement `OperatorAttributesBuilder#build()` in such a way
> that it can print DEBUG level logging to say "isOutputOnCheckpoint is not
> explicitly set". This can help user/SRE debug performance issues (or lack
> of the expected optimization) due to operators not explicitly setting the
> right operator attribute.
>
> For example, we might want a job to always use the longer checkpointing
> interval (i.e. execution.checkpointing.interval-during-backlog) if all
> running operators have isOutputOnCheckpoint==false, and use the short
> checkpointing interval otherwise. If a user has explicitly configured the
> execution.checkpointing.interval-during-backlog but the two-phase commit
> sink library has not been upgraded to set isOutputOnCheckpoint=true, then
> the job will end up using the long checkpointing interval, and it will be
> useful to figure out what is going wrong in this case by checking the log.
>
> Note that the default value of these fields of the OperatorAttributes
> instance built by OperatorAttributesBuilder will still be false. The
> following is mentioned in the Java doc of
> `OperatorAttributesBuilder#build()`:
>
>  /**
>   * If any operator attribute is null, we will log it at DEBUG level and
> use the following
>   * default values.
>   * - isOutputOnEOF defaults to false
>   * - isOutputOnCheckpoint defaults to false
>   * - isInternalSorterSupported defaults to false
>   */
>
>
> >
> > Second question, have you thought about cases where someone is
> > either bootstrapping from a streaming source like Kafka
> > or simply trying to catch up after a long period of downtime in a purely
> > streaming job? Generally speaking a cases where
> > user doesn't care about latency in the catch up phase, regardless if the
> > source is bounded or unbounded, but wants to process
> > the data as fast as possible, and then switch dynamically to real time
> > processing?
> >
>
> Yes, I have thought about this. We should allow this job to effectively run
> in batch mode when the job is in the catch-up phase. FLIP-327 is actually
> an important step toward addressing this use-case.
>
> In order to address the above use-case, all we need is a way for source
> operator (e.g. Kafka) to tell Flink runtime (via IsProcessingBacklog)
> whether it is in the catch-up phase.
>
> Since every Kafka message has event-timestamp, we can allow users to
> specify a job-level config such as backlog-watermark-lag-threshold, and
> consider a Kafka Source to have IsProcessingBacklog=true if system_time -
> watermark > backlog-watermark-lag-threshold. This effectively allows us to
> determine whether Kafka is in the catch up phase.
>
> Once we have this capability (I plan to work on this in FLIP-328), we can
> directly use the features proposed in FLIP-325 and FLIP-327 to optimize the
> above use-case.
>
> What do you think?
>
> Best,
> Dong
>
>
> >
> > Best,
> > Piotrek
> >
> > niedz., 2 lip 2023 o 16:15 Dong Lin  napisał(a):
> >
> > > Hi all,
> > >
> > > I am opening this thread to discuss FLIP-327: Support stream-batch
> > unified
> > > operator to improve job throughput when processing backlog data. The
> > design
> > > doc can be found at
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-327%3A+Support+stream-batch+unified+operator+t

Re: [DISCUSS] FLIP-325: Support configuring end-to-end allowed latency

[Jing Ge]

Hi Dong,

Thanks for the update!

Best regards,
Jing

On Sun, Jul 9, 2023 at 3:26 AM Dong Lin  wrote:

> Hi Jing,
>
> Thanks for the suggestions. Please see my reply inline.
>
> On Fri, Jul 7, 2023 at 3:50 PM Jing Ge  wrote:
>
> > Hi Dong,
> >
> > Thanks for your clarification.
> >
> >
> > > Actually, I think it could make sense to toggle isBacklog between true
> > and
> > > false while the job is running.
> > >
> >
> > If isBacklog is toggled too often back and forth(e.g. by unexpected
> > mistake, unstable system, etc), a large amount of RecordAttributes might
> be
> > triggered, which will lead to performance issues. This should not be the
> > right way to use RecordAttributes right? Devs and users should be aware
> of
> > it and know how to monitor, maintain, and fix issues.
> >
> > Your reply contains valuable information. It might make sense to add them
> > into the FLIP:
> >
> > 1. It is up to the operator to decide when to emit RecordAttributes. But
> > devs and users should be aware that the number of RecordAttributes should
> > not be too high to cause performance issues.
> >
>
> Sure, I have updated the FLIP to include the following statement:
>
> "Note: It is up to the operator implementation to decide when (and how
> often) to emit RecordAttributes, similar to how operators emit
> RecordAttributes. The overhead of emitting Watermark is similar to the
> overhead of emitting Watermark"
>
>
> > 2. Although users can decide how to configure them, the end-to-end
> latency
> > should be (commonly?) configured lower than the checkpoint interval.
> >
>
> Since this is related to the performance tuning rather than
> correctness/functionality of the core APIs, I added the following sentence
> in the Appendix section:
>
> "We expect that end-to-end latency will typically be configured with a
> value lower than the checkpoint interval"
>
> 3. The three ways you mentioned for how to derive isBacklog.
> >
>
> Sure, I have updated the FLIP to include the following information:
>
> "In the future, we expect IsProcessingBacklog can very likely be determined
> using the following strategies ..."
>
> Best,
> Dong
>
>
> >
> > WDYT?
> >
> > Best regards,
> > Jing
> >
> >
> > On Fri, Jul 7, 2023 at 3:13 AM Dong Lin  wrote:
> >
> > > Hi Jing,
> > >
> > > Thanks for the comments. Please see my reply inline.
> > >
> > > On Fri, Jul 7, 2023 at 5:40 AM Jing Ge 
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > Thank you all for the inspired discussion. Really appreciate it!
> > > >
> > > > @Dong I'd like to ask some (stupid) questions to make sure I
> understand
> > > > your thoughts correctly.
> > > >
> > > > 1. It will make no sense to send the same type of RecordAttributes
> > right?
> > > > e.g.  if one RecordAttributes(isBacklog=true) has been sent, a new
> > > > RecordAttributes will be only sent when isBacklog is changed to be
> > false,
> > > > and vice versa. In this way, the number of RecordAttributes will be
> > very
> > > > limited.
> > > >
> > >
> > > Yes, you are right. Actually, this is what we plan to do when we update
> > > operators to emit RecordAttributes via `Output#emitRecordAttributes()`.
> > >
> > > Note that the FLIP does not specify the frequency of how operators
> should
> > > invoke `Output#emitRecordAttributes()`. It is up to the operator
> > > to decide when to emit RecordAttributes.
> > >
> > >
> > > > 2. Since source readers can invoke Output#emitRecordAttributes to
> emit
> > > > RecordAttributes(isBacklog=true/false), it might be weird to send
> > > > RecordAttributes with different isBacklog back and forth too often.
> > Devs
> > > > and users should pay attention to it. Something is wrong when such a
> > > thing
> > > > happens(metrics for monitoring?). Is this correct?
> > > >
> > >
> > >
> >
> > > Actually, I think it could make sense to toggle isBacklog between true
> > and
> > > false while the job is running.
> > >
> > >
> >
> > > Suppose the job is reading from user-action data from Kafka and there
> is
> > a
> > > traffic spike for 2 hours. If the job keeps running in pure stream
> mode,
> > > 

[VOTE] Graduate the FileSink to @PublicEvolving

[Jing Ge]

Hi,

I'd like to start the VOTE for graduating the FlinkSink to @PublicEvolving.
The discussion thread can be found at [1]

The vote will be open until at least July 13 12pm GMT(72 hours) unless
there is an objection or insufficient votes.

Thanks,

Jing Ge

[1] https://lists.apache.org/thread/wxoo7py5pqqlz37l4w8jrq6qdvsdq5wc

Re: [VOTE] Release 2.0 must-have work items

[Jing Ge]

+1

On Mon, Jul 10, 2023 at 12:52 PM Yu Li  wrote:

> +1 (binding)
>
> Thanks for driving this and great to see us moving forward.
>
> Best Regards,
> Yu
>
>
> On Mon, 10 Jul 2023 at 11:59, Feng Wang  wrote:
>
> > +1
> > Thanks for driving this, looking forward to the next stage of flink.
> >
> > On Fri, Jul 7, 2023 at 5:31 PM Xintong Song 
> wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start the VOTE for the must-have work items for release 2.0
> > > [1]. The corresponding discussion thread is [2].
> > >
> > > Please note that once the vote is approved, any changes to the
> must-have
> > > items (adding / removing must-have items, changing the priority)
> requires
> > > another vote. Assigning contributors / reviewers, updating
> descriptions /
> > > progress, changes to nice-to-have items do not require another vote.
> > >
> > > The vote will be open until at least July 12, following the consensus
> > > voting process. Votes of PMC members are binding.
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > > [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> > >
> > > [2] https://lists.apache.org/thread/l3dkdypyrovd3txzodn07lgdwtwvhgk4
> > >
> >
>

Re: [DISUCSS] Deprecate multiple APIs in 1.18

[Jing Ge]

Hi Alex,

I would follow FLIP-197 and try to release them asap depending on dev
resources and how difficult those issues are. The fastest timeline is the
period defined in FLIP-197 in ideal conditions.

Best regards,
Jing

On Fri, Jul 7, 2023 at 12:20 PM Alexander Fedulov <
alexander.fedu...@gmail.com> wrote:

> @Xintong
> > - IIUC, the testing scenario you described is like blocking the source
> for
> > proceeding (emit data, finish, etc.) until a checkpoint is finished.
>
> It is more tricky than that - we need to prevent the Sink from receiving a
> checkpoint barrier until the Source is done emitting a given set of
> records. In
> the current tests, which are also used for V2 Sinks, SourceFunction
> controls
> when the Sink is "allowed" to commit by holding the checkpoint lock while
> producing the records. The lock is not available in the new Source by
> design
> and we need a solution that provides the same functionality (without
> modifying
> the Sinks). I am currently checking if a workaround is at all possible
> without
> adjusting anything in the Source interface.
>
> > I may not have understood all the details, but based on what you
> described
> > I'd hesitate to block the deprecation / removal of SourceFunction on
> this.
>
> I don't think we should, just wanted to highlight that there are some
> unknowns
> with respect to estimating the amount of work required.
>
> @Jing
> I want to understand in which release would you target graduation of the
> mentioned connectors to @Public/@PublicEvolving - basically the anticipated
> timeline of the steps in both options with respect to releases.
>
> Best,
> Alex
>
> On Fri, 7 Jul 2023 at 10:53, Xintong Song  wrote:
>
> > Thanks all for the discussion. I've created FLINK-32557 for this.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Thu, Jul 6, 2023 at 1:00 AM Jing Ge 
> wrote:
> >
> > > Hi Alex,
> > >
> > >
> > > > > 3. remove SinkFunction.
> > > > Which steps do you imply for the 1.18 release and for the 2.0
> release?
> > > >
> > >
> > > for 2.0 release. 1.18 will be released soon.
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > > On Wed, Jul 5, 2023 at 1:08 PM Alexander Fedulov <
> > > alexander.fedu...@gmail.com> wrote:
> > >
> > > > @Jing
> > > > Just to clarify, when you say:
> > > >
> > > > 3. remove SinkFunction.
> > > > Which steps do you imply for the 1.18 release and for the 2.0
> release?
> > >
> > > @Xintong
> > > > A side note - with the new Source API we lose the ability to control
> > > > checkpointing from the source since there is no lock anymore. This
> > > > functionality
> > > > is currently used in a variety of tests for the Sinks - the tests
> that
> > > rely
> > > > on tight
> > > > synchronization between specific elements passed from the source  to
> > the
> > > > sink before
> > > > allowing a checkpoint to complete (see FiniteTestSource [1]). Since
> > > FLIP-27
> > > > Sources rely
> > > > on decoupling via the mailbox, without exposing the lock, it is not
> > > > immediately clear
> > > > if it is possible to achieve the same functionality without major
> > > > extensions in the
> > > > runtime for such testing purposes. My hope initially was that only
> the
> > > > legacy Sinks
> > > > relied on this - this would have made it possible to drop
> > > > SourceFunction+SinkFunction
> > > > together, but, in fact, it also already became part of the new SinkV2
> > > > testing IT suits
> > > > [2]. Moreover, I know of at least one major connector that also
> relies
> > on
> > > > it for
> > > > verifying committed sink metadata for a specific set of records
> > (Iceberg)
> > > > [3]. In my
> > > > estimation this currently presents a major blocker for the
> > SourceFunction
> > > > removal.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/FiniteTestSource.java
> > > > [2]
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-connectors/flink-conne

Re: [DISCUSS] FLIP-325: Support configuring end-to-end allowed latency

[Jing Ge]

Hi Dong,

Thanks for your clarification.


> Actually, I think it could make sense to toggle isBacklog between true and
> false while the job is running.
>

If isBacklog is toggled too often back and forth(e.g. by unexpected
mistake, unstable system, etc), a large amount of RecordAttributes might be
triggered, which will lead to performance issues. This should not be the
right way to use RecordAttributes right? Devs and users should be aware of
it and know how to monitor, maintain, and fix issues.

Your reply contains valuable information. It might make sense to add them
into the FLIP:

1. It is up to the operator to decide when to emit RecordAttributes. But
devs and users should be aware that the number of RecordAttributes should
not be too high to cause performance issues.
2. Although users can decide how to configure them, the end-to-end latency
should be (commonly?) configured lower than the checkpoint interval.
3. The three ways you mentioned for how to derive isBacklog.

WDYT?

Best regards,
Jing


On Fri, Jul 7, 2023 at 3:13 AM Dong Lin  wrote:

> Hi Jing,
>
> Thanks for the comments. Please see my reply inline.
>
> On Fri, Jul 7, 2023 at 5:40 AM Jing Ge  wrote:
>
> > Hi,
> >
> > Thank you all for the inspired discussion. Really appreciate it!
> >
> > @Dong I'd like to ask some (stupid) questions to make sure I understand
> > your thoughts correctly.
> >
> > 1. It will make no sense to send the same type of RecordAttributes right?
> > e.g.  if one RecordAttributes(isBacklog=true) has been sent, a new
> > RecordAttributes will be only sent when isBacklog is changed to be false,
> > and vice versa. In this way, the number of RecordAttributes will be very
> > limited.
> >
>
> Yes, you are right. Actually, this is what we plan to do when we update
> operators to emit RecordAttributes via `Output#emitRecordAttributes()`.
>
> Note that the FLIP does not specify the frequency of how operators should
> invoke `Output#emitRecordAttributes()`. It is up to the operator
> to decide when to emit RecordAttributes.
>
>
> > 2. Since source readers can invoke Output#emitRecordAttributes to emit
> > RecordAttributes(isBacklog=true/false), it might be weird to send
> > RecordAttributes with different isBacklog back and forth too often. Devs
> > and users should pay attention to it. Something is wrong when such a
> thing
> > happens(metrics for monitoring?). Is this correct?
> >
>
>

> Actually, I think it could make sense to toggle isBacklog between true and
> false while the job is running.
>
>

> Suppose the job is reading from user-action data from Kafka and there is a
> traffic spike for 2 hours. If the job keeps running in pure stream mode,
> the watermark lag might keep increasing during this period because the
> job's processing capability can not catch up with the Kafka input
> throughput. In this case, it can be beneficial to dynamically switch
> isBacklog to true when watermarkLag exceeds a given threshold (e.g. 5
> minutes), and switch isBacklog to false again when the watermarkLag is low
> enough (30 seconds).
>
>
> > 3. Is there any relationship between end-to-end-latency and checkpoint
> > interval that users should pay attention to? In the example described in
> > the FLIP, both have the same value, 2 min. What about end-to-end-latency
> is
> > configured bigger than checkpoint interval? Could checkpoint between
> > end-to-end-latency be skipped?
> >
>
> This FLIP would not enforce any relationship between end-to-end latency and
> checkpoint interval. Users are free to configure end-to-end latency to be
> bigger than checkpoint interval.
>
> I don't think there exists any use-case which requires end-to-end latency
> to be higher than the checkpoint interval. Note that introducing a
> relationship between these two configs would increase code complexity and
> also make the documentation of these configs a bit more complex for users
> to understand.
>
> Since there is no correctness when a user sets end-to-end latency to be
> bigger than the checkpointing interval, I think it is simpler to just let
> the user decide how to configure them.
>
>
> > 4. Afaiu, one major discussion point is that isBacklog can be derived
> from
> > back pressure and there will be no need of RecordAttributes. In case a
> > Flink job has rich resources that there is no back pressure (it will be
> > difficult to perfectly have just enough resources that everything is fine
> > but will have back pressure only for backlog) but we want to improve the
> > throughput. We then need some other ways to derive isBacklog. That is the
> > reason why RecordAttributes has been i

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

[Jing Ge]

Hi Shammon,

Thanks for the update!

Best regards,
Jing

On Fri, Jul 7, 2023 at 4:46 AM Shammon FY  wrote:

> Thanks Jing, sounds good to me.
>
> I have updated the FLIP and renamed the lineage related classes to
> `LineageGraph`, `LineageVertex` and `LineageEdge` and keep it consistent
> with the job definition in Flink.
>
> Best,
> Shammon FY
>
> On Thu, Jul 6, 2023 at 8:25 PM Jing Ge  wrote:
>
> > Hi Shammon,
> >
> > Thanks for the clarification. Atlas might have his historical reason back
> > to the hadoop era or maybe even back to the hibernate where Entity and
> > Relation were commonly used. Flink already used Vertex and Edge to
> describe
> > DAG. Some popular tools like dbt are also using this convention[1] and,
> > afaik, most graph frameworks use vertex and edge too. It will be easier
> for
> > Flink devs and users to have a consistent naming convention for the same
> > concept, i.e. in this case, DAG.
> >
> > Best regards,
> > Jing
> >
> > [1]
> >
> >
> https://docs.getdbt.com/docs/dbt-cloud-apis/discovery-use-cases-and-examples#discovery
> >
> > On Wed, Jul 5, 2023 at 11:28 AM Shammon FY  wrote:
> >
> > > Hi Jing,
> > >
> > > Thanks for your feedback.
> > >
> > > > 1. TableColumnLineageRelation#sinkColumn() should return
> > > TableColumnLineageEntity instead of String, right?
> > >
> > > The `sinkColumn()` will return `String` which is the column name in the
> > > sink connector. I found the name of `TableColumnLineageEntity` may
> > > cause ambiguity and I have renamed it to
> > `TableColumnSourceLineageEntity`.
> > > In my mind the `TableColumnLineageRelation` represents the lineage for
> > each
> > > sink column, each column may be computed from multiple sources and
> > columns.
> > > I use `TableColumnSourceLineageEntity` to manage each source and its
> > > columns for the sink column, so `TableColumnLineageRelation` has a sink
> > > column name and `TableColumnSourceLineageEntity` list.
> > >
> > > > 2. Since LineageRelation already contains all information to build
> the
> > > lineage between sources and sink, do we still need to set the
> > LineageEntity
> > > in the source?
> > >
> > > The lineage interface of `DataStream` is very flexible. We have added
> > > `setLineageEntity` to the source to limit and verify user behavior,
> > > ensuring that users have not added non-existent sources as lineage.
> > >
> > > > 3. About the "Entity" and "Relation" naming, I was confused too, like
> > > Qingsheng mentioned. How about LineageVertex, LineageEdge, and
> > LineageEdges
> > > which contains multiple LineageEdge?
> > >
> > > We referred to `Atlas` for the name of lineage, it uses `Entity` and
> > > `Relation` to represent the lineage relationship and another metadata
> > > service `Datahub` uses `DataSet` to represent the entity. I think
> > `Entity`
> > > and `Relation` are nicer for lineage, what do you think of it?
> > >
> > > Best,
> > > Shammon FY
> > >
> > >
> > > On Thu, Jun 29, 2023 at 4:21 AM Jing Ge 
> > > wrote:
> > >
> > > > Hi Shammon,
> > > >
> > > > Thanks for your proposal. After reading the FLIP, I'd like to ask
> > > > some questions to make sure we are on the same page. Thanks!
> > > >
> > > > 1. TableColumnLineageRelation#sinkColumn() should return
> > > > TableColumnLineageEntity instead of String, right?
> > > >
> > > > 2. Since LineageRelation already contains all information to build
> the
> > > > lineage between sources and sink, do we still need to set the
> > > LineageEntity
> > > > in the source?
> > > >
> > > > 3. About the "Entity" and "Relation" naming, I was confused too, like
> > > > Qingsheng mentioned. How about LineageVertex, LineageEdge, and
> > > LineageEdges
> > > > which contains multiple LineageEdge? E.g. multiple sources join into
> > one
> > > > sink, or, edges of columns from one or different tables, etc.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Sun, Jun 25, 2023 at 2:06 PM Shammon FY 
> wrote:
> > > >
> > > > > Hi yuxia and Yun,
> > > > >
> > > > > Thanks for your input.
> > > >

Re: [DISCUSS] FLIP-325: Support configuring end-to-end allowed latency

[Jing Ge]

Hi,

Thank you all for the inspired discussion. Really appreciate it!

@Dong I'd like to ask some (stupid) questions to make sure I understand
your thoughts correctly.

1. It will make no sense to send the same type of RecordAttributes right?
e.g.  if one RecordAttributes(isBacklog=true) has been sent, a new
RecordAttributes will be only sent when isBacklog is changed to be false,
and vice versa. In this way, the number of RecordAttributes will be very
limited.

2. Since source readers can invoke Output#emitRecordAttributes to emit
RecordAttributes(isBacklog=true/false), it might be weird to send
RecordAttributes with different isBacklog back and forth too often. Devs
and users should pay attention to it. Something is wrong when such a thing
happens(metrics for monitoring?). Is this correct?

3. Is there any relationship between end-to-end-latency and checkpoint
interval that users should pay attention to? In the example described in
the FLIP, both have the same value, 2 min. What about end-to-end-latency is
configured bigger than checkpoint interval? Could checkpoint between
end-to-end-latency be skipped?

4. Afaiu, one major discussion point is that isBacklog can be derived from
back pressure and there will be no need of RecordAttributes. In case a
Flink job has rich resources that there is no back pressure (it will be
difficult to perfectly have just enough resources that everything is fine
but will have back pressure only for backlog) but we want to improve the
throughput. We then need some other ways to derive isBacklog. That is the
reason why RecordAttributes has been introduced. Did I understand it
correctly?

5. NIT: Just like we talked about in another thread, JavaBean naming
convention is recommended, i.e. isBacklog() & setBacklog() instead of
getIsBacklog() and setIsBacklog().

Best regards,
Jing

On Thu, Jul 6, 2023 at 2:38 PM Dong Lin  wrote:

> Hi Shammon,
>
> Thanks for your comments. Please see my reply inline.
>
>
> On Thu, Jul 6, 2023 at 12:47 PM Shammon FY  wrote:
>
> > Hi,
> >
> > Thanks for your replay @Dong. I really agree with Piotr's points and I
> > would like to share some thoughts from my side.
> >
> > About the latency for mini-batch mechanism in Flink SQL, I still think
> the
> > description in the FLIP is not right. If there are N operators and the
> > whole process time for data in the job is `t`, then the latency in
> > mini-batch will be `table.exec.mini-batch.allow-latency`+`t`, not `
> > table.exec.mini-batch.allow-latency`*N. I think this is one of the
> > foundations of this FLIP, and you may need to confirm it again.
> >
>
> Given that we agree to have a mechanism to support end-to-end latency for
> DataStream programs, I think the exact semantics of
> table.exec.mini-batch.allow-latency will not affect the motivation or API
> design of this FLIP. I have updated the FLIP to remove any mention of
> table.exec.mini-batch.allow-latency.
>
>
> >
> > I think supporting similar mechanisms in the runtime and balance latency
> > and throughput dynamically for all flink jobs is a very good idea, and I
> > have some questions for that.
> >
> > 1. We encounter a situation where the workload is high when processing
> > snapshot data and we need mini-batch in sql for performance reason. But
> the
> > workload is low when processing delta data, we need to automatically
> adjust
> > the mini-batch SQL for them, or even cancel the mini-batch during delta
> > processing. I think this FLIP meets our needs, but I think we need a
> > general solution which covers all source types in flink, and the
> > `isBacklog` in the FLIP is only one strategy.
> >
>
> The focus of this FLIP is to allow Flink runtime to adjust the behavior of
> operators (e.g. the buffer time) based on the IsBacklog status of sources
> and the user-specified execution.end-to-end-latency (effective only when
> there is no backlog). The FLIP assumes there is already a strategy for
> sources to determine the IsProcessingBacklog status without adding more
> strategies.
>
> I agree it is useful to introduce more strategies to determine the the
> IsProcessingBacklog status for sources. We can determine the
> IsProcessingBacklog status based on the backpressure metrics, the
> event-time watermark lag, or anything we find reasonable. I would like to
> work on this in follow-up FLIPs and that we don't work on too many things
> in the same FLIP.
>
> Would this be OK with you?
>
>
> > From the FLIP I think there should be two parts: dynamic trigger flush
> > event in JM and dynamic trigger flush operations in Operator. We need to
> > introduce much more general interfaces for them, such as
> > `DynamicFlushStrategy` in JM and `DynamicFlushOperation` in TM? As Piotr
> > mentioned above, we can collect many information from TM locally such as
> > backpressure, queue size and `Operator` can decide whether to buffer data
> > or process it immediately.  JM is also the same, it can decide to send
> > flush events on a regular basis or s

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

[Jing Ge]

Hi Shammon,

Thanks for the clarification. Atlas might have his historical reason back
to the hadoop era or maybe even back to the hibernate where Entity and
Relation were commonly used. Flink already used Vertex and Edge to describe
DAG. Some popular tools like dbt are also using this convention[1] and,
afaik, most graph frameworks use vertex and edge too. It will be easier for
Flink devs and users to have a consistent naming convention for the same
concept, i.e. in this case, DAG.

Best regards,
Jing

[1]
https://docs.getdbt.com/docs/dbt-cloud-apis/discovery-use-cases-and-examples#discovery

On Wed, Jul 5, 2023 at 11:28 AM Shammon FY  wrote:

> Hi Jing,
>
> Thanks for your feedback.
>
> > 1. TableColumnLineageRelation#sinkColumn() should return
> TableColumnLineageEntity instead of String, right?
>
> The `sinkColumn()` will return `String` which is the column name in the
> sink connector. I found the name of `TableColumnLineageEntity` may
> cause ambiguity and I have renamed it to `TableColumnSourceLineageEntity`.
> In my mind the `TableColumnLineageRelation` represents the lineage for each
> sink column, each column may be computed from multiple sources and columns.
> I use `TableColumnSourceLineageEntity` to manage each source and its
> columns for the sink column, so `TableColumnLineageRelation` has a sink
> column name and `TableColumnSourceLineageEntity` list.
>
> > 2. Since LineageRelation already contains all information to build the
> lineage between sources and sink, do we still need to set the LineageEntity
> in the source?
>
> The lineage interface of `DataStream` is very flexible. We have added
> `setLineageEntity` to the source to limit and verify user behavior,
> ensuring that users have not added non-existent sources as lineage.
>
> > 3. About the "Entity" and "Relation" naming, I was confused too, like
> Qingsheng mentioned. How about LineageVertex, LineageEdge, and LineageEdges
> which contains multiple LineageEdge?
>
> We referred to `Atlas` for the name of lineage, it uses `Entity` and
> `Relation` to represent the lineage relationship and another metadata
> service `Datahub` uses `DataSet` to represent the entity. I think `Entity`
> and `Relation` are nicer for lineage, what do you think of it?
>
> Best,
> Shammon FY
>
>
> On Thu, Jun 29, 2023 at 4:21 AM Jing Ge 
> wrote:
>
> > Hi Shammon,
> >
> > Thanks for your proposal. After reading the FLIP, I'd like to ask
> > some questions to make sure we are on the same page. Thanks!
> >
> > 1. TableColumnLineageRelation#sinkColumn() should return
> > TableColumnLineageEntity instead of String, right?
> >
> > 2. Since LineageRelation already contains all information to build the
> > lineage between sources and sink, do we still need to set the
> LineageEntity
> > in the source?
> >
> > 3. About the "Entity" and "Relation" naming, I was confused too, like
> > Qingsheng mentioned. How about LineageVertex, LineageEdge, and
> LineageEdges
> > which contains multiple LineageEdge? E.g. multiple sources join into one
> > sink, or, edges of columns from one or different tables, etc.
> >
> > Best regards,
> > Jing
> >
> > On Sun, Jun 25, 2023 at 2:06 PM Shammon FY  wrote:
> >
> > > Hi yuxia and Yun,
> > >
> > > Thanks for your input.
> > >
> > > For yuxia:
> > > > 1: What kinds of JobStatus will the `JobExecutionStatusEven`
> including?
> > >
> > > At present, we only need to notify the listener when a job goes to
> > > termination, but I think it makes sense to add generic `oldStatus` and
> > > `newStatus` in the listener and users can update the job state in their
> > > service as needed.
> > >
> > > > 2: I'm really confused about the `config()` included in
> > `LineageEntity`,
> > > where is it from and what is it for ?
> > >
> > > The `config` in `LineageEntity` is used for users to get options for
> > source
> > > and sink connectors. As the examples in the FLIP, users can add
> > > server/group/topic information in the config for kafka and create
> lineage
> > > entities for `DataStream` jobs, then the listeners can get this
> > information
> > > to identify the same connector in different jobs. Otherwise, the
> `config`
> > > in `TableLineageEntity` will be the same as `getOptions` in
> > > `CatalogBaseTable`.
> > >
> > > > 3: Regardless whether `inputChangelogMode` in
> `TableSinkLineageEntity`
> > is
> > > needed or not, since `TableSinkLineageEntity` contains

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

[Jing Ge]

Hi Dong,

one scenario I could imagine is that users could enable global object reuse
features but force deep copy for some user defined specific functions
because of any limitations. But that is only my gut feeling. And agree, we
could keep the solution simple for now as FLIP described and upgrade to 3VL
once there are such real requirements that are rising.

Best regards,
Jing

On Thu, Jul 6, 2023 at 12:30 PM Dong Lin  wrote:

> Hi Jing,
>
> Thank you for the detailed explanation. Please see my reply inline.
>
> On Thu, Jul 6, 2023 at 3:17 AM Jing Ge  wrote:
>
>> Hi Xuannan, Hi Dong,
>>
>> Thanks for your clarification.
>>
>> @Xuannan
>>
>> A Jira ticket has been created for the doc update:
>> https://issues.apache.org/jira/browse/FLINK-32546
>>
>> @Dong
>>
>> I don't have a concrete example. I just thought about it from a
>> conceptual or pattern's perspective. Since we have 1. coarse-grained global
>> switch(CGS as abbreviation), i.e. the pipeline.object-reuse and 2.
>> fine-grained local switch(FGS as abbreviation), i.e. the
>> objectReuseCompliant variable for specific operators/functions, there will
>> be the following patterns with appropriate combinations:
>>
>> pattern 1: coarse-grained switch only. Local object reuse will be
>> controlled by the coarse-grained switch:
>> 1.1 cgs == true -> local object reused enabled
>> 1.2 cgs == true  -> local object reused enabled
>> 1.3 cgs == false -> local object reused disabled, i.e. deep copy enabled
>> 1.4 cgs == false -> local object reused disabled, i.e. deep copy enabled
>>
>> afaiu, this is the starting point. I wrote 4 on purpose to make the
>> regression check easier. We can consider it as the combinations with
>> cgs(true/false) and fgs(true/false) while fgs is ignored.
>>
>> Now we introduce fine-grained switch. There will be two patterns:
>>
>> pattern 2: fine-grained switch over coarse-grained switch. Coarse-grained
>> switch will be ignored when the local fine-grained switch has different
>> value:
>> 2.1 cgs == true and fgs == true -> local object reused enabled
>> 2.2 cgs == true and fgs == false -> local object reused disabled, i.e.
>> deep copy enabled
>> 2.3 cgs == false and fgs == true -> local object reused enabled
>> 2.4 cgs == false and fgs == false -> local object reused disabled, i.e.
>> deep copy enabled
>>
>> cgs is actually ignored.
>>
>> Current FLIP is using a slightly different pattern:
>>
>> pattern 3: fine-grained switch over coarse-grained switch only when
>> coarse-grained switch is off, i..e cgs OR fgs:
>> 3.1 cgs == true and fgs == true -> local object reused enabled
>> 3.2 cgs == true and fgs == false -> local object reused enabled
>> 3.3 cgs == false and fgs == true -> local object reused enabled
>> 3.4 cgs == false and fgs == false -> local object reused disabled, i.e.
>> deep copy enabled
>>
>> All of those patterns are rational and each has different focus. It
>> depends on the real requirement to choose one of them.
>>
>> As we can see, if fgs is using 2VL, there is a regression between pattern
>> 1 and pattern 2. You are absolutely right in this case. That's why I
>> suggested 3VL, i.e. fgs will have triple values: true, false, unknown(e.g.
>> null)
>>
>> pattern 4: 3VL fgs with the null as init value (again, there are just two
>> combination, I made it 4 on purpose):
>> 4.1 cgs == true and fgs == null -> local object reused enabled
>> 4.2 cgs == true and fgs == null -> local object reused enabled
>> 4.3 cgs == false and fgs == null -> local object reused disabled, i.e.
>> deep copy enabled
>> 4.4 cgs == false and fgs == null -> local object reused disabled, i.e.
>> deep copy enabled
>>
>> Since the default value of fgs is null, pattern 4 is backward compatible
>> with pattern 1, which means no regression.
>>
>> Now we will set value to fgs and follow the pattern 2:
>> 4.5 cgs == true and fgs == true -> local object reused enabled
>> 4.6 cgs == true and fgs == false -> local object reused disabled, i.e.
>> deep copy enabled
>> 4.7 cgs == false and fgs == true -> local object reused enabled
>> 4.8 cgs == false and fgs == false -> local object reused disabled, i.e.
>> deep copy enabled
>>
>> Pattern 4 contains pattern 3 with the following combinations(force
>> enabling local object reuse):
>> 4.5 cgs == true and fgs == true -> local object reused enabled
>> 4.2 cgs == true and fgs == null -> local object reused enab

Re: [ANNOUNCE] Flink 1.18 Feature Freeze Extended until July 24th, 2023

[Jing Ge]

Thanks for driving it and sharing the update!

Best regards,
Jing

On Thu, Jul 6, 2023 at 9:21 AM yuxia  wrote:

> Thanks for the update and thanks for your efforts.
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "Rui Fan" <1996fan...@gmail.com>
> 收件人: "dev" , re...@apache.org
> 抄送: "Jing Ge" , snuyan...@gmail.com, "Konstantin
> Knauf" 
> 发送时间: 星期四, 2023年 7 月 06日 下午 3:06:28
> 主题: Re: [ANNOUNCE] Flink 1.18 Feature Freeze Extended until July 24th, 2023
>
> Thanks for the update, and thank you for your efforts for the 1.18 release!
>
> Best,
> Rui Fan
>
> On Thu, Jul 6, 2023 at 2:40 PM Qingsheng Ren  wrote:
>
> > Hi devs,
> >
> > Recently we collected some feedback from developers, and in order to give
> > more time for polishing some important features in 1.18, we decide to
> > extend the feature freezing date to:
> >
> > July 24th, 2023, at 00:00 CEST(UTC+2)
> >
> > which gives us ~2 weeks for development from now. There will be no
> > extension after Jul 24, so please arrange new features in the next
> release
> > if they cannot be finished before the closing date.
> >
> > Thanks everyone for your work in 1.18!
> >
> > Best regards,
> > Qingsheng, Jing, Konstantin and Sergey
> >
>

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

[Jing Ge]

Hi Xuannan, Hi Dong,

Thanks for your clarification.

@Xuannan

A Jira ticket has been created for the doc update:
https://issues.apache.org/jira/browse/FLINK-32546

@Dong

I don't have a concrete example. I just thought about it from a conceptual
or pattern's perspective. Since we have 1. coarse-grained global switch(CGS
as abbreviation), i.e. the pipeline.object-reuse and 2. fine-grained local
switch(FGS as abbreviation), i.e. the objectReuseCompliant variable for
specific operators/functions, there will be the following patterns with
appropriate combinations:

pattern 1: coarse-grained switch only. Local object reuse will be
controlled by the coarse-grained switch:
1.1 cgs == true -> local object reused enabled
1.2 cgs == true  -> local object reused enabled
1.3 cgs == false -> local object reused disabled, i.e. deep copy enabled
1.4 cgs == false -> local object reused disabled, i.e. deep copy enabled

afaiu, this is the starting point. I wrote 4 on purpose to make the
regression check easier. We can consider it as the combinations with
cgs(true/false) and fgs(true/false) while fgs is ignored.

Now we introduce fine-grained switch. There will be two patterns:

pattern 2: fine-grained switch over coarse-grained switch. Coarse-grained
switch will be ignored when the local fine-grained switch has different
value:
2.1 cgs == true and fgs == true -> local object reused enabled
2.2 cgs == true and fgs == false -> local object reused disabled, i.e. deep
copy enabled
2.3 cgs == false and fgs == true -> local object reused enabled
2.4 cgs == false and fgs == false -> local object reused disabled, i.e.
deep copy enabled

cgs is actually ignored.

Current FLIP is using a slightly different pattern:

pattern 3: fine-grained switch over coarse-grained switch only when
coarse-grained switch is off, i..e cgs OR fgs:
3.1 cgs == true and fgs == true -> local object reused enabled
3.2 cgs == true and fgs == false -> local object reused enabled
3.3 cgs == false and fgs == true -> local object reused enabled
3.4 cgs == false and fgs == false -> local object reused disabled, i.e.
deep copy enabled

All of those patterns are rational and each has different focus. It depends
on the real requirement to choose one of them.

As we can see, if fgs is using 2VL, there is a regression between pattern 1
and pattern 2. You are absolutely right in this case. That's why I
suggested 3VL, i.e. fgs will have triple values: true, false, unknown(e.g.
null)

pattern 4: 3VL fgs with the null as init value (again, there are just two
combination, I made it 4 on purpose):
4.1 cgs == true and fgs == null -> local object reused enabled
4.2 cgs == true and fgs == null -> local object reused enabled
4.3 cgs == false and fgs == null -> local object reused disabled, i.e. deep
copy enabled
4.4 cgs == false and fgs == null -> local object reused disabled, i.e. deep
copy enabled

Since the default value of fgs is null, pattern 4 is backward compatible
with pattern 1, which means no regression.

Now we will set value to fgs and follow the pattern 2:
4.5 cgs == true and fgs == true -> local object reused enabled
4.6 cgs == true and fgs == false -> local object reused disabled, i.e. deep
copy enabled
4.7 cgs == false and fgs == true -> local object reused enabled
4.8 cgs == false and fgs == false -> local object reused disabled, i.e.
deep copy enabled

Pattern 4 contains pattern 3 with the following combinations(force enabling
local object reuse):
4.5 cgs == true and fgs == true -> local object reused enabled
4.2 cgs == true and fgs == null -> local object reused enabled
4.7 cgs == false and fgs == true -> local object reused enabled
4.4 cgs == false and fgs == null -> local object reused disabled, i.e. deep
copy enabled

Comparing pattern 4 to pattern 3, user will have one additional flexibility
to control(force disabling) the local object reuse capability because of
3VL, i.e. 4.2+4.6 vs. 3.2.

It is commonly used in the hierarchical RBAC to enable more fine-grained
access control of sub role.

I hope I have been able to explain myself clearly. Looking forward to your
feedback.

Best regards,
Jing



On Wed, Jul 5, 2023 at 12:47 PM Dong Lin  wrote:

> Hi Jing,
>
> Thanks for the comments! Please find below my comments, which are based on
> the offline discussion with Xuannan.
>
> On Wed, Jul 5, 2023 at 1:36 AM Jing Ge  wrote:
>
>> Hi Xuannan, Hi Dong
>>
>> Thanks for the Proposal! After reading the FLIP, I'd like to ask some
>> questions:
>>
>> 1. Naming convention for boolean variables. It is recommended to follow
>> JavaBean [1], i.e. objectReuseCompliant as the variable name with
>> isObjectReuseCompliant() and setObjectReuseCompliant() as the methods' name.
>>
>>
> Good point. We have updated the FLIP as suggested.
>
>
>>
>> 2.
>>
>>-

[jira] [Created] (FLINK-32546) update Code Style Guide with Java properties naming convention

[Jing Ge (Jira)]

Jing Ge created FLINK-32546:
---

 Summary: update Code Style Guide with Java properties naming 
convention
 Key: FLINK-32546
 URL: https://issues.apache.org/jira/browse/FLINK-32546
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Jing Ge


The class [properties|https://en.wikipedia.org/wiki/Property_(programming)] 
must be accessible using {_}get{_}, {_}set{_}, _is_ (can be used for boolean 
properties instead of get), _to_ and other methods (so-called [accessor 
methods|https://en.wikipedia.org/wiki/Accessor] and [mutator 
methods|https://en.wikipedia.org/wiki/Mutator_method]) according to a standard 
[naming 
convention|https://en.wikipedia.org/wiki/Naming_conventions_(programming)]. 

 

[https://en.wikipedia.org/wiki/JavaBeans]

[https://flink.apache.org/how-to-contribute/code-style-and-quality-preamble/]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISUCSS] Deprecate multiple APIs in 1.18

[Jing Ge]

Hi Alex,


> > 3. remove SinkFunction.
> Which steps do you imply for the 1.18 release and for the 2.0 release?
>

for 2.0 release. 1.18 will be released soon.

Best regards,
Jing


On Wed, Jul 5, 2023 at 1:08 PM Alexander Fedulov <
alexander.fedu...@gmail.com> wrote:

> @Jing
> Just to clarify, when you say:
>
> 3. remove SinkFunction.
> Which steps do you imply for the 1.18 release and for the 2.0 release?

@Xintong
> A side note - with the new Source API we lose the ability to control
> checkpointing from the source since there is no lock anymore. This
> functionality
> is currently used in a variety of tests for the Sinks - the tests that rely
> on tight
> synchronization between specific elements passed from the source  to the
> sink before
> allowing a checkpoint to complete (see FiniteTestSource [1]). Since FLIP-27
> Sources rely
> on decoupling via the mailbox, without exposing the lock, it is not
> immediately clear
> if it is possible to achieve the same functionality without major
> extensions in the
> runtime for such testing purposes. My hope initially was that only the
> legacy Sinks
> relied on this - this would have made it possible to drop
> SourceFunction+SinkFunction
> together, but, in fact, it also already became part of the new SinkV2
> testing IT suits
> [2]. Moreover, I know of at least one major connector that also relies on
> it for
> verifying committed sink metadata for a specific set of records (Iceberg)
> [3]. In my
> estimation this currently presents a major blocker for the SourceFunction
> removal.
>
> [1]
>
> https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/FiniteTestSource.java
> [2]
>
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/sink/StreamingExecutionFileSinkITCase.java#L132
> [3]
>
> https://github.com/apache/iceberg/blob/master/flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/BoundedTestSource.java#L75C1-L85C2
>
> Best,
> Alex
>
> On Wed, 5 Jul 2023 at 10:47, Chesnay Schepler  wrote:
>
> > There's a whole bunch of metric APIs that would need to be deprecated.
> > That is of course if the metric FLIPs are being accepted.
> >
> > Which makes me wonder if we aren't doing things the wrong way around;
> > shouldn't the decision to deprecate an API be part of the FLIP
> discussion?
> >
> > On 05/07/2023 07:39, Xintong Song wrote:
> > > Thanks all for the discussion.
> > >
> > > It seems to me there's a consensus on marking the following as
> deprecated
> > > in 1.18:
> > > - DataSet API
> > > - SourceFunction
> > > - Queryable State
> > > - All Scala APIs
> > >
> > > More time is needed for deprecating SinkFunction.
> > >
> > > I'll leave this discussion open for a few more days. And if there's no
> > > objections, I'll create JIRA tickets accordingly.
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Wed, Jul 5, 2023 at 1:34 PM Xintong Song 
> > wrote:
> > >
> > >> Thanks for the input, Jing. I'd also be +1 for option 1.
> > >>
> > >> Best,
> > >>
> > >> Xintong
> > >>
> > >>
> > >>
> > >> On Mon, Jul 3, 2023 at 7:20 PM Jing Ge 
> > wrote:
> > >>
> > >>> Hi Xingtong,
> > >>>
> > >>> Option 1, secure plan would be:
> > >>>
> > >>> 1. graduate kafka, File, JDBC connectors to @Public
> > >>> 2. graduate SinkV2 to @Public
> > >>> 3. remove SinkFunction.
> > >>>
> > >>> Option 2, risky plan but at a fast pace:
> > >>>
> > >>> 1. graduate SinkV2 to @Public and expecting more maintenance effort
> > since
> > >>> there are many known and unsolved issues.
> > >>> 2. remove SinkFunction.
> > >>> 3. It depends on the connectors' contributors whether connectors can
> > >>> upgrade to Flink 2.0, since we moved forward with SinkV2 API without
> > >>> taking
> > >>> care of implementations in external connectors.
> > >>>
> > >>> I am ok with both of them and personally prefer option 1.
> > >>>
> > >>> Best regards,
> > >>> Jing
> > >>>
> > >>>
> > >&g

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

[Jing Ge]

Hi Xuannan, Hi Dong

Thanks for the Proposal! After reading the FLIP, I'd like to ask some
questions:

1. Naming convention for boolean variables. It is recommended to follow
JavaBean [1], i.e. objectReuseCompliant as the variable name with
isObjectReuseCompliant() and setObjectReuseCompliant() as the methods' name.


2.

   -

   *If pipeline.object-reuse is set to true, records emitted by this
   operator will be re-used.*
   -

   *Otherwise, if getIsObjectReuseCompliant() returns true, records emitted
   by this operator will be re-used.*
   -

   *Otherwise, records emitted by this operator will be deep-copied before
   being given to the next operator in the chain.*


If I understand you correctly,  the hard coding objectReusedCompliant
should have higher priority over the configuration, the checking logic
should be:

   -

   *If getIsObjectReuseCompliant() returns true, records emitted by this
   operator will be re-used.*
   -

   *Otherwise, if pipeline.object-reuse is set to true, records emitted by
   this operator will be re-used.*
   -

   *Otherwise, records emitted by this operator will be deep-copied before
   being given to the next operator in the chain.*


The results are the same but the checking logics are different, but there
are some additional thoughts, which lead us to the next question.

3. Current design lets specific operators enable object reuse and ignore
the global config. There could be another thought, on the contrary: if an
operator has hard coded the objectReuseCompliant as false, i.e. disable
object reuse on purpose, records should not be reused even if the global
config pipeline.object-reused is set to true, which turns out that the
objectReuseCompliant could be a triple value logic: ture: force object
reusing; false: force deep-copying; unknown: depends on
pipeline.object-reuse config.


Best regards,
Jing


[1] https://en.wikipedia.org/wiki/JavaBeans

On Mon, Jul 3, 2023 at 4:25 AM Xuannan Su  wrote:

> Hi all,
>
> Dong(cc'ed) and I are opening this thread to discuss our proposal to
> add operator attribute to allow operator to specify support for
> object-reuse [1].
>
> Currently, the default configuration for pipeline.object-reuse is set
> to false to avoid data corruption, which can result in suboptimal
> performance. We propose adding APIs that operators can utilize to
> inform the Flink runtime whether it is safe to reuse the emitted
> records. This enhancement would enable Flink to maximize its
> performance using the default configuration.
>
> Please refer to the FLIP document for more details about the proposed
> design and implementation. We welcome any feedback and opinions on
> this proposal.
>
> Best regards,
>
> Dong and Xuannan
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255073749
>

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

[Jing Ge]

Congratulations!

Best regards,
Jing


On Mon, Jul 3, 2023 at 3:21 PM yuxia  wrote:

> Congratulations!
>
> Best regards,
> Yuxia
>
> --
> *发件人: *"Pushpa Ramakrishnan" 
> *收件人: *"Xintong Song" 
> *抄送: *"dev" , "User" 
> *发送时间: *星期一, 2023年 7 月 03日 下午 8:36:30
> *主题: *Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award
>
> Congratulations \uD83E\uDD73
>
> On 03-Jul-2023, at 3:30 PM, Xintong Song  wrote:
>
> 
> Dear Community,
>
> I'm pleased to share this good news with everyone. As some of you may have
> already heard, Apache Flink has won the 2023 SIGMOD Systems Award [1].
>
> "Apache Flink greatly expanded the use of stream data-processing." --
> SIGMOD Awards Committee
>
> SIGMOD is one of the most influential data management research conferences
> in the world. The Systems Award is awarded to an individual or set of
> individuals to recognize the development of a software or hardware system
> whose technical contributions have had significant impact on the theory or
> practice of large-scale data management systems. Winning of the award
> indicates the high recognition of Flink's technological advancement and
> industry influence from academia.
>
> As an open-source project, Flink wouldn't have come this far without the
> wide, active and supportive community behind it. Kudos to all of us who
> helped make this happen, including the over 1,400 contributors and many
> others who contributed in ways beyond code.
>
> Best,
>
> Xintong (on behalf of the Flink PMC)
>
>
> [1] https://sigmod.org/2023-sigmod-systems-award/
>
>
>

Re: [DISCUSS] Graduate the FileSink to @PublicEvolving

[Jing Ge]

Hi,

If there are no other concerns, I will start voting. Thanks!

Best Regards,
Jing

On Mon, Jun 26, 2023 at 11:35 AM Jing Ge  wrote:

> Hi,
>
> @Galen @Yuxia
>
> Your points are valid. Speaking of removing deprecated API, I have the
> same concern. As a matter of fact, I have been raising it in the discussion
> thread of API deprecation process[1]. This is another example that we
> should care about more factors than the migration period, thanks for
> the hint! I will add one more update into that thread with the reference of
> this thread.
>
> In a nutshell, this thread is focusing on the graduation process. Your
> valid concerns should be taken care of by the deprecation process.
> Please don't hesitate to share your thoughts in that thread.
>
>
> Best regards,
> Jing
>
> [1] https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9
>
>
> On Sun, Jun 25, 2023 at 3:48 AM yuxia  wrote:
>
>> Thanks Jing for briging this to dicuss.
>> I agree it's not a blocker for graduting the FileSink to @PublicEvolving
>> since the Sink which is the rootcause has marked as @PublicEvolving.
>> But I do also share the same concern with Galen. At least it should be a
>> blocker for removing StreamingFileSink.
>> Btw, seems it's really a big headache for migrating to Sink, we may need
>> to pay more attention to this ticket and try to fix it.
>>
>> Best regards,
>> Yuxia
>>
>> - 原始邮件 -
>> 发件人: "Galen Warren" 
>> 收件人: "dev" 
>> 发送时间: 星期五, 2023年 6 月 23日 下午 7:47:24
>> 主题: Re: [DISCUSS] Graduate the FileSink to @PublicEvolving
>>
>> Thanks Jing. I can only offer my perspective on this, others may view it
>> differently.
>>
>> If FileSink is subject to data loss in the "stop-on-savepoint then
>> restart"
>> scenario, that makes it unusable for me, and presumably for anyone who
>> uses
>> it in a long-running streaming application and who cannot tolerate data
>> loss. I still use the (deprecated!) StreamingFileSink for this reason.
>>
>> The bigger picture here is that StreamingFileSink is deprecated and will
>> presumably ultimately be removed, to be replaced with FileSink. Graduating
>> the status of FileSink seems to be a step along that path; I'm concerned
>> about continuing down that path with such a critical issue present.
>> Ultimately, my concern is that FileSink will graduate fully and that
>> StreamingFileSink will be removed and that there will be no remaining
>> option to reliably stop/start streaming jobs that write to files without
>> incurring the risk of data loss.
>>
>> I'm sure I'd feel better about things if there were an ongoing effort to
>> address this FileSink issue and/or a commitment that StreamingFileSink
>> would not be removed until this issue is addressed.
>>
>> My two cents -- thanks.
>>
>>
>> On Fri, Jun 23, 2023 at 1:47 AM Jing Ge 
>> wrote:
>>
>> > Hi Galen,
>> >
>> > Thanks for the hint which is helpful for us to have a clear big picture.
>> > Afaiac, this will not be a blocking issue for the graduation. There will
>> > always be some (potential) bugs in the implementation. The API is very
>> > stable from 2020. The timing is good to graduate. WDYT?
>> > Furthermore, I'd like to have more opinions. All opinions together will
>> > help the community build a mature API graduation process.
>> >
>> > Best regards,
>> > Jing
>> >
>> > On Tue, Jun 20, 2023 at 12:48 PM Galen Warren
>> >  wrote:
>> >
>> > > Is this issue still unresolved?
>> > >
>> > >
>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/FLINK-30238
>> > >
>> > > Based on prior discussion, I believe this could lead to data loss with
>> > > FileSink.
>> > >
>> > >
>> > >
>> > > On Tue, Jun 20, 2023, 5:41 AM Jing Ge 
>> > wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > The FileSink has been marked as @Experimental[1] since Oct. 2020.
>> > > > According to FLIP-197[2], I would like to propose to graduate it
>> > > > to @PublicEvloving in the upcoming 1.18 release.
>> > > >
>> > > > On the other hand, as a related topic, FileSource was marked
>> > > > as @PublicEvolving[3] 3 years ago. It deserves a graduation
>> discussion
>> > > too.
>> > > >

Re: [DISUCSS] Deprecate multiple APIs in 1.18

[Jing Ge]

Hi Xingtong,

Option 1, secure plan would be:

1. graduate kafka, File, JDBC connectors to @Public
2. graduate SinkV2 to @Public
3. remove SinkFunction.

Option 2, risky plan but at a fast pace:

1. graduate SinkV2 to @Public and expecting more maintenance effort since
there are many known and unsolved issues.
2. remove SinkFunction.
3. It depends on the connectors' contributors whether connectors can
upgrade to Flink 2.0, since we moved forward with SinkV2 API without taking
care of implementations in external connectors.

I am ok with both of them and personally prefer option 1.

Best regards,
Jing


On Fri, Jun 30, 2023 at 3:41 AM Xintong Song  wrote:

> I see. Thanks for the explanation. I may have not looked into this deeply
> enough, and would trust the decision from you and the community members who
> participated in the discussion & vote.
>
> Best,
>
> Xintong
>
>
>
> On Thu, Jun 29, 2023 at 10:28 PM Alexander Fedulov <
> alexander.fedu...@gmail.com> wrote:
>
> > > However, I'm not sure about 2.
> >
> > I am not aware of a bylaw that states the specific requirements in order
> to
> > mark something as @Deprecated. My understanding from the discussion and
> the
> > vote was that the community recognizes the necessity to make it explicit
> > that
> > the usage of the SourceFunction API is discouraged. This can actually
> > stimulate
> > authors of connectors that rely on this very specific and non-baseline
> > functionality to contribute extensions to the new Source API themselves
> in
> > order to
> > close the gap. ExternallyInducedSource, for instance, was driven by
> Pravega
> > to
> > begin with, since it was only needed for their purposes [1]. We are not
> > removing
> > anything - until 2.0 everything will continue to work and we can work on
> > resolving the limitations until then, I personally don't see a big issue
> > here.
> >
> > >Do you think it is feasible to resolve them by the feature freeze date
> of
> > 1.18?
> > No, these are rather complex additions that would probably require
> FLIP(s).
> >
> > [1]
> >
> >
> https://flink.apache.org/2022/01/20/pravega-flink-connector-101/#checkpoint-integration
> >
> > On Thu, 29 Jun 2023 at 14:25, Xintong Song 
> wrote:
> >
> > > Thanks for the explanation, Alex.
> > >
> > > Not blocking the deprecation on 1 & 3 makes sense to me. However, I'm
> not
> > > sure about 2.
> > >
> > > It sounds to me that, without FLINK-28051 & FLINK-28054, some of the
> > > connectors cannot migrate to the new Source API, or at least further
> > > investigation is needed to understand the situation. If this is the
> case,
> > > we probably should not deprecate the API until these issues are
> resolved.
> > > Do you think it is feasible to resolve them by the feature freeze date
> of
> > > 1.18?
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Thu, Jun 29, 2023 at 8:02 PM Alexander Fedulov <
> > > alexander.fedu...@gmail.com> wrote:
> > >
> > > > @Xintong
> > > > The original discussion [1] and vote [2] converged on the idea that
> it
> > is
> > > > better
> > > > to make it clear to the users that they should stop using
> > SourceFunction
> > > > since it
> > > > is going away. The longer we do not have this indication, the more
> user
> > > > implementations will be based on it and the more pain will be induced
> > > when
> > > > we
> > > > finally drop it. Users now have an alternative API that they should
> use
> > > and
> > > > which
> > > > is fully functional, from that perspective nothing blocks marking it
> > > > @Deprecated.
> > > > As for the remaining work items - there are primarily three kinds:
> > > >
> > > > 1. Where Flink internally uses SourceFunction, without exposing this
> > fact
> > > > to the
> > > >outside world:
> > > >- FLINK-28050 [3]
> > > >- FLINK-28229 [4]
> > > >- FLINK-28048 [5]
> > > >
> > > > 2. Very specific edge cases that might not be covered by the Source
> API
> > > as
> > > > is:
> > > >- FLINK-28054 [6]
> > > >- FLINK-28051 [7]
> > > >
> > > > 3. Usability improvements - something that was easily doable with
> > > > SourceFunction,
> > > >but requires deep knowledge of the new, significantly more
> complex,
> > > > Source API
> > > >to achieve:
> > > >- FLINK-28056 [8]
> > > >
> > > > In my mind, none of those are blockers for proceeding with adding the
> > > > @Deprecated
> > > > annotation:
> > > > (1) is a simple case of encapsulation, internals should not concern
> the
> > > API
> > > > users
> > > > (2) is really only relevant for "exotic" use cases. Does not mean we
> > > should
> > > > not
> > > > consider those, but since it is irrelevant for 99.9% of the users, I
> do
> > > not
> > > > think
> > > > we should get stuck here.
> > > > (3) is purely a nice to have. Formally speaking, all of the tools are
> > > > there, it is
> > > > just that due to the complexity of the new Source API some "simple"
> > > things
> > > > become
> > > > non-trivial and ideally we want to

Re: [DISCUSS] Release 2.0 Work Items

[Jing Ge]

Hi Sergey,

Thanks for the clarification! I will not hijack this thread to discuss
Scala code strategy.

Best regards,
Jing

On Mon, Jul 3, 2023 at 10:51 AM Sergey Nuyanzin  wrote:

> Hi Jing,
>
> Maybe I was not clear enough, sorry.
> However the main reason for this item about Calcite rules is not abandoning
> Scala.
> The main reason are changes in Calcite itself where there was introduced
> code generator framework (immutables)
> to generate config java classes for rules and old api (which is used in
> Scala Calcirte rules) for that is marked as deprecated.
> Since Immutables implies code generation while java compilation it seems
> impossible to use for rules in Scala code.
>
> On Mon, Jul 3, 2023 at 10:44 AM Jing Ge 
> wrote:
>
> > Hi,
> >
> > Speaking of "Move Calcite rules from Scala to Java", I was wondering if
> > this thread is the right place to talk about it. Afaik, the Flink
> community
> > has decided to abandon Scala. That is the reason, I guess, we want to
> move
> > those Calcite rules from Scala to Java. On the other side, new Scala code
> > will be added while developing new features[1]. Do we have any thoughts
> > wrt the Scala code strategy?
> >
> > Best regards,
> > Jing
> >
> >
> >
> > [1] https://lists.apache.org/thread/tnygl4n3q1fx75cl2vclc78j8mrxmz6y
> >
> > On Mon, Jul 3, 2023 at 10:31 AM Xintong Song 
> > wrote:
> >
> > > Thanks all for the discussion.
> > >
> > >
> > > IIUC, we need to make the following changes. Please correct me if I get
> > it
> > > wrong.
> > >
> > >
> > > 1. Disaggregated State Management - Clarify that only the public API
> > > related part is must-have for 2.0.
> > >
> > > 2. Java version support - Split it into 3 items: a) make java 17 the
> > > default (must-have), b) drop java 8 (must-have), and c) drop java 11
> > > (nice-to-have)
> > >
> > > 3. Add MetricGroup#getLogicalScope - Should be promoted to must-have
> > >
> > > 4. ProcessFunction API - Should be downgrade to nice-to-have
> > >
> > > 5. Configuration - Add an item "revisit all config option types and
> > default
> > > values", which IIUC should also be a must-have
> > >
> > >
> > > There seems to be no changes needed for "Move Calcite rules from Scala
> to
> > > Java" as it's already nice-to-have.
> > >
> > >
> > > If there's no objections, I'll update the wiki page accordingly, and
> > start
> > > a VOTE in the next couple of days.
> > >
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Fri, Jun 30, 2023 at 12:53 AM Teoh, Hong
>  > >
> > > wrote:
> > >
> > > > Thanks Xintong for driving the effort.
> > > >
> > > > I’d add a +1 to reworking configs, as suggested by @Jark and
> @Chesnay,
> > > > especially the types. We have various configs that encode Time /
> > > MemorySize
> > > > that are Long instead!
> > > >
> > > > Regards,
> > > > Hong
> > > >
> > > >
> > > >
> > > > > On 29 Jun 2023, at 16:19, Yuan Mei  wrote:
> > > > >
> > > > > CAUTION: This email originated from outside of the organization. Do
> > not
> > > > click links or open attachments unless you can confirm the sender and
> > > know
> > > > the content is safe.
> > > > >
> > > > >
> > > > >
> > > > > Thanks for driving this effort, Xintong!
> > > > >
> > > > > To Chesnay
> > > > >> I'm curious as to why the "Disaggregated State Management" item is
> > > > >> marked as a must-have; will it require changes that break
> something?
> > > > >> What prevents it from being added in 2.1?
> > > > >
> > > > > As to "Disaggregated State Management".
> > > > >
> > > > > We plan to provide a new type of state backend to support DFS as
> > > primary
> > > > > storage.
> > > > > To achieve this, we at least need to include two parts of amends
> (not
> > > > > entirely sure yet, since we are still in the designing and
> prototype
> > > > phase)
> > > > >
> > > > > 1. Statebackend

Re: [VOTE] FLIP-321: introduce an API deprecation process

[Jing Ge]

+1(binding)

On Mon, Jul 3, 2023 at 10:19 AM Stefan Richter
 wrote:

> +1 (binding)
>
>
> > On 3. Jul 2023, at 10:08, Martijn Visser 
> wrote:
> >
> > +1 (binding)
> >
> >
> >
> > On Mon, Jul 3, 2023 at 10:03 AM Xintong Song  > wrote:
> >
> >> +1 (binding)
> >>
> >> Best,
> >>
> >> Xintong
> >>
> >>
> >>
> >> On Sat, Jul 1, 2023 at 11:26 PM Dong Lin  wrote:
> >>
> >>> Thanks for the FLIP.
> >>>
> >>> +1 (binding)
> >>>
> >>> On Fri, Jun 30, 2023 at 5:39 PM Becket Qin 
> wrote:
> >>>
>  Hi folks,
> 
>  I'd like to start the VOTE for FLIP-321[1] which proposes to introduce
> >> an
>  API deprecation process to Flink. The discussion thread for the FLIP
> >> can
> >>> be
>  found here[2].
> 
>  The vote will be open until at least July 4, following the consensus
> >>> voting
>  process.
> 
>  Thanks,
> 
>  Jiangjie (Becket) Qin
> 
>  [1]
> 
> 
> >>>
> >>
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%253A%2BIntroduce%2Ban%2BAPI%2Bdeprecation%2Bprocess&source=gmail-imap&ust=168897655400&usg=AOvVaw24XYJrIcv_vYj1fJVQ7TNY
>  [2]
> https://www.google.com/url?q=https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9&source=gmail-imap&ust=168897655400&usg=AOvVaw1yaMLBBkFfvbBhvyAbHYfX
>
>

Re: [DISCUSS] Release 2.0 Work Items

[Jing Ge]

Hi,

Speaking of "Move Calcite rules from Scala to Java", I was wondering if
this thread is the right place to talk about it. Afaik, the Flink community
has decided to abandon Scala. That is the reason, I guess, we want to move
those Calcite rules from Scala to Java. On the other side, new Scala code
will be added while developing new features[1]. Do we have any thoughts
wrt the Scala code strategy?

Best regards,
Jing



[1] https://lists.apache.org/thread/tnygl4n3q1fx75cl2vclc78j8mrxmz6y

On Mon, Jul 3, 2023 at 10:31 AM Xintong Song  wrote:

> Thanks all for the discussion.
>
>
> IIUC, we need to make the following changes. Please correct me if I get it
> wrong.
>
>
> 1. Disaggregated State Management - Clarify that only the public API
> related part is must-have for 2.0.
>
> 2. Java version support - Split it into 3 items: a) make java 17 the
> default (must-have), b) drop java 8 (must-have), and c) drop java 11
> (nice-to-have)
>
> 3. Add MetricGroup#getLogicalScope - Should be promoted to must-have
>
> 4. ProcessFunction API - Should be downgrade to nice-to-have
>
> 5. Configuration - Add an item "revisit all config option types and default
> values", which IIUC should also be a must-have
>
>
> There seems to be no changes needed for "Move Calcite rules from Scala to
> Java" as it's already nice-to-have.
>
>
> If there's no objections, I'll update the wiki page accordingly, and start
> a VOTE in the next couple of days.
>
>
> Best,
>
> Xintong
>
>
>
> On Fri, Jun 30, 2023 at 12:53 AM Teoh, Hong 
> wrote:
>
> > Thanks Xintong for driving the effort.
> >
> > I’d add a +1 to reworking configs, as suggested by @Jark and @Chesnay,
> > especially the types. We have various configs that encode Time /
> MemorySize
> > that are Long instead!
> >
> > Regards,
> > Hong
> >
> >
> >
> > > On 29 Jun 2023, at 16:19, Yuan Mei  wrote:
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> > >
> > >
> > >
> > > Thanks for driving this effort, Xintong!
> > >
> > > To Chesnay
> > >> I'm curious as to why the "Disaggregated State Management" item is
> > >> marked as a must-have; will it require changes that break something?
> > >> What prevents it from being added in 2.1?
> > >
> > > As to "Disaggregated State Management".
> > >
> > > We plan to provide a new type of state backend to support DFS as
> primary
> > > storage.
> > > To achieve this, we at least need to include two parts of amends (not
> > > entirely sure yet, since we are still in the designing and prototype
> > phase)
> > >
> > > 1. Statebackend Change
> > > 2. State Access Change
> > >
> > > Not all of the interfaces related are `@Internal`. Some of the
> interfaces
> > > like `StateBackend` is `@PublicEvolving`
> > > So, you are right in the sense that "Disaggregated State Management"
> > itself
> > > probably does not need to be a "Must Have"
> > >
> > > But I was hoping changes that related to public APIs can be finalized
> and
> > > merged in Flink 2.0 (I will fix the wiki accordingly).
> > >
> > > I also agree with Jark that 2.0 is a good chance to rework the default
> > > value of configurations.
> > >
> > > Best
> > > Yuan
> > >
> > >
> > > On Thu, Jun 29, 2023 at 8:43 PM Chesnay Schepler 
> > wrote:
> > >
> > >> Something else configuration-related is that there are a bunch of
> > >> options where the type isn't quite correct (e.g., a String where it
> > >> could be an enum, a string where it should be an int or something).
> > >> Could do a pass over those as well.
> > >>
> > >> On 29/06/2023 13:50, Jark Wu wrote:
> > >>> Hi,
> > >>>
> > >>> I think one more thing we need to consider to do in 2.0 is changing
> the
> > >>> default value of configuration to improve out-of-box user experience.
> > >>>
> > >>> Currently, in order to run a Flink job, users may need to set
> > >>> a bunch of configurations, such as minibatch, checkpoint interval,
> > >>> exactly-once,
> > >>> incremental-checkpoint, etc. It's very verbose and hard to use for
> > >>> beginners.
> > >>> Most of them can have a universally applicable value.  Because
> changing
> > >> the
> > >>> default value is a breaking change. I think It's worth considering
> > >> changing
> > >>> them in 2.0.
> > >>>
> > >>> What do you think?
> > >>>
> > >>> Best,
> > >>> Jark
> > >>>
> > >>>
> > >>> On Wed, 28 Jun 2023 at 14:10, Sergey Nuyanzin 
> > >> wrote:
> > >>>
> >  Hi Chesnay
> > 
> > > "Move Calcite rules from Scala to Java": I would hope that this
> would
> > >> be
> > > an entirely internal change, and could thus be an incremental
> process
> > > independent of major releases.
> > > What is the actual scale of this item; how much are we actually
> >  re-writing?
> > 
> >  Thanks for asking
> >  yes, you're right, that should be internal change.
> >  Yeah I was also thinking about incremental change (rule by rule 

Re: [VOTE] FLIP-309: Support using larger checkpointing interval when source is processing backlog

[Jing Ge]

+1(binding)

On Thu, Jun 29, 2023 at 7:47 AM Leonard Xu  wrote:

> +1 (binding)
>
> Best,
> Leonard
>
> > On Jun 29, 2023, at 1:25 PM, Jingsong Li  wrote:
> >
> > +1 binding
> >
> > On Thu, Jun 29, 2023 at 11:03 AM Dong Lin  wrote:
> >>
> >> Hi all,
> >>
> >> We would like to start the vote for FLIP-309: Support using larger
> >> checkpointing interval when source is processing backlog [1]. This FLIP
> was
> >> discussed in this thread [2].
> >>
> >> Flink 1.18 release will feature freeze on July 11. We hope to make this
> >> feature available in Flink 1.18.
> >>
> >> The vote will be open until at least July 4th (at least 72 hours),
> following
> >> the consensus voting process.
> >>
> >> Cheers,
> >> Yunfeng and Dong
> >>
> >> [1]
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-309%3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog
> >> [2] https://lists.apache.org/thread/l1l7f30h7zldjp6ow97y70dcthx7tl37
>
>

Re: [DISUCSS] Deprecate multiple APIs in 1.18

[Jing Ge]

Hi Xintong,

2. SourceFunction / SinkFunction
This is one with challenges. There were many discussions in the past.

2.1 SourceFunction
The voting was passed[1], but there are still some ongoing issues[2]. The
SourceFunction is not marked as @deprecated yet[3], afaik.

2.2 SinkFunction
SinkV2 is a complex one and e.g. even the most important implementation
KafkaSink has not graduated to @Public yet[4]. FlieSink is @Experimental
and there were some concerns with data loss issues when I proposed to
graduate it to @PublicEvolving. It deserves a more serious graduation
process. Otherwise, users would not migrate to SinkV2 and all SinkFunction
and related implementations could not be removed. You can find more details
and other issues from Dong's and my replies at [5]

Best regards,
Jing

[1] https://lists.apache.org/thread/hrpsddgz65hjvhjozhg72s0wsmxz145p
[2] https://github.com/apache/flink/pull/20049
[3]
https://github.com/apache/flink/blob/c11fd82e0cc63904b43d2aca3a79d85dfeb57c2f/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java#L98
[4]
https://github.com/apache/flink-connector-kafka/blob/910fc5a3f06ab7d95f79c0fc4b5ce64e2aaa8db2/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/KafkaSink.java#L57
[5] https://lists.apache.org/thread/q62nj89rrz0t5xtggy5n65on95f2rmmx


On Thu, Jun 29, 2023 at 10:45 AM Martijn Visser 
wrote:

> Hi Xintong,
>
> With regards to the deprecation of the Scala APIs, during the PR
> review it was requested to not mark all APIs as deprecated but only
> the entry point [1], to avoid a fully striked through experience in
> the IDE. I think the same idea was applicable on the DataSet API. I
> think it depends on how formal we want to treat this: if really
> formal, then we should deprecate them in 1.18. I think in both cases,
> it's quite well known that they are deprecated. I'm +0 for either way,
> as long as we're all agreeing that they can be removed in 2.0.
>
> With regards to Queryable State and Source/SinkFunction, +1 to mark
> these as deprecated.
>
> Best regards,
>
> Martijn
>
> [1]
> https://github.com/apache/flink/pull/21176#pullrequestreview-1159706808
>
> On Thu, Jun 29, 2023 at 10:23 AM Xintong Song 
> wrote:
> >
> > Hi devs,
> >
> > Looking at the release 2.0 proposals [1], I noticed that many APIs that
> are
> > proposed to be removed in 2.0 are not (fully) deprecated yet. We might
> want
> > to properly mark them as `@Deprecated` in 1.18 if we agree they should be
> > removed in 2.0. Moreover, according to FLIP-321 [2] (not voted yet but
> IMO
> > is close to consensus IMO), a migration period is required after APIs are
> > deprecated and before they can be removed.
> >
> > I might not be familiar with the status of all the APIs below. So I'd
> like
> > to bring them up and see if there's any concern regarding deprecating
> them
> > in 1.18. If there's concern for deprecating API, we can start a separate
> > discussion thread for it. For those with no objections, I'd create JIRA
> > tickets and try to properly deprecate them in 1.18.
> >
> > 1. DataSet API
> > It's described as "legacy", "soft deprecated" in user documentation [3].
> > However, it's not annotated with `@Deprecated` in codes. According to
> > FLIP-131 [4], DataSet API should be deprecated when DataStream API and
> > Table API / SQL meet certain requirements. AFAICS, all the requirements
> > mentioned in the FLIP are already fulfilled. We should annotate it as
> > `@Deprecated` now.
> >
> > 2. SourceFunction / SinkFunction
> > They are described as deprecated in the roadmap[5], and I don't find
> > anything regarding them in user documentation. But they are also not
> > annotated with `@Deprecated` in codes. TBH, I'm not aware of any formal
> > decision to deprecate these. AFAICS, the replacement for SourceFunction
> > (Source) has already been promoted to `@Public`, while the replacement
> for
> > SinkFunction (SinkV2) is still `@PublicEvolving`. I found a discussion[6]
> > regarding promoting SinkV2 to `@Public`, but it's unclear to me what the
> > conclusion is.
> >
> > 3. Queryable State
> > It's described as approaching end-of-life in the roadmap [5], but is
> > neither deprecated in codes nor in user documentation [7]. I also found a
> > discussion [8] about rescuing it from deprecation, and it seems to me
> there
> > are more negative opinions than positive ones.
> >
> > 4. All Scala APIs
> > I think we agreed to drop Scala API support in FLIP-265 [9], and have
> tried
> > to deprecate them in FLINK-29740 [10]. Also, both user documentation and
> > roadmap[5] shows that scala API supports are deprecated. However, AFAICS,
> > none of the APIs in `flink-streaming-scala` are annotated with
> > `@Deprecated`, and only `ExecutionEnvironment` and `package` are marked
> > `@Deprecated` in `flink-scala`.
> >
> > Looking forward to your feedback.
> >
> > Best,
> >
> > Xintong
> >
> >
> > [1] https://cwiki.apache.org/confluence/display

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

[Jing Ge]

Hi Shammon,

Thanks for your proposal. After reading the FLIP, I'd like to ask
some questions to make sure we are on the same page. Thanks!

1. TableColumnLineageRelation#sinkColumn() should return
TableColumnLineageEntity instead of String, right?

2. Since LineageRelation already contains all information to build the
lineage between sources and sink, do we still need to set the LineageEntity
in the source?

3. About the "Entity" and "Relation" naming, I was confused too, like
Qingsheng mentioned. How about LineageVertex, LineageEdge, and LineageEdges
which contains multiple LineageEdge? E.g. multiple sources join into one
sink, or, edges of columns from one or different tables, etc.

Best regards,
Jing

On Sun, Jun 25, 2023 at 2:06 PM Shammon FY  wrote:

> Hi yuxia and Yun,
>
> Thanks for your input.
>
> For yuxia:
> > 1: What kinds of JobStatus will the `JobExecutionStatusEven` including?
>
> At present, we only need to notify the listener when a job goes to
> termination, but I think it makes sense to add generic `oldStatus` and
> `newStatus` in the listener and users can update the job state in their
> service as needed.
>
> > 2: I'm really confused about the `config()` included in `LineageEntity`,
> where is it from and what is it for ?
>
> The `config` in `LineageEntity` is used for users to get options for source
> and sink connectors. As the examples in the FLIP, users can add
> server/group/topic information in the config for kafka and create lineage
> entities for `DataStream` jobs, then the listeners can get this information
> to identify the same connector in different jobs. Otherwise, the `config`
> in `TableLineageEntity` will be the same as `getOptions` in
> `CatalogBaseTable`.
>
> > 3: Regardless whether `inputChangelogMode` in `TableSinkLineageEntity` is
> needed or not, since `TableSinkLineageEntity` contains
> `inputChangelogMode`, why `TableSourceLineageEntity` don't contain
> changelogmode?
>
> At present, we do not actually use the changelog mode. It can be deleted,
> and I have updated FLIP.
>
> > Btw, since there're a lot interfaces proposed, I think it'll be better to
> give an example about how to implement a listener in this FLIP to make us
> know better about the interfaces.
>
> I have added the example in the FLIP and the related interfaces and
> examples are in branch [1].
>
> For Yun:
> > I have one more question on the lookup-join dim tables, it seems this
> FLIP does not touch them, and will them become part of the
> List sources() or adding another interface?
>
> You're right, currently lookup join dim tables were not considered in the
> 'proposed changed' section of this FLIP. But the interface for lineage is
> universal and we can give `TableLookupSourceLineageEntity` which implements
> `TableSourceLineageEntity` in the future without modifying the public
> interface.
>
> > By the way, if you want to focus on job lineage instead of data column
> lineage in this FLIP, why we must introduce so many column-lineage related
> interface here?
>
> The lineage information in SQL jobs includes table lineage and column
> lineage. Although SQL jobs currently do not support column lineage, we
> would like to support this in the next step. So we have comprehensively
> considered the table lineage and column lineage interfaces here, and
> defined these two interfaces together clearly
>
>
> [1]
>
> https://github.com/FangYongs/flink/commit/d4bfe57e7a5315b790e79b8acef8b11e82c9187c
>
> Best,
> Shammon FY
>
>
> On Sun, Jun 25, 2023 at 4:17 PM Yun Tang  wrote:
>
> > Hi Shammon,
> >
> > I like the idea in general and it will help to analysis the job lineages
> > no matter FlinkSQL or Flink jar jobs in production environments.
> >
> > For Qingsheng's concern, I'd like the name of JobType more than
> > RuntimeExecutionMode, as the latter one is not easy to understand for
> users.
> >
> > I have one more question on the lookup-join dim tables, it seems this
> FLIP
> > does not touch them, and will them become part of the List
> > sources()​ or adding another interface?
> >
> > By the way, if you want to focus on job lineage instead of data column
> > lineage in this FLIP, why we must introduce so many column-lineage
> related
> > interface here?
> >
> >
> > Best
> > Yun Tang
> > 
> > From: Shammon FY 
> > Sent: Sunday, June 25, 2023 16:13
> > To: dev@flink.apache.org 
> > Subject: Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener
> >
> > Hi Qingsheng,
> >
> > Thanks for your valuable feedback.
> >
> > > 1. Is there any specific use case to expose the batch / streaming info
> to
> > listeners or meta services?
> >
> > I agree with you that Flink is evolving towards batch-streaming
> > unification, but the lifecycle of them is different. If a job processes a
> > bound dataset, it will end after completing the data processing,
> otherwise,
> > it will run for a long time. In our scenario, we will regularly schedule
> > some Flink jobs to process boun

Re: [VOTE] FLIP-303: Support REPLACE TABLE AS SELECT statement

[Jing Ge]

+1(binding)

On Wed, Jun 28, 2023 at 1:51 PM Mang Zhang  wrote:

> +1 (no-binding)
>
>
> --
>
> Best regards,
> Mang Zhang
>
>
>
>
>
> At 2023-06-28 17:48:15, "yuxia"  wrote:
> >Hi everyone,
> >Thanks for all the feedback about FLIP-303: Support REPLACE TABLE AS
> SELECT statement[1]. Based on the discussion [2], we have come to a
> consensus, so I would like to start a vote.
> >The vote will be open for at least 72 hours (until July 3th, 10:00AM GMT)
> unless there is an objection or an insufficient number of votes.
> >
> >
> >[1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-303%3A+Support+REPLACE+TABLE+AS+SELECT+statement
> >[2] https://lists.apache.org/thread/39mwckdsdgck48tzsdfm66hhnxorjtz3
> >
> >
> >Best regards,
> >Yuxia
>

Re: [DISCUSS] FLIP-303: Support REPLACE TABLE AS SELECT statement

[Jing Ge]

Hi Yuxia,

Thanks for the proposal. Many engines like Snowflake, Databricks support
it. +1

"3:Check the atomicity is enabled, it requires both the options
table.rtas-ctas.atomicity-enabled is set to true and the corresponding
table sink implementation SupportsStaging."

Typo? "Option" instead of "options"? It sounds like there are more options
that need to be set.

Best regards,
jing




On Tue, Jun 27, 2023 at 8:37 AM yuxia  wrote:

> Hi, all.
> Thanks for the feedback.
>
> If there are no other questions or concerns for the FLIP[1], I'd like to
> start the vote tomorrow (6.28).
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-303%3A+Support+REPLACE+TABLE+AS+SELECT+statement
>
> Best regards,
> Yuxia
>
>
> 发件人: "zhangmang1" 
> 收件人: "dev" , luoyu...@alumni.sjtu.edu.cn
> 发送时间: 星期二, 2023年 6 月 27日 下午 12:03:35
> 主题: Re:Re: [DISCUSS] FLIP-303: Support REPLACE TABLE AS SELECT statement
>
> Hi yuxia,
>
> +1 for this new feature.
> In particular, the CREATE OR REPLACE TABLE syntax is more usable and
> faster for users.
>
>
>
>
>
> --
> Best regards,
> Mang Zhang
>
>
>
>
> At 2023-06-26 09:46:40, "yuxia"  wrote:
> >Hi, folks.
> >To save the time of reviewers, I would like to summary the main changes
> of this FLIP[1]. The FLIP is just to introduce REPLACE TABLE AS SELECT
> statement which is almost similar to CREATE TABLE AS SELECT statement, and
> a syntax CREATE OR REPLACE TABLE AS to wrap both. This FLIP is try to
> complete such kinds of statement.
> >
> >The changes are as follows:
> >1: Add enum REPLACE_TABLE_AS, CREATE_OR_REPLACE_TABLE_AS in
> StagingPurpose which is proposed in FLIP-305[2].
> >
> >2: Change the configuration from `table.ctas.atomicity-enabled` proposed
> in FLIP-305[2] to `table.rtas-ctas.atomicity-enabled` to make it take
> effect not only for create table as, but for replace table as && create or
> replace table as. The main reason is that these statements are almost same
> which belongs to same statement family and I would not like to introduce a
> new different configuration which actually do the same thing. Also, IIRC,
> in the offline dicussion about FLIP-218[1], it also wants to introduce
> `table.rtas-ctas.atomicity-enabled`, but as FLIP-218 is only to support
> CTAS, it's not suitable to introduce a configuration implying rtas which is
> not supported. So, we change the configuration to
> `table.ctas.atomicity-enabled`. Since CTAS has been supported, I think it's
> reasonable to revist it and introduce `table.rtas-ctas.atomicity-enabled` a
> to unify them in this FLIP for supporting REPLACE TABLE AS statement.
> >
> >
> >Again, look forward to your feedback.
> >
> >[1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-303%3A+Support+REPLACE+TABLE+AS+SELECT+statement
> >[2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-305%3A+Support+atomic+for+CREATE+TABLE+AS+SELECT%28CTAS%29+statement
> >[3]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=199541185
> >
> >Best regards,
> >Yuxia
> >
> >- 原始邮件 -
> >发件人: "yuxia" 
> >收件人: "dev" 
> >发送时间: 星期四, 2023年 6 月 15日 下午 7:58:27
> >主题: [DISCUSS] FLIP-303: Support REPLACE TABLE AS SELECT statement
> >
> >Hi, devs.
> >As the FLIPs FLIP-218[1] & FLIP-305[2] for Flink to supports CREATE TABLE
> AS SELECT statement has been accepted.
> >I would like to start a discussion about FLIP-303: Support REPLACE TABLE
> AS SELECT+statement[3] to complete such kinds of statements.
> >With REPLACE TABLE AS SELECT statement, users won't need to drop the
> table firstly, and use CREATE TABLE AS SELECT then. Since the statement is
> much similar to CREATE TABLE AS statement, the design is much similar to
> FLIP-218[1] & FLIP-305[2] apart from some parts specified to REPLACE TABLE
> AS SELECT statement.
> >Just kindly remind, to understand this FLIP better, you may need read
> FLIP-218[1] & FLIP-305[2] to get more context.
> >
> >Look forward to your feedback.
> >
> >[1]:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=199541185
> >[2]:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-305%3A+Support+atomic+for+CREATE+TABLE+AS+SELECT%28CTAS%29+statement
> >[3]:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-303%3A+Support+REPLACE+TABLE+AS+SELECT+statement
> >
> >:) just notice I miss "[DISCUSS]" in the title of the previous email [4],
> so I send it again here with the correct email title. Please ignore the
> previous email and discuss in this thread.
> >Sorry for the noise.
> >
> >[4]: https://lists.apache.org/thread/jy39xwxn1o2035y5411xynwtbyfgg76t
> >
> >
> >Best regards,
> >Yuxia
>
>

Re: [DISCUSS] FLIP-316: Introduce SQL Driver

[Jing Ge]

Hi Paul,

Thanks for driving it and thank you all for the informative discussion! The
FLIP is in good shape now. As described in the FLIP, SQL Driver will be
mainly used to run Flink SQLs in two scenarios: 1. SQL client/gateway in
application mode and 2. external system integration. Would you like to add
one section to describe(better with script/code example) how to use it in
these two scenarios from users' perspective?

NIT: the pictures have transparent background when readers click on it. It
would be great if you can replace them with pictures with white background.

Best regards,
Jing

On Mon, Jun 26, 2023 at 1:31 PM Paul Lam  wrote:

> Hi Shengkai,
>
> > * How can we ship the json plan to the JobManager?
>
> The Flink K8s module should be responsible for file distribution. We could
> introduce
> an option like `kubernetes.storage.dir`. For each flink cluster, there
> would be a
> dedicated subdirectory, with the pattern like
> `${kubernetes.storage.dir}/${cluster-id}`.
>
> All resources-related options (e.g. pipeline jars, json plans) that are
> configured with
> scheme `file://`  would be uploaded to the resource directory
> and downloaded to the
> jobmanager, before SQL Driver accesses the files with the original
> filenames.
>
>
> > * Classloading strategy
>
>
> We could directly specify the SQL Gateway jar as the jar file in
> PackagedProgram.
> It would be treated like a normal user jar and the SQL Driver is loaded
> into the user
> classloader. WDYT?
>
> > * Option `$internal.sql-gateway.driver.sql-config` is string type
> > I think it's better to use Map type here
>
> By Map type configuration, do you mean a nested map that contains all
> configurations?
>
> I hope I've explained myself well, it’s a file that contains the extra SQL
> configurations, which would be shipped to the jobmanager.
>
> > * PoC branch
>
> Sure. I’ll let you know once I get the job done.
>
> Best,
> Paul Lam
>
> > 2023年6月26日 14:27，Shengkai Fang  写道：
> >
> > Hi, Paul.
> >
> > Thanks for your update. I have a few questions about the new design:
> >
> > * How can we ship the json plan to the JobManager?
> >
> > The current design only exposes an option about the URL of the json
> plan. It seems the gateway is responsible to upload to an external stroage.
> Can we reuse the PipelineOptions.JARS to ship to the remote filesystem?
> >
> > * Classloading strategy
> >
> > Currently, the Driver is in the sql-gateway package. It means the Driver
> is not in the JM's classpath directly. Because the sql-gateway jar is now
> in the opt directory rather than lib directory. It may need to add the
> external dependencies as Python does[1]. BTW, I think it's better to move
> the Driver into the flink-table-runtime package, which is much easier to
> find(Sorry for the wrong opinion before).
> >
> > * Option `$internal.sql-gateway.driver.sql-config` is string type
> >
> > I think it's better to use Map type here
> >
> > * PoC branch
> >
> > Because this FLIP involves many modules, do you have a PoC branch to
> verify it does work?
> >
> > Best,
> > Shengkai
> >
> > [1]
> https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940
> <
> https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940
> >
> > Paul Lam mailto:paullin3...@gmail.com>>
> 于2023年6月19日周一 14:09写道：
> > Hi Shengkai,
> >
> > Sorry for my late reply. It took me some time to update the FLIP.
> >
> > In the latest FLIP design, SQL Driver is placed in flink-sql-gateway
> module. PTAL.
> >
> > The FLIP does not cover details about the K8s file distribution, but its
> general usage would
> > be very much the same as YARN setups. We could make follow-up
> discussions in the jira
> > tickets.
> >
> > Best,
> > Paul Lam
> >
> >> 2023年6月12日 15:29，Shengkai Fang  fskm...@gmail.com>> 写道：
> >>
> >>
> >> > If it’s the case, I’m good with introducing a new module and making
> SQL Driver
> >> > an internal class and accepts JSON plans only.
> >>
> >> I rethink this again and again. I think it's better to move the
> SqlDriver into the sql-gateway module because the sql client relies on the
> sql-gateway to submit the sql and the sql-gateway has the ability to
> generate the ExecNodeGraph now. +1 to support accepting JSON plans only.
> >>
> >> * Upload configuration through command line parameter
> >>
> >> ExecNodeGraph only contains the job's information but it doesn't
> contain the checkpoint dir, checkpoint interval, execution mode and so on.
> So I think we should also upload the configuration.
> >>
> >> * KubernetesClusterDescripter and
> KubernetesApplicationClusterEntrypoint are responsible for the jar
> upload/download
> >>
> >> +1 for the change.
> >>
> >> Could you update the FLIP about the current discussion?
> >>
> >> Best,
> >> Shengkai
> >>
> >>
> >>
> >>
> >>
> >>
> >> Yang Wang mailto:wangyang0...@apache.org>>
> 于2023年6月12日周一 11:41写道：
> >> Sorry for the

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

[Jing Ge]

t; > > is reached.
> > >
> > > Sorry, I didn't read the previous detailed discussion because the
> > > discussion list was so long.
> > >
> > > I don't really like either of these options.
> > >
> > > Considering that DataStream is such an important API, can we offer a
> > third
> > > option:
> > >
> > > 3. Maintain the DataStream API throughout 2.X and remove it until 3.x.
> > But
> > > there's no need to assume that 2.X is a short version, it's still a
> > normal
> > > major version.
> > >
> > > Best,
> > > Jingsong
> > >
> > > Becket Qin 于2023年6月22日 周四16:02写道：
> > >
> > > > Thanks much for the input, John, Stefan and Jing.
> > > >
> > > > I think Xingtong has well summarized the pros and cons of the two
> > > options.
> > > > Let's collect a few more opinions here and we can move forward with
> the
> > > one
> > > > more people prefer.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Wed, Jun 21, 2023 at 3:20 AM Jing Ge 
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Thanks Xingtong for the summary. If I could only choose one of the
> > > given
> > > > > two options, I would go with option 1. I understood that option 2
> > > worked
> > > > > great with Kafka. But the bridge release will still confuse users
> and
> > > my
> > > > > gut feeling is that many users will skip 2.0 and be waiting for 3.0
> > or
> > > > even
> > > > > 3.x. And since fewer users will use Flink 2.x, the development
> focus
> > > will
> > > > > be on Flink 3.0 with the fact that the current Flink release is
> 1.17
> > > and
> > > > we
> > > > > are preparing 2.0 release. That is weird for me.
> > > > >
> > > > > THB, I would not name the change from @Public to @Retired as a
> > > demotion.
> > > > > The purpose of @Retire is to extend the API lifecycle with one more
> > > > stage,
> > > > > like in the real world, people born, studied, graduated, worked,
> and
> > > > > retired. Afaiu from the previous discussion, there are two rules
> we'd
> > > > like
> > > > > to follow simultaneously:
> > > > >
> > > > > 1. Public APIs can only be changed between major releases.
> > > > > 2. A smooth migration phase should be offered to users, i.e. at
> > least 2
> > > > > minor releases after APIs are marked as @deprecated. There should
> be
> > > new
> > > > > APIs as the replacement.
> > > > >
> > > > > Agree, those rules are good to improve the user friendliness.
> Issues
> > we
> > > > > discussed are rising because we want to fulfill both of them. If we
> > > take
> > > > > care of deprecation very seriously, APIs can be marked as
> > @Deprecated,
> > > > only
> > > > > when the new APIs as the replacement provide all functionalities
> the
> > > > > deprecated APIs have. In an ideal case without critical bugs that
> > might
> > > > > stop users adopting the new APIs. Otherwise the expected
> > "replacement"
> > > > will
> > > > > not happen. Users will still stick to the deprecated APIs, because
> > the
> > > > new
> > > > > APIs can not be used. For big features, it will need at least 4
> minor
> > > > > releases(ideal case), i.e. 2+ years to remove deprecated APIs:
> > > > >
> > > > > - 1st minor release to build the new APIs as the replacement and
> > > waiting
> > > > > for feedback. It might be difficult to mark the old API as
> deprecated
> > > in
> > > > > this release, because we are not sure if the new APIs could cover
> > 100%
> > > > > functionalities.
> > > > > -  In the lucky case,  mark all old APIs as deprecated in the 2nd
> > minor
> > > > > release. (I would even suggest having the new APIs released at
> least
> > > for
> > > > > two minor releases before marking it as deprecated to make sure
> they
> > > can
> > > > > really repla

Re: [DISCUSS] Graduate the FileSink to @PublicEvolving

[Jing Ge]

Hi,

@Galen @Yuxia

Your points are valid. Speaking of removing deprecated API, I have the same
concern. As a matter of fact, I have been raising it in the discussion
thread of API deprecation process[1]. This is another example that we
should care about more factors than the migration period, thanks for
the hint! I will add one more update into that thread with the reference of
this thread.

In a nutshell, this thread is focusing on the graduation process. Your
valid concerns should be taken care of by the deprecation process.
Please don't hesitate to share your thoughts in that thread.


Best regards,
Jing

[1] https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9


On Sun, Jun 25, 2023 at 3:48 AM yuxia  wrote:

> Thanks Jing for briging this to dicuss.
> I agree it's not a blocker for graduting the FileSink to @PublicEvolving
> since the Sink which is the rootcause has marked as @PublicEvolving.
> But I do also share the same concern with Galen. At least it should be a
> blocker for removing StreamingFileSink.
> Btw, seems it's really a big headache for migrating to Sink, we may need
> to pay more attention to this ticket and try to fix it.
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "Galen Warren" 
> 收件人: "dev" 
> 发送时间: 星期五, 2023年 6 月 23日 下午 7:47:24
> 主题: Re: [DISCUSS] Graduate the FileSink to @PublicEvolving
>
> Thanks Jing. I can only offer my perspective on this, others may view it
> differently.
>
> If FileSink is subject to data loss in the "stop-on-savepoint then restart"
> scenario, that makes it unusable for me, and presumably for anyone who uses
> it in a long-running streaming application and who cannot tolerate data
> loss. I still use the (deprecated!) StreamingFileSink for this reason.
>
> The bigger picture here is that StreamingFileSink is deprecated and will
> presumably ultimately be removed, to be replaced with FileSink. Graduating
> the status of FileSink seems to be a step along that path; I'm concerned
> about continuing down that path with such a critical issue present.
> Ultimately, my concern is that FileSink will graduate fully and that
> StreamingFileSink will be removed and that there will be no remaining
> option to reliably stop/start streaming jobs that write to files without
> incurring the risk of data loss.
>
> I'm sure I'd feel better about things if there were an ongoing effort to
> address this FileSink issue and/or a commitment that StreamingFileSink
> would not be removed until this issue is addressed.
>
> My two cents -- thanks.
>
>
> On Fri, Jun 23, 2023 at 1:47 AM Jing Ge 
> wrote:
>
> > Hi Galen,
> >
> > Thanks for the hint which is helpful for us to have a clear big picture.
> > Afaiac, this will not be a blocking issue for the graduation. There will
> > always be some (potential) bugs in the implementation. The API is very
> > stable from 2020. The timing is good to graduate. WDYT?
> > Furthermore, I'd like to have more opinions. All opinions together will
> > help the community build a mature API graduation process.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Jun 20, 2023 at 12:48 PM Galen Warren
> >  wrote:
> >
> > > Is this issue still unresolved?
> > >
> > >
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/FLINK-30238
> > >
> > > Based on prior discussion, I believe this could lead to data loss with
> > > FileSink.
> > >
> > >
> > >
> > > On Tue, Jun 20, 2023, 5:41 AM Jing Ge 
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > The FileSink has been marked as @Experimental[1] since Oct. 2020.
> > > > According to FLIP-197[2], I would like to propose to graduate it
> > > > to @PublicEvloving in the upcoming 1.18 release.
> > > >
> > > > On the other hand, as a related topic, FileSource was marked
> > > > as @PublicEvolving[3] 3 years ago. It deserves a graduation
> discussion
> > > too.
> > > > To keep this discussion lean and efficient, let's focus on FlieSink
> in
> > > this
> > > > thread. There will be another discussion thread for the FileSource.
> > > >
> > > > I was wondering if anyone might have any concerns. Looking forward to
> > > > hearing from you.
> > > >
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/4006de973525c5284e9bc8fa6196ab7624189261/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java#L129
> > > > [2]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process
> > > > [3]
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/4006de973525c5284e9bc8fa6196ab7624189261/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/FileSource.java#L95
> > > >
> > >
> >
>

Re: [VOTE] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

[Jing Ge]

+1(binding)

Best Regards,
Jing

On Fri, Jun 23, 2023 at 5:50 PM Lijie Wang  wrote:

> Hi all,
>
> Thanks for all the feedback about the FLIP-324: Introduce Runtime Filter
> for Flink Batch Jobs[1]. This FLIP was discussed in [2].
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours (until June 29th 12:00 GMT) unless there is an objection or
> insufficient votes.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-324%3A+Introduce+Runtime+Filter+for+Flink+Batch+Jobs
> [2] https://lists.apache.org/thread/mm0o8fv7x7k13z11htt88zhy7lo8npmg
>
> Best,
> Lijie
>

Re: [DISCUSS] Graduate the FileSink to @PublicEvolving

[Jing Ge]

Hi Galen,

Thanks for the hint which is helpful for us to have a clear big picture.
Afaiac, this will not be a blocking issue for the graduation. There will
always be some (potential) bugs in the implementation. The API is very
stable from 2020. The timing is good to graduate. WDYT?
Furthermore, I'd like to have more opinions. All opinions together will
help the community build a mature API graduation process.

Best regards,
Jing

On Tue, Jun 20, 2023 at 12:48 PM Galen Warren
 wrote:

> Is this issue still unresolved?
>
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/FLINK-30238
>
> Based on prior discussion, I believe this could lead to data loss with
> FileSink.
>
>
>
> On Tue, Jun 20, 2023, 5:41 AM Jing Ge  wrote:
>
> > Hi all,
> >
> > The FileSink has been marked as @Experimental[1] since Oct. 2020.
> > According to FLIP-197[2], I would like to propose to graduate it
> > to @PublicEvloving in the upcoming 1.18 release.
> >
> > On the other hand, as a related topic, FileSource was marked
> > as @PublicEvolving[3] 3 years ago. It deserves a graduation discussion
> too.
> > To keep this discussion lean and efficient, let's focus on FlieSink in
> this
> > thread. There will be another discussion thread for the FileSource.
> >
> > I was wondering if anyone might have any concerns. Looking forward to
> > hearing from you.
> >
> >
> > Best regards,
> > Jing
> >
> >
> >
> >
> >
> >
> > [1]
> >
> >
> https://github.com/apache/flink/blob/4006de973525c5284e9bc8fa6196ab7624189261/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java#L129
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process
> > [3]
> >
> >
> https://github.com/apache/flink/blob/4006de973525c5284e9bc8fa6196ab7624189261/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/FileSource.java#L95
> >
>

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

[Jing Ge]

Hi Ron,

Thanks for sharing your thoughts! It makes sense. It would be helpful if
these references of Hive, Polardb, etc. could be added into the FLIP.

Best regards,
Jing

On Tue, Jun 20, 2023 at 5:41 PM liu ron  wrote:

> Hi, Jing
>
> The default value for this ratio is a reference to other systems, such as
> Hive. As long as Runtime Filter can filter out more than half of the data,
> we can benefit from it. Of course, normally, as long as we can get the
> statistics, ndv are present, the use of rowCount should be less, so I think
> the formula is valid in most cases. This formula we are also borrowed from
> some systems, such as the polardb of AliCloud. your concern is valuable for
> this FLIP, but currently, we do not know how to adjust is reasonably, too
> complex may lead to the user also can not understand, so I think we should
> first follow the simple way, the subsequent gradual optimization. The first
> step may be that we can verify the reasonableness of current formula by
> TPC-DS case.
>
> Best,
> Ron
>
> Jing Ge  于2023年6月20日周二 19:46写道：
>
> > Hi Ron,
> >
> > Thanks for the clarification. That answered my questions.
> >
> > Regarding the ratio, since my gut feeling is that any value less than 0.8
> > or 0.9 won't help too much(I might be wrong). I was thinking of adapting
> > the formula to somehow map the current 0.9-1 to 0-1, i.e. if user config
> > 0.5, it will be mapped to e.g. 0.95 (or e.g. 0.85, the real number
> > needs more calculation) for the current formula described in the FLIP.
> But
> > I am not sure it is a feasible solution. It deserves more discussion.
> Maybe
> > some real performance tests could give us some hints.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Jun 20, 2023 at 5:19 AM liu ron  wrote:
> >
> > > Hi, Jing
> > >
> > > Thanks for your feedback.
> > >
> > > > Afaiu, the runtime Filter will only be Injected when the gap between
> > the
> > > build data size and prob data size is big enough. Let's make an extreme
> > > example. If the small table(build side) has one row and the large
> > > table(probe side) contains tens of billions of rows. This will be the
> > ideal
> > > use case for the runtime filter and the improvement will be
> significant.
> > Is
> > > this correct?
> > >
> > > Yes, you are right.
> > >
> > > > Speaking of the "Conditions of injecting Runtime Filter" in the FLIP,
> > > will
> > > the value of max-build-data-size and min-prob-data-size depend on the
> > > parallelism config? I.e. with the same data-size setting, is it
> possible
> > to
> > > inject or don't inject runtime filters by adjusting the parallelism?
> > >
> > > First, let me clarify two points. The first is that RuntimeFilter
> decides
> > > whether to inject or not in the optimization phase, but we do not
> > consider
> > > operator parallelism in the SQL optimization phase currently, which is
> > set
> > > at the ExecNode level. The second is that in batch mode, the default
> > > AdaptiveBatchScheduler[1] is now used, which will derive the
> parallelism
> > of
> > > the downstream operator based on the amount of data produced by the
> > > upstream operator, that is, the parallelism is determined by runtime
> > > adaptation. In the above case, we cannot decide whether to inject
> > > BloomFilter in the optimization stage based on parallelism.
> > > A more important point is that the purpose of Runtime Filter is to
> reduce
> > > the amount of data for shuffle, and thus the amount of data processed
> by
> > > the downstream join operator. Therefore, I understand that regardless
> of
> > > the parallelism of the probe, the amount of data in the shuffle must be
> > > reduced after inserting the Runtime Filter, which is beneficial to the
> > join
> > > operator, so whether to insert the RuntimeFilter or not is not
> dependent
> > on
> > > the parallelism.
> > >
> > > > Does it make sense to reconsider the formula of ratio
> > > calculation to help users easily control the filter injection?
> > >
> > > Only when ndv does not exist will row count be considered. when size
> uses
> > > the default value and ndv cannot be taken, it is true that this
> condition
> > > may always hold, but this does not seem to affect anything, and the
> user
> > is
> > > also likely to change the value of the size. One question, how do yo

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

[Jing Ge]

> >>>>>> There're various ways to do that, e.g., release notes, warnings in
> >>>> logs,
> >>>>>> etc.
> >>>>>>
> >>>>>> Another possible alternative: whenever there's a deprecated Public
> >> API
> >>>>> that
> >>>>>> reaches a major version bump before the migration period, and we
> >> also
> >>>>> don't
> >>>>>> want to carry it for all the next major release series, we may
> >> consider
> >>>>>> releasing more minor releases for the previous major version after
> >> the
> >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we bump
> >> to
> >>>> 2.0,
> >>>>>> we can release one more 1.20 after 2.0. That should provide users
> >>>> another
> >>>>>> choice rather than upgrading to 2.0, while satisfying the
> >>>> 2-minor-release
> >>>>>> migration period.
> >>>>>>
> >>>>>> I think my major point is, we should not carry APIs deprecated in a
> >>>>>> previous major version along all the next major version series. I'd
> >>>> like
> >>>>> to
> >>>>>> try giving users more commitments, i.e. the migration period, as
> >> long
> >>>> as
> >>>>> it
> >>>>>> does not prevent us from making breaking changes. If it doesn't
> >> work,
> >>>> I'd
> >>>>>> be in favor of not providing the migration period, but fallback to
> >> only
> >>>>>> guarantee the compatibility within the major version.
> >>>>>>
> >>>>>> Best,
> >>>>>>
> >>>>>> Xintong
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler  <mailto:vvcep...@apache.org>
> >>>
> >>>>> wrote:
> >>>>>>
> >>>>>>> Hi Becket,
> >>>>>>>
> >>>>>>> Thanks for the reply! I’d like to continue the conversation about
> >>>>>>> compatibility outside of this FLIP thread, but for now, I can
> >> accept
> >>>>> your
> >>>>>>> decision. It’s certainly an improvement.
> >>>>>>>
> >>>>>>> Thanks again,
> >>>>>>> John
> >>>>>>>
> >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> >>>>>>>> Hi John,
> >>>>>>>>
> >>>>>>>> Completely agree with all you said.
> >>>>>>>>
> >>>>>>>> Can we consider only dropping deprecated APIs in major releases
> >>>>> across
> >>>>>>> the
> >>>>>>>>> board? I understand that Experimental and PublicEvolving APIs
> >> are
> >>>> by
> >>>>>>>>> definition less stable, but it seems like this should be
> >> reflected
> >>>>> in
> >>>>>>> the
> >>>>>>>>> required deprecation period alone. I.e. that we must keep them
> >>>>> around
> >>>>>>> for
> >>>>>>>>> at least zero or one minor release, not that we can drop them
> >> in a
> >>>>>>> minor or
> >>>>>>>>> patch release.
> >>>>>>>>
> >>>>>>>> Personally speaking, I would love to do this, for exactly the
> >>>> reason
> >>>>>> you
> >>>>>>>> mentioned. However, I did not propose this due to the following
> >>>>>> reasons:
> >>>>>>>>
> >>>>>>>> 1. I am hesitating a little bit about changing the accepted
> >> FLIPs
> >>>> too
> >>>>>>> soon.
> >>>>>>>> 2. More importantly, to avoid slowing down our development. At
> >> this
> >>>>>>> point,
> >>>>>>>> Flink still lacks some design / routines to support good API
> >>>>>>> evolvability /

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

[Jing Ge]

Hi Ron,

Thanks for the clarification. That answered my questions.

Regarding the ratio, since my gut feeling is that any value less than 0.8
or 0.9 won't help too much(I might be wrong). I was thinking of adapting
the formula to somehow map the current 0.9-1 to 0-1, i.e. if user config
0.5, it will be mapped to e.g. 0.95 (or e.g. 0.85, the real number
needs more calculation) for the current formula described in the FLIP. But
I am not sure it is a feasible solution. It deserves more discussion. Maybe
some real performance tests could give us some hints.

Best regards,
Jing

On Tue, Jun 20, 2023 at 5:19 AM liu ron  wrote:

> Hi, Jing
>
> Thanks for your feedback.
>
> > Afaiu, the runtime Filter will only be Injected when the gap between the
> build data size and prob data size is big enough. Let's make an extreme
> example. If the small table(build side) has one row and the large
> table(probe side) contains tens of billions of rows. This will be the ideal
> use case for the runtime filter and the improvement will be significant. Is
> this correct?
>
> Yes, you are right.
>
> > Speaking of the "Conditions of injecting Runtime Filter" in the FLIP,
> will
> the value of max-build-data-size and min-prob-data-size depend on the
> parallelism config? I.e. with the same data-size setting, is it possible to
> inject or don't inject runtime filters by adjusting the parallelism?
>
> First, let me clarify two points. The first is that RuntimeFilter decides
> whether to inject or not in the optimization phase, but we do not consider
> operator parallelism in the SQL optimization phase currently, which is set
> at the ExecNode level. The second is that in batch mode, the default
> AdaptiveBatchScheduler[1] is now used, which will derive the parallelism of
> the downstream operator based on the amount of data produced by the
> upstream operator, that is, the parallelism is determined by runtime
> adaptation. In the above case, we cannot decide whether to inject
> BloomFilter in the optimization stage based on parallelism.
> A more important point is that the purpose of Runtime Filter is to reduce
> the amount of data for shuffle, and thus the amount of data processed by
> the downstream join operator. Therefore, I understand that regardless of
> the parallelism of the probe, the amount of data in the shuffle must be
> reduced after inserting the Runtime Filter, which is beneficial to the join
> operator, so whether to insert the RuntimeFilter or not is not dependent on
> the parallelism.
>
> > Does it make sense to reconsider the formula of ratio
> calculation to help users easily control the filter injection?
>
> Only when ndv does not exist will row count be considered. when size uses
> the default value and ndv cannot be taken, it is true that this condition
> may always hold, but this does not seem to affect anything, and the user is
> also likely to change the value of the size. One question, how do you think
> we should make it easier for users to control the  filter injection？
>
>
> [1]:
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#adaptive-batch-scheduler
>
> Best,
> Ron
>
> Jing Ge  于2023年6月20日周二 07:11写道：
>
> > Hi Lijie,
> >
> > Thanks for your proposal. It is a really nice feature. I'd like to ask a
> > few questions to understand your thoughts.
> >
> > Afaiu, the runtime Filter will only be Injected when the gap between the
> > build data size and prob data size is big enough. Let's make an extreme
> > example. If the small table(build side) has one row and the large
> > table(probe side) contains tens of billions of rows. This will be the
> ideal
> > use case for the runtime filter and the improvement will be significant.
> Is
> > this correct?
> >
> > Speaking of the "Conditions of injecting Runtime Filter" in the FLIP,
> will
> > the value of max-build-data-size and min-prob-data-size depend on the
> > parallelism config? I.e. with the same data-size setting, is it possible
> to
> > inject or don't inject runtime filters by adjusting the parallelism?
> >
> > In the FLIP, there are default values for the new configuration
> parameters
> > that will be used to check the injection condition. If ndv cannot be
> > estimated, row count will be used. Given the max-build-data-size is 10MB
> > and the min-prob-data-size is 10GB, in the worst case, the
> min-filter-ratio
> > will be 0.999, i.e. the probeNdv is 1000 times buildNdv . If we consider
> > the duplication and the fact that the large table might have more columns
> > than the small table, the probeNdv should still be 100 or 10 

[DISCUSS] Graduate the FileSink to @PublicEvolving

[Jing Ge]

Hi all,

The FileSink has been marked as @Experimental[1] since Oct. 2020.
According to FLIP-197[2], I would like to propose to graduate it
to @PublicEvloving in the upcoming 1.18 release.

On the other hand, as a related topic, FileSource was marked
as @PublicEvolving[3] 3 years ago. It deserves a graduation discussion too.
To keep this discussion lean and efficient, let's focus on FlieSink in this
thread. There will be another discussion thread for the FileSource.

I was wondering if anyone might have any concerns. Looking forward to
hearing from you.


Best regards,
Jing






[1]
https://github.com/apache/flink/blob/4006de973525c5284e9bc8fa6196ab7624189261/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java#L129
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process
[3]
https://github.com/apache/flink/blob/4006de973525c5284e9bc8fa6196ab7624189261/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/FileSource.java#L95

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

[Jing Ge]

Hi Lijie,

Thanks for your proposal. It is a really nice feature. I'd like to ask a
few questions to understand your thoughts.

Afaiu, the runtime Filter will only be Injected when the gap between the
build data size and prob data size is big enough. Let's make an extreme
example. If the small table(build side) has one row and the large
table(probe side) contains tens of billions of rows. This will be the ideal
use case for the runtime filter and the improvement will be significant. Is
this correct?

Speaking of the "Conditions of injecting Runtime Filter" in the FLIP, will
the value of max-build-data-size and min-prob-data-size depend on the
parallelism config? I.e. with the same data-size setting, is it possible to
inject or don't inject runtime filters by adjusting the parallelism?

In the FLIP, there are default values for the new configuration parameters
that will be used to check the injection condition. If ndv cannot be
estimated, row count will be used. Given the max-build-data-size is 10MB
and the min-prob-data-size is 10GB, in the worst case, the min-filter-ratio
will be 0.999, i.e. the probeNdv is 1000 times buildNdv . If we consider
the duplication and the fact that the large table might have more columns
than the small table, the probeNdv should still be 100 or 10 times
buildNdv, which ends up with a min-filter-ratio equals to 0.99 or 0.9. Both
are bigger than the default value 0.5 in the FLIP. If I am not mistaken,
commonly, a min-filter-ratio less than 0.99 will always allow injecting the
runtime filter. Does it make sense to reconsider the formula of ratio
calculation to help users easily control the filter injection?

Best regards,
Jing

On Mon, Jun 19, 2023 at 4:42 PM Lijie Wang  wrote:

> Hi Stefan,
>
> >> bypassing the dataflow
> I believe it's a possible solution, but it may require more coordination
> and extra conditions (such as DFS), I do think it should be excluded from
> the first version. I'll put it in Future+Improvements as a potential
> improvement.
>
> Thanks again for your quick reply :)
>
> Best,
> Lijie
>
> Stefan Richter  于2023年6月19日周一 20:51写道：
>
> >
> > Hi Lijie,
> >
> > I think you understood me correctly. But I would not consider this a true
> > cyclic dependency in the dataflow because I would not suggest to send the
> > filter through an edge in the job graph from join to scan. I’d rather
> > bypass the stream graph for exchanging bringing the filter to the scan.
> For
> > example, the join could report the filter after the build phase, e.g. to
> > the JM or a predefined DFS folder. And when the probe scan is scheduled,
> > the JM provides the filter information to the scan when it gets scheduled
> > for execution or the scan looks in DFS if it can find any filter that it
> > can use as part of initialization. I’m not suggesting to do it exactly in
> > those ways, but just to show what I mean by "bypassing the dataflow".
> >
> > Anyways, I’m fine with excluding this optimization from the current FLIP
> > if you believe it would be hard to implement in Flink.
> >
> > Best,
> > Stefan
> >
> >
> > > On 19. Jun 2023, at 14:07, Lijie Wang 
> wrote:
> > >
> > > Hi Stefan,
> > >
> > > If I understand correctly(I hope so), the hash join operator needs to
> > send
> > > the bloom filter to probe scan, and probe scan also needs to send the
> > > filtered data to the hash join operator. This means there will be a
> cycle
> > > in the data flow, it will be hard for current Flink to schedule this
> kind
> > > of graph. I admit we can find a way to do this, but that's probably a
> > > bit outside the scope of this FLIP.  So let's do these complex
> > > optimizations later, WDYT？
> > >
> > > Best,
> > > Lijie
> > >
> > > Stefan Richter  > srich...@confluent.io.invalid>> 于2023年6月19日周一 18:15写道：
> > >
> > >> Hi Lijie,
> > >>
> > >> Exactly, my proposal was to build the bloom filter in the hash
> > operator. I
> > >> don’t know about all the details about the implementation of Flink’s
> > join
> > >> operator, but I’d assume that even if the join is a two input operator
> > it
> > >> gets scheduled for 2 different pipelines. First the build phase with
> the
> > >> scan from the dimension table and after that’s completed the probe
> phase
> > >> with the scan of the fact table. I’m not proposing the use the bloom
> > filter
> > >> only in the join operator, but rather send the bloom filter to the
> probe
> > >> scan before starting the probe. I assume this would require some form
> of
> > >> side channel to transport the filter and coordination to tell the
> > sources
> > >> that such a filter is available. I cannot answer how hard those would
> > be to
> > >> implement, but the idea doesn’t seem impossible to me.
> > >>
> > >> Best,
> > >> Stefan
> > >>
> > >>
> > >>> On 19. Jun 2023, at 11:56, Lijie Wang 
> > wrote:
> > >>>
> > >>> Hi Stefan,
> > >>>
> > >>> Now I know what you mean about point 1. But currently it is
> unfeasible
> > >> for
> > >>> Flink, because the building of the 

Re: [DISCUSS] FLIP-311: Support Call Stored Procedure

[Jing Ge]

Thanks for your reply. Nice feature!

Best regards,
Jing

On Wed, Jun 14, 2023 at 3:11 AM yuxia  wrote:

> Yes, you're right.
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -
> 发件人: "Jing Ge" 
> 收件人: "dev" 
> 发送时间: 星期三, 2023年 6 月 14日 上午 4:46:58
> 主题: Re: [DISCUSS] FLIP-311: Support Call Stored Procedure
>
> Hi yuxia,
>
> Thanks for your proposal and sorry for the late reply. The FLIP is in good
> shape. If I am not mistaken, Everything, that a stored procedure could do,
> could also be done by a Flink job. The current stored procedure design is
> to empower Catalogs to provide users some commonly used logics/functions
> centrally and out-of-the-box, i.e. DRY. Is that correct?
>
> Best regards,
> Jing
>
> On Thu, Jun 8, 2023 at 10:32 AM Jark Wu  wrote:
>
> > Thank you for the proposal, yuxia! The FLIP looks good to me.
> >
> > Best,
> > Jark
> >
> > > 2023年6月8日 11:39，yuxia  写道：
> > >
> > > Hi, all.
> > > Thanks everyone for the valuable input. If there are are no further
> > concerns about this FLIP[1], I would like to start voting next monday
> > (6/12).
> > >
> > > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-311%3A+Support+Call+Stored+Procedure
> > >
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > - 原始邮件 -
> > > 发件人: "Martijn Visser" 
> > > 收件人: "dev" 
> > > 发送时间: 星期二, 2023年 6 月 06日 下午 3:57:56
> > > 主题: Re: [DISCUSS] FLIP-311: Support Call Stored Procedure
> > >
> > > Hi Yuxia,
> > >
> > > Thanks for the clarification. I would be +0 overall, because I think
> > > without actually allowing creation/customization of stored procedures,
> > the
> > > value for the majority of Flink users will be minimal.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Tue, Jun 6, 2023 at 3:52 AM yuxia 
> > wrote:
> > >
> > >> Hi, Martijn.
> > >> Thanks for you feedback.
> > >> 1: In this FLIP we don't intend to allow users to customize their own
> > >> stored procedure for we don't want to expose too much to users too
> > early as
> > >> the FLIP said.
> > >> The procedures are supposed to be provided only by Catalog. Catalog
> devs
> > >> can write their build-in procedures, and return the procedure in
> method
> > >> Catalog.getProcedure(ObjectPath procedurePath);
> > >> So, there won't be SQL syntax to create/save a stored procedure in
> this
> > >> FLIP. If we find we do need it, we can propse the SQL syntax to
> create a
> > >> stored procedure in another dedicated FLIP.
> > >>
> > >> 2: The syntax `Call procedure_name(xx)` proposed in this FLIP is the
> > >> default syntax in Calcite for call stored procedures. Actaully, we
> don't
> > >> need to do any modifcation in flink-sql-parser module for syntax of
> > calling
> > >> a procedure. MySQL[1], Postgres[2], Oracle[3] also use the syntax to
> > call a
> > >> stored procedure.
> > >>
> > >>
> > >> [1] https://dev.mysql.com/doc/refman/8.0/en/call.html
> > >> [2] https://www.postgresql.org/docs/15/sql-call.html
> > >> [3]
> https://docs.oracle.com/javadb/10.8.3.0/ref/rrefcallprocedure.html
> > >>
> > >> Best regards,
> > >> Yuxia
> > >>
> > >> - 原始邮件 -
> > >> 发件人: "Martijn Visser" 
> > >> 收件人: "dev" 
> > >> 发送时间: 星期一, 2023年 6 月 05日 下午 8:35:44
> > >> 主题: Re: [DISCUSS] FLIP-311: Support Call Stored Procedure
> > >>
> > >> Hi Yuxia,
> > >>
> > >> Thanks for the FLIP. I have a couple of questions:
> > >>
> > >> 1. The syntax talks about how to CALL or SHOW the available stored
> > >> procedures, but not on how to create one. Will there not be a SQL
> > syntax to
> > >> create/save a stored procedure?
> > >> 2. Is there a default syntax in Calcite for stored procedures? What do
> > >> other databases do, do they use CALL/SHOW or something like EXEC, USE?
> > >>
> > >> Best regards,
> > >>
> > >> Martijn
> > >>
> > >> On Mon, Jun 5, 2023 at 3:23 AM yuxia 
> > wrote:
> > >>
> > >>> Hi, Jane.
> > >>> Thanks for yo

Re: [VOTE] FLIP-308: Support Time Travel

[Jing Ge]

+1(binding)

On Mon, Jun 19, 2023 at 1:57 PM Benchao Li  wrote:

> +1 (binding)
>
> Lincoln Lee  于2023年6月19日周一 19:40写道：
>
> > +1 (binding)
> >
> > Best,
> > Lincoln Lee
> >
> >
> > yuxia  于2023年6月19日周一 19:30写道：
> >
> > > +1 (binding)
> > > Thanks Feng driving it.
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > - 原始邮件 -
> > > 发件人: "Feng Jin" 
> > > 收件人: "dev" 
> > > 发送时间: 星期一, 2023年 6 月 19日 下午 7:22:06
> > > 主题: [VOTE] FLIP-308: Support Time Travel
> > >
> > > Hi everyone
> > >
> > > Thanks for all the feedback about the FLIP-308: Support Time Travel[1].
> > > [2] is the discussion thread.
> > >
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours(excluding weekends,until June 22, 12:00AM GMT) unless there is an
> > > objection or an insufficient number of votes.
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-308%3A+Support+Time+Travel
> > > [2]https://lists.apache.org/thread/rpozdlf7469jmc7q7vc0s08pjnmscz00
> > >
> > >
> > > Best,
> > > Feng
> > >
> >
>
>
> --
>
> Best,
> Benchao Li
>

Re: [NOTICE] Experimental Java 17 support now available on master

[Jing Ge]

Hi Kurt,

Thanks for your contribution. I am a little bit confused about the email
title, since your PR[1] is not merged into the master yet. I guess, with
"Experimental Java 17 support", you meant it is available on your branch
which is based on the master.

If I am not mistaken, there is no vote thread of FLIP 317 on ML. Would you
like to follow the standard process[2] defined by the Flink community?
Thanks!


Best regards,
Jing

[1] https://github.com/apache/flink/pull/22660
[2]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

On Sun, Jun 18, 2023 at 1:18 AM Kurt Ostfeld 
wrote:

> I built the Flink master branch and tried running this simple Flink app
> that uses a Java record:
>
>
> https://github.com/kurtostfeld/flink-kryo-upgrade-demo/blob/main/flink-record-demo/src/main/java/demo/app/Main.java
>
> It fails with the normal exception that Kryo 2.x throws when you try to
> serialize a Java record. The full stack trace is here:
> https://pastebin.com/HGhGKUWt
>
> I tried removing this line:
>
>
> https://github.com/kurtostfeld/flink-kryo-upgrade-demo/blob/main/flink-record-demo/src/main/java/demo/app/Main.java#L36
>
> and that had no impact, I got the same error.
>
> In the other thread, you said that the plan was to use PojoSerializer to
> serialize records rather than Kryo. Currently, the Flink code bases uses
> Kryo 2.x by default for generic user data types, and that will fail when
> the data type is a record or contains records. Ultimately, if Flink wants
> to fully support Java records, it seems that it has to move off of Kryo
> 2.x. PojoSerializer is part of what is basically a custom serialization
> library internal to Flink that is an alternative to Kryo. That's one
> option: move off of Kryo to a Flink-internal serialization library. The
> other two options are upgrade to the new Kryo or use a different
> serialization library.
>
> The Kryo 5.5.0 upgrade PR I submitted (
> https://github.com/apache/flink/pull/22660) with FLIP 317 (
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-317%3A+Upgrade+Kryo+from+2.24.0+to+5.5.0)
> works with records. The Flink app linked above that uses records works with
> the PR and that's what I posted to this mailing list a few weeks ago. I
> rebased the pull request on to the latest master branch and it's passing
> all tests. From my testing, it supports stateful upgrades, including
> checkpoints. If you can demonstrate a scenario where stateful upgrades
> error I can try to resolve that.

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

[Jing Ge]

Hi All,

The @Public -> @PublicEvolving proposed by Xintong is a great idea.
Especially, after he suggest @PublicRetired, i.e. @PublicEvolving --(2
minor release)--> @Public --> @deprecated --(1 major
release)--> @PublicRetired. It will provide a lot of flexibility without
breaking any rules we had. @Public APIs are allowed to change between major
releases. Changing annotations is acceptable and provides additional
tolerance i.e. user-friendliness, since the APIs themself are not changed.

I had similar thoughts when I was facing those issues. I want to move one
step further and suggest introducing one more annotation @Retired.

Not like the @PublicRetired which is a compromise of downgrading @Public to
@PublicEvolving. As I mentioned earlier in my reply, Java standard
@deprecated should be used in the early stage of the deprecation process
and doesn't really meet our requirement. Since Java does not allow us to
extend annotation, I think it would be feasible to have the new @Retired to
help us monitor and manage the deprecation process, house cleaning, etc.

Some ideas could be(open for discussion):

@Retired:

1. There must be a replacement with functionality compatibility before APIs
can be marked as @Retired, i.e. DISCUSS and VOTE processes on the ML are
mandatory (a FLIP is recommended).
2. APIs marked as @Retired will be removed after 1 minor release sharply
(using ArchUnit to force it, needs further check whether it is possible).
Devs who marked them as @Retired are responsible to remove them.
3. Both @Public -> @Retired and @PublicEvolving -> @Retired are
recommended. @Experimental -> @Retired and @Internal -> @Retired could also
be used if it can increase user-friendliness or dev-friendliness, but not
mandatory.
4. Some variables will be defined in @Retired to support the deprecation
process management. Further extension is possible, since the annotation is
built by us.


Best regards,
Jing

On Fri, Jun 16, 2023 at 10:31 AM Becket Qin  wrote:

> Hi Xintong,
>
> Thanks for the explanation. Please see the replies inline below.
>
> I agree. And from my understanding, demoting a Public API is also a kind of
> > such change, just like removing one, which can only happen with major
> > version bumps. I'm not proposing to allow demoting Public APIs anytime,
> but
> > only in the case major version bumps happen before reaching the
> > 2-minor-release migration period. Actually, demoting would be a weaker
> > change compared to removing the API immediately upon major version bumps,
> > in order to keep the commitment about the 2-minor-release migration
> period.
> > If the concern is that `@Public` -> `@PublicEvolving` sounds against
> > conventions, we may introduce a new annotation if necessary, e.g.,
> > `@PublicRetiring`, to avoid confusions.
>
> As an end user who only uses Public APIs, if I don't change my code at all,
> my expectation is the following:
> 1. Upgrading from 1.x to 2.x may have issues.
> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with all
> the 2.x versions.
> Actually I think there are some dependency version resolution policies out
> there which picks the highest minor version when the dependencies pull in
> multiple minor versions of the same jar, which may be broken if we remove
> the API in minor releases.
>
> I'm not sure about this. Yes, it's completely "legal" that we bump up the
> > major version whenever a breaking change is needed. However, this also
> > weakens the value of the commitment that public APIs will stay stable
> > within the major release series, as the series can end anytime. IMHO,
> short
> > major release series are not something "make the end users happy", but
> > backdoors that allow us as the developers to make frequent breaking
> > changes. On the contrary, with the demoting approach, we can still have
> > longer major release series, while only allowing Public APIs deprecated
> at
> > the end of the previous major version to be removed in the next major
> > version.
>
> I totally agree that frequent major version bumps are not ideal, but here
> we are comparing it with a minor version bump which removes a Public API.
> So the context is that we have already decided to remove this Public API
> while keeping everything else backwards compatible. I think a major version
> bump is a commonly understood signal here, compared with a minor version
> change. From end users' perspective, for those who are not impacted, in
> this case upgrading a major version is not necessarily more involved than
> upgrading a minor version - both should be as smooth as a dependency
> version change. For those who are impacted, they will lose the Public API
> anyways and a major version bump ensures there is no surprise.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song 
> wrote:
>
> > Public API is a well defined common concept, and one of its
> >> convention is that it only changes with a major version change.
> >>
> >

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

[Jing Ge]

ule to decide when to remove deprecated APIs without
caring of anything else like functionality compatibility. If the migration
period described in this FLIP is only the minimum time, I think we still
have the house cleaning issue unsolved. Minimum means deprecated APIs can
not be removed before the migration period expires. The issue I was aware
of is when/how to remove APIs after the migration period expired, e.g.
PublicEvolving APIs that have been marked as deprecated years ago.


Best regards,
Jing

[1] https://lists.apache.org/thread/m3o48c2d8j9g5t9s89hqs6qvr924s71o

On Wed, Jun 14, 2023 at 9:56 AM Becket Qin  wrote:

> Hi Jing,
>
> Thanks for the feedback. Please see the answers to your questions below:
>
> *"Always add a "Since X.X.X" comment to indicate when was a class /
> > interface / method marked as deprecated."*
> >  Could you describe it with a code example? Do you mean Java comments?
>
> It is just a comment such as "Since 1.18. Use X
> <
> https://kafka.apache.org/31/javadoc/org/apache/kafka/clients/admin/Admin.html#incrementalAlterConfigs(java.util.Map)
> >XX
> instead.". And we can then look it up in the deprecated list[1] in each
> release and see which method should / can be deprecated.
>
> *"At least 1 patch release for the affected minor release for
> > Experimental APIs"*
> > The rule is absolutely right. However, afaiac, deprecation is different
> as
> > modification. As a user/dev, I would appreciate, if I do not need to do
> any
> > migration work for any deprecated API between patch releases upgrade.
> BTW,
> > if experimental APIs are allowed to change between patches, could we just
> > change them instead of marking them as deprecated and create new ones to
> > replace them?
>
> Deprecating an API is just a more elegant way of replacing an API with a
> new one. The only difference between the two is whether the old API is kept
> and coexists with the new API for some releases or not. For end users,
> deprecation-then-remove is much more friendly than direct replacement.
>
> 1. How to make sure the new APIs cover all functionality, i.e. backward
> > compatible, before removing the deprecated APIs? Since the
> > functionalities could only be built with the new APIs iteratively, there
> > will be a while (might be longer than the migration period) that the new
> > APIs are not backward compatible with the deprecated ones.
>
>

> This is orthogonal to the deprecation process, and may not even be required
> in some cases if the function changes by itself. But in general, this
> relies on the developer to decide. A simple test on readiness is to see if
> all the UT / IT cases written with the old API can be migrated to the new
> one and still work.  If the new API is not ready, we probably should not
> deprecate the old one to begin with.
>


>
> 2. Is it allowed to remove the deprecated APIs after the defined migration
> > period expires while the new APis are still not backward compatible?
>
>

> By "backwards compatible", do you mean functionally equivalent? If the new
> APIs are designed to be not backwards compatible, then removing the
> deprecated source code is definitely allowed. If we don't think the new API
> is ready to take over the place for the old one, then we should wait. The
> migration period is the minimum time we have to wait before removing the
> source code. A longer migration period is OK.
>
>

> 3. For the case of core API upgrade with downstream implementations, e.g.
> > connectors, What is the feasible deprecation strategy? Option1 bottom-up:
> > make sure the downstream implementation is backward compatible before
> > removing the deprecated core APIs. Option2 top-down: once the downstream
> > implementation of new APIs works fine, we can remove the deprecated core
> > APIs after the migration period expires. The implementation of the
> > deprecated APIs will not get any further update in upcoming releases(it
> has
> > been removed). There might be some missing features in the downstream
> > implementation of new APIs compared to the old implementation. Both
> options
> > have their own pros and cons.
>
> The downstream projects such as connectors in Flink should also follow the
> migration path we tell our users. i.e. If there is a cascading backwards
> incompatible change in the connectors due to a backwards incompatible
> change in the core, and as a result a longer migration period is required,
> then I think we should postpone the removal of source code. But in general,
> we should be able to provide the same migration period in the connectors as
> the flink core, if the connectors are upg

Re: [VOTE] FLIP-287: Extend Sink#InitContext to expose TypeSerializer, ObjectReuse and JobID

[Jing Ge]

+1(binding)

Best Regards,
Jing

On Fri, Jun 16, 2023 at 10:10 AM Lijie Wang 
wrote:

> +1 (binding)
>
> Thanks for driving it, Joao.
>
> Best,
> Lijie
>
> Joao Boto  于2023年6月16日周五 15:53写道：
>
> > Hi all,
> >
> > Thank you to everyone for the feedback on FLIP-287[1]. Based on the
> > discussion thread [2], we have come to a consensus on the design and are
> > ready to take a vote to contribute this to Flink.
> >
> > I'd like to start a vote for it. The vote will be open for at least 72
> > hours(excluding weekends, unless there is an objection or an insufficient
> > number of votes.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240880853
> > [2]https://lists.apache.org/thread/wb3myhqsdz81h08ygwx057mkn1hc3s8f
> >
> >
> > Best,
> > Joao Boto
> >
>

Re: [VOTE] FLIP-246: Dynamic Kafka Source (originally Multi Cluster Kafka Source)

[Jing Ge]

+1 (binding)

Best regards,
Jing

On Thu, Jun 15, 2023 at 7:55 PM Mason Chen  wrote:

> Hi all,
>
> Thank you to everyone for the feedback on FLIP-246 [1]. Based on the
> discussion thread [2], we have come to a consensus on the design and are
> ready to take a vote to contribute this to Flink.
>
> This voting thread will be open for at least 72 hours (excluding weekends,
> until June 20th 10:00AM PST) unless there is an objection or an
> insufficient number of votes.
>
> (Optional) If you have an opinion on the naming of the connector, please
> include it in your vote:
> 1. DynamicKafkaSource
> 2. MultiClusterKafkaSource
> 3. DiscoveringKafkaSource
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217389320
> [2] https://lists.apache.org/thread/vz7nw5qzvmxwnpktnofc9p13s1dzqm6z
>
> Best,
> Mason
>

Re: [VOTE] FLIP-295: Support lazy initialization of catalogs and persistence of catalog configurations

[Jing Ge]

+1 (binding)

Best Regards,
Jing

On Wed, Jun 14, 2023 at 3:28 PM Rui Fan <1996fan...@gmail.com> wrote:

> +1(binding)
>
> Best,
> Rui Fan
>
> On Wed, Jun 14, 2023 at 16:24 Hang Ruan  wrote:
>
> > +1 (non-binding)
> >
> > Thanks for Feng driving it.
> >
> > Best,
> > Hang
> >
> > Feng Jin  于2023年6月14日周三 10:36写道：
> >
> > > Hi everyone
> > >
> > > Thanks for all the feedback about the FLIP-295: Support lazy
> > initialization
> > > of catalogs and persistence of catalog configurations[1].
> > > [2] is the discussion thread.
> > >
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours(excluding weekends,until June 19, 10:00AM GMT) unless there is an
> > > objection or an insufficient number of votes.
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > [2]https://lists.apache.org/thread/dcwgv0gmngqt40fl3694km53pykocn5s
> > >
> > >
> > > Best,
> > > Feng
> > >
> >
>

Re: [VOTE] FLIP-294: Support Customized Catalog Modification Listener

[Jing Ge]

+1 (binding)

Best Regards,
Jing

On Wed, Jun 14, 2023 at 4:07 PM Benchao Li  wrote:

> +1 (binding)
>
> Shammon FY  于2023年6月14日周三 19:52写道：
>
> > Hi all:
> >
> > Thanks for all the feedback for FLIP-294: Support Customized Catalog
> > Modification Listener [1]. I would like to start a vote for it according
> to
> > the discussion in thread [2].
> >
> > The vote will be open for at least 72 hours(excluding weekends, until
> June
> > 19, 19:00 PM GMT) unless there is an objection or an insufficient number
> of
> > votes.
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Catalog+Modification+Listener
> > [2] https://lists.apache.org/thread/185mbcwnpokfop4xcb22r9bgfp2m68fx
> >
> >
> > Best,
> > Shammon FY
> >
>
>
> --
>
> Best,
> Benchao Li
>

Re: Re: [VOTE] FLIP-311: Support Call Stored Procedure

[Jing Ge]

+1(binding)

Best Regards,
Jing

On Tue, Jun 13, 2023 at 9:03 AM Mang Zhang  wrote:

> +1 (no-binding)
>
>
>
>
> --
>
> Best regards,
> Mang Zhang
>
>
>
>
>
> 在 2023-06-13 13:19:31，"Lincoln Lee"  写道：
> >+1 (binding)
> >
> >Best,
> >Lincoln Lee
> >
> >
> >Jingsong Li  于2023年6月13日周二 10:07写道：
> >
> >> +1
> >>
> >> On Mon, Jun 12, 2023 at 10:32 PM Rui Fan <1996fan...@gmail.com> wrote:
> >> >
> >> > +1 (binding)
> >> >
> >> > Best,
> >> > Rui Fan
> >> >
> >> > On Mon, Jun 12, 2023 at 22:20 Benchao Li 
> wrote:
> >> >
> >> > > +1 (binding)
> >> > >
> >> > > yuxia  于2023年6月12日周一 17:58写道：
> >> > >
> >> > > > Hi everyone,
> >> > > > Thanks for all the feedback about FLIP-311: Support Call Stored
> >> > > > Procedure[1]. Based on the discussion [2], we have come to a
> >> consensus,
> >> > > so
> >> > > > I would like to start a vote.
> >> > > > The vote will be open for at least 72 hours (until June 15th,
> 10:00AM
> >> > > GMT)
> >> > > > unless there is an objection or an insufficient number of votes.
> >> > > >
> >> > > >
> >> > > > [1]
> >> > > >
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-311%3A+Support+Call+Stored+Procedure
> >> > > > [2]
> https://lists.apache.org/thread/k6s50gcgznon9v1oylyh396gb5kgrwmd
> >> > > >
> >> > > > Best regards,
> >> > > > Yuxia
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > >
> >> > > Best,
> >> > > Benchao Li
> >> > >
> >>
>

Re: [DISCUSS] FLIP-311: Support Call Stored Procedure

[Jing Ge]

Hi yuxia,

Thanks for your proposal and sorry for the late reply. The FLIP is in good
shape. If I am not mistaken, Everything, that a stored procedure could do,
could also be done by a Flink job. The current stored procedure design is
to empower Catalogs to provide users some commonly used logics/functions
centrally and out-of-the-box, i.e. DRY. Is that correct?

Best regards,
Jing

On Thu, Jun 8, 2023 at 10:32 AM Jark Wu  wrote:

> Thank you for the proposal, yuxia! The FLIP looks good to me.
>
> Best,
> Jark
>
> > 2023年6月8日 11:39，yuxia  写道：
> >
> > Hi, all.
> > Thanks everyone for the valuable input. If there are are no further
> concerns about this FLIP[1], I would like to start voting next monday
> (6/12).
> >
> > [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-311%3A+Support+Call+Stored+Procedure
> >
> >
> > Best regards,
> > Yuxia
> >
> > - 原始邮件 -
> > 发件人: "Martijn Visser" 
> > 收件人: "dev" 
> > 发送时间: 星期二, 2023年 6 月 06日 下午 3:57:56
> > 主题: Re: [DISCUSS] FLIP-311: Support Call Stored Procedure
> >
> > Hi Yuxia,
> >
> > Thanks for the clarification. I would be +0 overall, because I think
> > without actually allowing creation/customization of stored procedures,
> the
> > value for the majority of Flink users will be minimal.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Tue, Jun 6, 2023 at 3:52 AM yuxia 
> wrote:
> >
> >> Hi, Martijn.
> >> Thanks for you feedback.
> >> 1: In this FLIP we don't intend to allow users to customize their own
> >> stored procedure for we don't want to expose too much to users too
> early as
> >> the FLIP said.
> >> The procedures are supposed to be provided only by Catalog. Catalog devs
> >> can write their build-in procedures, and return the procedure in method
> >> Catalog.getProcedure(ObjectPath procedurePath);
> >> So, there won't be SQL syntax to create/save a stored procedure in this
> >> FLIP. If we find we do need it, we can propse the SQL syntax to create a
> >> stored procedure in another dedicated FLIP.
> >>
> >> 2: The syntax `Call procedure_name(xx)` proposed in this FLIP is the
> >> default syntax in Calcite for call stored procedures. Actaully, we don't
> >> need to do any modifcation in flink-sql-parser module for syntax of
> calling
> >> a procedure. MySQL[1], Postgres[2], Oracle[3] also use the syntax to
> call a
> >> stored procedure.
> >>
> >>
> >> [1] https://dev.mysql.com/doc/refman/8.0/en/call.html
> >> [2] https://www.postgresql.org/docs/15/sql-call.html
> >> [3] https://docs.oracle.com/javadb/10.8.3.0/ref/rrefcallprocedure.html
> >>
> >> Best regards,
> >> Yuxia
> >>
> >> - 原始邮件 -
> >> 发件人: "Martijn Visser" 
> >> 收件人: "dev" 
> >> 发送时间: 星期一, 2023年 6 月 05日 下午 8:35:44
> >> 主题: Re: [DISCUSS] FLIP-311: Support Call Stored Procedure
> >>
> >> Hi Yuxia,
> >>
> >> Thanks for the FLIP. I have a couple of questions:
> >>
> >> 1. The syntax talks about how to CALL or SHOW the available stored
> >> procedures, but not on how to create one. Will there not be a SQL
> syntax to
> >> create/save a stored procedure?
> >> 2. Is there a default syntax in Calcite for stored procedures? What do
> >> other databases do, do they use CALL/SHOW or something like EXEC, USE?
> >>
> >> Best regards,
> >>
> >> Martijn
> >>
> >> On Mon, Jun 5, 2023 at 3:23 AM yuxia 
> wrote:
> >>
> >>> Hi, Jane.
> >>> Thanks for you input. I think we can add the auxiliary command show
> >>> procedures in this FLIP.
> >>> Following the syntax for show functions proposed in FLIP-297.
> >>> The syntax will be
> >>> SHOW PROCEDURES [ ( FROM | IN ) [catalog_name.]database_name ] [ [NOT]
> >>> (LIKE | ILIKE)  ].
> >>> I have updated to this FLIP.
> >>>
> >>> The other auxiliary commands maybe not suitable currently or need a
> >>> further/dedicated dicussion. Let's keep this FLIP focus.
> >>>
> >>> [1]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-297%3A+Improve+Auxiliary+Sql+Statements
> >>>
> >>> Best regards,
> >>> Yuxia
> >>>
> >>> - 原始邮件 -
> >>> 发件人: "Jane Chan" 
> >>> 收件人: "dev" 
> >>> 发送时间: 星期六, 2023年 6 月 03日 下午 7:04:39
> >>> 主题: Re: [DISCUSS] FLIP-311: Support Call Stored Procedure
> >>>
> >>> Hi Yuxia,
> >>>
> >>> Thanks for bringing this to the discussion. The call procedure is a
> >> widely
> >>> used feature and will be very useful for users.
> >>>
> >>> I just have one question regarding the usage. The FLIP mentioned that
> >>>
> >>> Flink will allow connector developers to develop their own built-in
> >> stored
>  procedures, and then enables users to call these predefiend stored
>  procedures.
> 
> >>> In this FLIP, we don't intend to allow users to customize their own
> >> stored
>  procedure  for we don't want to expose too much to users too early.
> >>>
> >>>
> >>> If I understand correctly, we might need to provide some auxiliary
> >> commands
> >>> to inform users what built-in procedures are provided and how to use
> >> them.
> >>> For example, Snowflake provides commands like [1] [2], and MySQL
> provides

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

[Jing Ge]

> This is by design. Most of these are @Public APIs that we had to carry
> around until Flink 2.0, because that was the initial guarantee that we
> gave people.
>

True, I knew @Public APIs could not be removed before the next major
release. I meant house cleaning without violation of these annotations'
design concept. i.e especially cleaning up for @PublicEvolving APIs since
they are customer-facing. Regular cleaning up with all other @Experimental
and @Internal APIs would be even better, if there might be some APIs marked
as @deprecated.

Best regards,
Jing



On Tue, Jun 13, 2023 at 4:25 PM Chesnay Schepler  wrote:

> On 13/06/2023 12:50, Jing Ge wrote:
> > One major issue we have, afaiu, is caused by the lack of
> housekeeping/house
> > cleaning, there are many APIs that were marked as deprecated a few years
> > ago and still don't get removed. Some APIs should be easy to remove and
> > others will need some more clear rules, like the issue discussed at [1].
>
This is by design. Most of these are @Public APIs that we had to carry
> around until Flink 2.0, because that was the initial guarantee that we
> gave people.


>
> As for the FLIP, I like the idea of explicitly writing down a
> deprecation period for APIs, particularly PublicEvolving ones.
> For Experimental I don't think it'd be a problem if we could change them
> right away,
> but looking back a bit I don't think it hurts us to also enforce some
> deprecation period.
> 1 release for both of these sound fine to me.
>
>
> My major point of contention is the removal of Public APIs between minor
> versions.
> This to me would a major setback towards a simpler upgrade path for users.
> If these can be removed in minor versions than what even is a major
> release?
> The very definition we have for Public APIs is that they stick around
> until the next major one.
> Any rule that theoretically allows for breaking changes in Public API in
> every minor release is in my opinion not a viable option.
>
>
> The "carry recent Public APIs forward into the next major release" thing
> seems to presume a linear release history (aka, if 2.0 is released after
> 1.20, then there will be no 1.21), which I doubt will be the case. The
> idea behind it is good, but I'd say the right conclusion would be to not
> make that API public if we know a new major release hits in 3 months and
> is about to modify it. With a regular schedule for major releases this
> wouldn't be difficult to do.
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

[Jing Ge]

Hi Becket,

Thanks for driving this important topic! There were many discussions
previously that ended up with waiting up for a clear API deprecation
process definition. This FLIP will help a lot.

I'd like to ask some questions to understand your thoughts.

Speaking of the FLIP,

*"Always add a "Since X.X.X" comment to indicate when was a class /
interface / method marked as deprecated."*
 Could you describe it with a code example? Do you mean Java comments?

*"At least 1 patch release for the affected minor release for
Experimental APIs"*
The rule is absolutely right. However, afaiac, deprecation is different as
modification. As a user/dev, I would appreciate, if I do not need to do any
migration work for any deprecated API between patch releases upgrade. BTW,
if experimental APIs are allowed to change between patches, could we just
change them instead of marking them as deprecated and create new ones to
replace them?

One major issue we have, afaiu, is caused by the lack of housekeeping/house
cleaning, there are many APIs that were marked as deprecated a few years
ago and still don't get removed. Some APIs should be easy to remove and
others will need some more clear rules, like the issue discussed at [1].

Some common questions could be:

1. How to make sure the new APIs cover all functionality, i.e. backward
compatible, before removing the deprecated APIs? Since the
functionalities could only be built with the new APIs iteratively, there
will be a while (might be longer than the migration period) that the new
APIs are not backward compatible with the deprecated ones.
2. Is it allowed to remove the deprecated APIs after the defined migration
period expires while the new APis are still not backward compatible?
3. For the case of core API upgrade with downstream implementations, e.g.
connectors, What is the feasible deprecation strategy? Option1 bottom-up:
make sure the downstream implementation is backward compatible before
removing the deprecated core APIs. Option2 top-down: once the downstream
implementation of new APIs works fine, we can remove the deprecated core
APIs after the migration period expires. The implementation of the
deprecated APIs will not get any further update in upcoming releases(it has
been removed). There might be some missing features in the downstream
implementation of new APIs compared to the old implementation. Both options
have their own pros and cons.


Best regards,
Jing


[1] https://lists.apache.org/thread/m3o48c2d8j9g5t9s89hqs6qvr924s71o


On Mon, Jun 12, 2023 at 6:31 PM Stefan Richter
 wrote:

> Hi,
>
> Thanks a lot for bringing up this topic and for the initial proposal. As
> more and more people are looking into running Flink as a continuous service
> this discussion is becoming very relevant.
>
> What I would like to see is a clearer definition for what we understand by
> stability and compatibility. Our current policy only talks about being able
> to “compile” and “run” with a different version. As far as I can see, there
> is no guarantee about the stability of observable behavior. I believe it’s
> important for the community to include this important aspect in the
> guarantees that we give as our policy.
>
> For all changes that we do to the stable parts of the API, we should also
> consider how easy or difficult different types of changes would be for
> running Flink as a service with continuous delivery. For example,
> introducing a new interface to evolve the methods would make it easier to
> write adapter code than changing method signatures in-place on the existing
> interface. Those concerns should be considered in our process for evolving
> interfaces.
>
> Best,
> Stefan
>
>
>
>   
> Stefan Richter
> Principal Engineer II
>
> Follow us:  <
> https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>
> 
>
>
>
> > On 11. Jun 2023, at 14:30, Becket Qin  wrote:
> >
> > Hi folks,
> >
> > As one of the release 2.0 efforts, the release managers were discussing
> our
> > API lifecycle policies. There have been FLIP-196[1] and FLIP-197[2] that
> > are relevant to this topic. These two FLIPs defined the stability
> guarantee
> > of the programming APIs with various different stability annotations, and
> > the promotion process. A recap of the conclusion is following:
> >
> > Stability:
> > @Internal API: can change between major/minor/patch releases.
> > @Experimental API: can change between major/minor/patch releases.
> > @PublicEvolving API: can change between major/minor releases.
> > @Public API: can only change between major releases.
> >
> > Promotion:
> > An @Experimental API should be promoted to @PublicEvolving after two
> > releases, and a @PublicEvolving API should be promoted to @Public API
> after
> > two releases, unless there is a documented reason not to do so.
> >
> > One thing not mentioned in these two FLIPs is the API dep

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

[Jing Ge]

Hi Aitozi,

Thanks for taking care of that part. I have no other concern.

Best regards,
Jing


On Mon, Jun 12, 2023 at 5:38 PM Aitozi  wrote:

> BTW, If there are no other more blocking issue / comments, I would like to
> start a VOTE in another thread this wednesday 6.14
>
> Thanks,
> Aitozi.
>
> Aitozi  于2023年6月12日周一 23:34写道：
>
> > Hi, Jing,
> > Thanks for your explanation. I get your point now.
> >
> > For the performance part, I think it's a good idea to run with returning
> a
> > big table case, the memory consumption
> > should be a point to be taken care about. Because in the ordered mode,
> the
> > head element in buffer may affect the
> > total memory consumption.
> >
> >
> > Thanks,
> > Aitozi.
> >
> >
> >
> > Jing Ge  于2023年6月12日周一 20:28写道：
> >
> >> Hi Aitozi,
> >>
> >> Which key will be used for lookup is not an issue, only one row will be
> >> required for each key in order to enrich it. True, it depends on the
> >> implementation whether multiple rows or single row for each key will be
> >> returned. However, for the lookup & enrichment scenario, one row/key is
> >> recommended, otherwise, like I mentioned previously, enrichment won't
> >> work.
> >>
> >> I am a little bit concerned about returning a big table for each key,
> >> since
> >> it will take the async call longer to return and need more memory. The
> >> performance tests should cover this scenario. This is not a blocking
> issue
> >> for this FLIP.
> >>
> >> Best regards,
> >> Jing
> >>
> >> On Sat, Jun 10, 2023 at 4:11 AM Aitozi  wrote:
> >>
> >> > Hi Jing,
> >> > I means the join key is not necessary to be the primary key or
> >> unique
> >> > index of the database.
> >> > In this situation, we may queried out multi rows for one join key. I
> >> think
> >> > that's why the
> >> > LookupFunction#lookup will return a collection of RowData.
> >> >
> >> > BTW, I think the behavior of lookup join will not affect the semantic
> of
> >> > the async udtf.
> >> > We use the Async TableFunction here and the table function can collect
> >> > multiple rows.
> >> >
> >> > Thanks,
> >> > Atiozi.
> >> >
> >> >
> >> >
> >> > Jing Ge  于2023年6月10日周六 00:15写道：
> >> >
> >> > > Hi Aitozi,
> >> > >
> >> > > The keyRow used in this case contains all keys[1].
> >> > >
> >> > > Best regards,
> >> > > Jing
> >> > >
> >> > > [1]
> >> > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> >> > >
> >> > >
> >> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi  wrote:
> >> > >
> >> > > > Hi Jing,
> >> > > >
> >> > > >  The performance test is added to the FLIP.
> >> > > >
> >> > > >  As I know, The lookup join can return multi rows, it depends
> on
> >> > > > whether  the join key
> >> > > > is the primary key of the external database or not. The `lookup`
> [1]
> >> > will
> >> > > > return a collection of
> >> > > > joined result, and each of them will be collected
> >> > > >
> >> > > >
> >> > > > [1]:
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> >> > > >
> >> > > >
> >> > > > Thanks,
> >> > > > Aitozi.
> >> > > >
> >> > > > Jing Ge  于2023年6月9日周五 17:05写道：
> >> > > >
> >> > > > > Hi Aitozi,
> >> > > > >
> >> > > > > Thanks for the feedback. Looking forward to the performance
> tests.
> >> > > > >
> >> > > > > Afaik, lookup returns one row for each key [1] [2].
> Conceptually,
&

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

[Jing Ge]

Hi Aitozi,

Which key will be used for lookup is not an issue, only one row will be
required for each key in order to enrich it. True, it depends on the
implementation whether multiple rows or single row for each key will be
returned. However, for the lookup & enrichment scenario, one row/key is
recommended, otherwise, like I mentioned previously, enrichment won't work.

I am a little bit concerned about returning a big table for each key, since
it will take the async call longer to return and need more memory. The
performance tests should cover this scenario. This is not a blocking issue
for this FLIP.

Best regards,
Jing

On Sat, Jun 10, 2023 at 4:11 AM Aitozi  wrote:

> Hi Jing,
> I means the join key is not necessary to be the primary key or unique
> index of the database.
> In this situation, we may queried out multi rows for one join key. I think
> that's why the
> LookupFunction#lookup will return a collection of RowData.
>
> BTW, I think the behavior of lookup join will not affect the semantic of
> the async udtf.
> We use the Async TableFunction here and the table function can collect
> multiple rows.
>
> Thanks,
> Atiozi.
>
>
>
> Jing Ge  于2023年6月10日周六 00:15写道：
>
> > Hi Aitozi,
> >
> > The keyRow used in this case contains all keys[1].
> >
> > Best regards,
> > Jing
> >
> > [1]
> >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> >
> >
> > On Fri, Jun 9, 2023 at 3:42 PM Aitozi  wrote:
> >
> > > Hi Jing,
> > >
> > >  The performance test is added to the FLIP.
> > >
> > >  As I know, The lookup join can return multi rows, it depends on
> > > whether  the join key
> > > is the primary key of the external database or not. The `lookup` [1]
> will
> > > return a collection of
> > > joined result, and each of them will be collected
> > >
> > >
> > > [1]:
> > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> > >
> > >
> > > Thanks,
> > > Aitozi.
> > >
> > > Jing Ge  于2023年6月9日周五 17:05写道：
> > >
> > > > Hi Aitozi,
> > > >
> > > > Thanks for the feedback. Looking forward to the performance tests.
> > > >
> > > > Afaik, lookup returns one row for each key [1] [2]. Conceptually, the
> > > > lookup function is used to enrich column(s) from the dimension table.
> > If,
> > > > for the given key, there will be more than one row, there will be no
> > way
> > > to
> > > > know which row will be used to enrich the key.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > [2]
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi  wrote:
> > > >
> > > > > Hi Jing
> > > > > Thanks for your good questions. I have updated the example to
> the
> > > > FLIP.
> > > > >
> > > > > > Only one row for each lookup
> > > > > lookup can also return multi rows, based on the query result. [1]
> > > > >
> > > > > [1]:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > > > >
> > > > > > If we use async calls with lateral join, my gut feeling is
> > > > > that we might have many more async calls than lookup join. I am not
> > > > really
> > > > > sure if we will be facing potential issues in this case or not.
> > > > >
> > > > > IMO, the work pattern is similar to the lookup function, for each
> row

Re: [VOTE] FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement

[Jing Ge]

+1(binding) Thanks!

Best regards,
Jing

On Mon, Jun 12, 2023 at 12:01 PM yuxia  wrote:

> +1 (binding)
> Thanks Mang driving it.
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "zhangmang1" 
> 收件人: "dev" 
> 发送时间: 星期一, 2023年 6 月 12日 下午 5:31:10
> 主题: [VOTE] FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS)
> statement
>
> Hi everyone,
>
> Thanks for all the feedback about FLIP-305: Support atomic for CREATE
> TABLE AS SELECT(CTAS) statement[1].
> [2] is the discussion thread.
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours (until June 15th, 10:00AM GMT) unless there is an objection or an
> insufficient number of votes.[1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-305%3A+Support+atomic+for+CREATE+TABLE+AS+SELECT%28CTAS%29+statement
> [2]https://lists.apache.org/thread/n6nsvbwhs5kwlj5kjgv24by2tk5mh9xd
>
>
>
>
>
>
>
> --
>
> Best regards,
> Mang Zhang
>

Re: [DISCUSS] FLIP-307: Flink connector Redshift

[Jing Ge]

Hi Samrat,

The FLIP looks good, thanks!

Best regards,
Jing


On Tue, Jun 6, 2023 at 8:16 PM Samrat Deb  wrote:

> Hi Jing,
>
> >  I would suggest adding that information into the
> FLIP.
>
> Updated now, please review the new version of flip whenever time.
>
> > +1 Looking forward to your PR :-)
> I will request for your review once m ready with PR :-)
>
> Bests,
> Samrat
>
> On Tue, Jun 6, 2023 at 11:43 PM Samrat Deb  wrote:
>
> > Hi Martijn,
> >
> > > If I understand this correctly, the Redshift sink
> > would not be able to support exactly-once, is that correct?
> >
> > As I delve deeper into the study of Redshift's capabilities, I have
> > discovered that it does support "merge into" operations [1] and some
> > merge into examples [2].
> > This opens up the possibility of implementing exactly-once semantics with
> > the connector.
> > However, I believe it would be prudent to start with a more focused scope
> > for the initial phase of implementation and defer the exact-once support
> > for subsequent iterations.
> >
> > Before finalizing the approach, I would greatly appreciate your thoughts
> > and suggestions on this matter.
> > Should we prioritize the initial implementation without exactly-once
> > support, or would you advise incorporating it right from the start?
> > Your insights and experiences would be immensely valuable in making this
> > decision.
> >
> >
> > [1]
> >
> https://docs.aws.amazon.com/redshift/latest/dg/t_updating-inserting-using-staging-tables-.html
> > [2] https://docs.aws.amazon.com/redshift/latest/dg/merge-examples.html
> >
> > Bests,
> > Samrat
> >
> > On Mon, Jun 5, 2023 at 7:09 PM Jing Ge 
> wrote:
> >
> >> Hi Samrat,
> >>
> >> Thanks for the feedback. I would suggest adding that information into
> the
> >> FLIP.
> >>
> >> +1 Looking forward to your PR :-)
> >>
> >> Best regards,
> >> Jing
> >>
> >> On Sat, Jun 3, 2023 at 9:19 PM Samrat Deb 
> wrote:
> >>
> >> > Hi Jing Ge,
> >> >
> >> > >>> Do you already have any prototype? I'd like to join the reviews.
> >> > The prototype is in progress. I will raise the dedicated PR for review
> >> soon
> >> > also notify in this thread as well .
> >> >
> >> > >>> Will the Redshift connector provide additional features
> >> > beyond the mediator/wrapper of the jdbc connector?
> >> >
> >> > Here are the additional features that the Flink connector for AWS
> >> Redshift
> >> > can provide on top of using JDBC:
> >> >
> >> > 1. Integration with AWS Redshift Workload Management (WLM): AWS
> Redshift
> >> > allows you to configure WLM[1] to manage query prioritization and
> >> resource
> >> > allocation. The Flink connector for Redshift will be agnostic to the
> >> > configured WLM and utilize it for scaling in and out for the sink.
> This
> >> > means that the connector can leverage the WLM capabilities of Redshift
> >> to
> >> > optimize the execution of queries and allocate resources efficiently
> >> based
> >> > on your defined workload priorities.
> >> >
> >> > 2. Abstraction of AWS Redshift Quotas and Limits: AWS Redshift imposes
> >> > certain quotas and limits[2] on various aspects such as the number of
> >> > clusters, concurrent connections, queries per second, etc. The Flink
> >> > connector for Redshift will provide an abstraction layer for users,
> >> > allowing them to work with Redshift without having to worry about
> these
> >> > specific limits. The connector will handle the management of
> connections
> >> > and queries within the defined quotas and limits, abstracting away the
> >> > complexity and ensuring compliance with Redshift's restrictions.
> >> >
> >> > These features aim to simplify the integration of Flink with AWS
> >> Redshift,
> >> > providing optimized resource utilization and transparent handling of
> >> > Redshift-specific limitations.
> >> >
> >> > Bests,
> >> > Samrat
> >> >
> >> > [1]
> >> >
> >> >
> >>
> https://docs.aws.amazon.com/redshift/latest/dg/cm-c-implementing-workload-management.html
> >> > [2]
> >> >
> >> >
> >>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

[Jing Ge]

Hi Aitozi,

The keyRow used in this case contains all keys[1].

Best regards,
Jing

[1]
https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49


On Fri, Jun 9, 2023 at 3:42 PM Aitozi  wrote:

> Hi Jing,
>
>  The performance test is added to the FLIP.
>
>  As I know, The lookup join can return multi rows, it depends on
> whether  the join key
> is the primary key of the external database or not. The `lookup` [1] will
> return a collection of
> joined result, and each of them will be collected
>
>
> [1]:
>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
>
>
> Thanks,
> Aitozi.
>
> Jing Ge  于2023年6月9日周五 17:05写道：
>
> > Hi Aitozi,
> >
> > Thanks for the feedback. Looking forward to the performance tests.
> >
> > Afaik, lookup returns one row for each key [1] [2]. Conceptually, the
> > lookup function is used to enrich column(s) from the dimension table. If,
> > for the given key, there will be more than one row, there will be no way
> to
> > know which row will be used to enrich the key.
> >
> > [1]
> >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > [2]
> >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> >
> > Best regards,
> > Jing
> >
> > On Fri, Jun 9, 2023 at 5:18 AM Aitozi  wrote:
> >
> > > Hi Jing
> > > Thanks for your good questions. I have updated the example to the
> > FLIP.
> > >
> > > > Only one row for each lookup
> > > lookup can also return multi rows, based on the query result. [1]
> > >
> > > [1]:
> > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > >
> > > > If we use async calls with lateral join, my gut feeling is
> > > that we might have many more async calls than lookup join. I am not
> > really
> > > sure if we will be facing potential issues in this case or not.
> > >
> > > IMO, the work pattern is similar to the lookup function, for each row
> > from
> > > the left table,
> > > it will evaluate the eval method once, so the async call numbers will
> not
> > > change.
> > > and the maximum calls in flight is limited by the Async operators
> buffer
> > > capacity
> > > which will be controlled by the option.
> > >
> > > BTW, for the naming of these option, I updated the FLIP about this you
> > can
> > > refer to
> > > the section of "ConfigOption" and "Rejected Alternatives"
> > >
> > > In the end, for the performance evaluation, I'd like to do some tests
> and
> > > will update it to the FLIP doc
> > >
> > > Thanks,
> > > Aitozi.
> > >
> > >
> > > Jing Ge  于2023年6月9日周五 07:23写道：
> > >
> > > > Hi Aitozi,
> > > >
> > > > Thanks for the clarification. The code example looks interesting. I
> > would
> > > > suggest adding them into the FLIP. The description with code examples
> > > will
> > > > help readers understand the motivation and how to use it. Afaiac, it
> > is a
> > > > valid feature for Flink users.
> > > >
> > > > As we knew, lookup join is based on temporal join, i.e. FOR
> SYSTEM_TIME
> > > AS
> > > > OF which is also used in your code example. Temporal join performs
> the
> > > > lookup based on the processing time match. Only one row for each
> > > > lookup(afaiu, I need to check the source code to double confirm) will
> > > > return for further enrichment. One the other hand, lateral join will
> > have
> > > > sub-queries correlated with every individual value of the reference
> > table
> > > > from the preceding part of the query and each sub query will return
> > > > multiple rows. If we use async calls with lateral join, my gut
> feeling
> > is
> > 

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

[Jing Ge]

Hi Aitozi,

Thanks for the feedback. Looking forward to the performance tests.

Afaik, lookup returns one row for each key [1] [2]. Conceptually, the
lookup function is used to enrich column(s) from the dimension table. If,
for the given key, there will be more than one row, there will be no way to
know which row will be used to enrich the key.

[1]
https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
[2]
https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196

Best regards,
Jing

On Fri, Jun 9, 2023 at 5:18 AM Aitozi  wrote:

> Hi Jing
> Thanks for your good questions. I have updated the example to the FLIP.
>
> > Only one row for each lookup
> lookup can also return multi rows, based on the query result. [1]
>
> [1]:
>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
>
> > If we use async calls with lateral join, my gut feeling is
> that we might have many more async calls than lookup join. I am not really
> sure if we will be facing potential issues in this case or not.
>
> IMO, the work pattern is similar to the lookup function, for each row from
> the left table,
> it will evaluate the eval method once, so the async call numbers will not
> change.
> and the maximum calls in flight is limited by the Async operators buffer
> capacity
> which will be controlled by the option.
>
> BTW, for the naming of these option, I updated the FLIP about this you can
> refer to
> the section of "ConfigOption" and "Rejected Alternatives"
>
> In the end, for the performance evaluation, I'd like to do some tests and
> will update it to the FLIP doc
>
> Thanks,
> Aitozi.
>
>
> Jing Ge  于2023年6月9日周五 07:23写道：
>
> > Hi Aitozi,
> >
> > Thanks for the clarification. The code example looks interesting. I would
> > suggest adding them into the FLIP. The description with code examples
> will
> > help readers understand the motivation and how to use it. Afaiac, it is a
> > valid feature for Flink users.
> >
> > As we knew, lookup join is based on temporal join, i.e. FOR SYSTEM_TIME
> AS
> > OF which is also used in your code example. Temporal join performs the
> > lookup based on the processing time match. Only one row for each
> > lookup(afaiu, I need to check the source code to double confirm) will
> > return for further enrichment. One the other hand, lateral join will have
> > sub-queries correlated with every individual value of the reference table
> > from the preceding part of the query and each sub query will return
> > multiple rows. If we use async calls with lateral join, my gut feeling is
> > that we might have many more async calls than lookup join. I am not
> really
> > sure if we will be facing potential issues in this case or not. Possible
> > issues I can think of now e.g. too many PRC calls, too many async calls
> > processing, the sub query will return a table which might be (too) big,
> and
> > might cause performance issues. I would suggest preparing some use cases
> > and running some performance tests to check it. These are my concerns
> about
> > using async calls with lateral join and I'd like to share with you, happy
> > to discuss with you and hear different opinions, hopefully the
> > discussion could help me understand it more deeply. Please correct me if
> I
> > am wrong.
> >
> > Best regards,
> > Jing
> >
> >
> > On Thu, Jun 8, 2023 at 7:22 AM Aitozi  wrote:
> >
> > > Hi Mason,
> > > Thanks for your input. I think if we support the user defined async
> > > table function,
> > > user will be able to use it to hold a batch data then handle it at one
> > time
> > > in the customized function.
> > >
> > > AsyncSink is meant for the sink operator. I have not figure out how to
> > > integrate in this case.
> > >
> > > Thanks,
> > > Atiozi.
> > >
> > >
> > > Mason Chen  于2023年6月8日周四 12:40写道：
> > >
> > > > Hi Aitozi,
> > > >
> > > > I think it makes sense to make it easier for SQL users to make RPCs.
> Do
> > > you
> > > > think your proposal can extend to the ability to batch data for the
> > RPC?
> > > > This is also another common strategy to increase throughp

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

[Jing Ge]

 process as an async RPC call process.
> > > >
> > > > Let's see how we can perform an async RPC call with lateral join:
> > > >
> > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > (2) Run query with
> > > >
> > > > Create function f1 as '...' ;
> > > >
> > > > SELECT o.order_id, o.total, c.country, c.zip FROM Orders  lateral
> table
> > > > (f1(order_id)) as T(...);
> > > >
> > > > As you can see, the lateral join version is more simple and intuitive
> > to
> > > > users. Users do not have to wrap a
> > > > LookupTableSource for the purpose of using async udtf.
> > > >
> > > > In the end, We can also see the user defined async table function is
> an
> > > > enhancement of the current lateral table join
> > > > which only supports sync lateral join now.
> > > >
> > > > Best,
> > > > Aitozi.
> > > >
> > > >
> > > > Jing Ge  于2023年6月2日周五 19:37写道：
> > > >
> > > >> Hi Aitozi,
> > > >>
> > > >> Thanks for the update. Just out of curiosity, what is the difference
> > > >> between the RPC call or query you mentioned and the lookup in a very
> > > >> general way? Since Lateral join is used in the FLIP. Is there any
> > > special
> > > >> thought for that? Sorry for asking so many questions. The FLIP
> > contains
> > > >> limited information to understand the motivation.
> > > >>
> > > >> Best regards,
> > > >> Jing
> > > >>
> > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi  wrote:
> > > >>
> > > >> > Hi Jing,
> > > >> > I have updated the proposed changes to the FLIP. IMO, lookup
> has
> > > its
> > > >> > clear
> > > >> > async call requirement is due to its IO heavy operator. In our
> > usage,
> > > >> sql
> > > >> > users have
> > > >> > logic to do some RPC call or query the third-party service which
> is
> > > >> also IO
> > > >> > intensive.
> > > >> > In these case, we'd like to leverage the async function to improve
> > the
> > > >> > throughput.
> > > >> >
> > > >> > Thanks,
> > > >> > Aitozi.
> > > >> >
> > > >> > Jing Ge  于2023年6月1日周四 22:55写道：
> > > >> >
> > > >> > > Hi Aitozi,
> > > >> > >
> > > >> > > Sorry for the late reply. Would you like to update the proposed
> > > >> changes
> > > >> > > with more details into the FLIP too?
> > > >> > > I got your point. It looks like a rational idea. However, since
> > > lookup
> > > >> > has
> > > >> > > its clear async call requirement, are there any real use cases
> > that
> > > >> > > need this change? This will help us understand the motivation.
> > After
> > > >> all,
> > > >> > > lateral join and temporal lookup join[1] are quite different.
> > > >> > >
> > > >> > > Best regards,
> > > >> > > Jing
> > > >> > >
> > > >> > >
> > > >> > > [1]
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > >> > >
> > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi 
> > > wrote:
> > > >> > >
> > > >> > > > Hi Jing,
> > > >> > > > What do you think about it? Can we move forward this
> > feature?
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Aitozi.
> > > >> > > >
> > > >> > > > Aitozi  于2023年5月29日周一 09:56写道：
> > > >> > > >
> > > >> > > > > Hi Jing,
> > > >> > > > > > "Do you mean to support the AyncTableFunction beyon

Call for Presentations: Flink Forward Seattle 2023

[Jing Ge]

Dear Flink developers & users,

We hope this email finds you well. We are excited to announce the Call for
Presentations for the upcoming Flink Forward Seattle 2023, the premier
event dedicated to Apache Flink and stream processing technologies. As a
prominent figure in the field, we invite you to submit your innovative
research, insightful experiences, and cutting-edge use cases for
consideration as a speaker at the conference.

Flink Forward Conference 2023 Details:
Date: November 6-7(training), November 8 (conference)
Location: Seattle United States

Flink Forward is a conference dedicated to the Apache Flink® community. In
2023 we shall have a full conference day following a 2-days long training
session. The conference gathers an international audience of CTOs/CIOs,
developers, data architects, data scientists, Apache Flink® core
committers, and the stream processing community, to share experiences,
exchange ideas and knowledge, and receive hands-on training sessions led by
Flink experts. We are seeking compelling presentations and
thought-provoking talks that cover a broad range of topics related to
Apache Flink, including but not limited to:

Flink architecture and internals
Flink performance optimization
Advanced Flink features and enhancements
Real-world use cases and success stories
Flink ecosystem and integrations
Stream processing at scale
Best practices for Flink application development

If you have an inspiring story, valuable insights, real-world application,
research breakthroughs, use case, best practice, or compelling vision of
the future for Flink, we encourage you to present it to a highly skilled
and enthusiastic community. We welcome submissions from both industry
professionals and academic researchers.

To submit your proposal, please visit the Flink Forward Conference website
at https://www.flink-forward.org/seattle-2023/call-for-presentations. The
submission form will require you to provide an abstract of your talk, along
with a brief biography and any supporting materials. The deadline for
submissions is July 12th 11:59 pm PDT.

We believe your contribution will greatly enrich the Flink Forward
Conference and provide invaluable insights to our attendees. This is an
excellent opportunity to connect with a diverse community of Flink
enthusiasts, network with industry experts, and gain recognition for your
expertise. We look forward to receiving your submission and welcoming you
as a speaker at the Flink Forward Conference.

Thank you for your time and consideration.

Best regards,

-- 

Jing Ge | Head of Engineering

j...@ververica.com

<https://www.ververica.com/>

Follow us @VervericaData

--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference - Tickets on SALE now!
<https://eu.eventscloud.com/ereg/newreg.php?eventid=200259741&#>

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Chausseestrasse 20, 10115 Berlin, Germany

--

Ververica GmbH

Registered at Amtsgericht Charlottenburg: HRB 158244 B

Managing Directors: Alexander Walden, Karl Anton Wehner, Yip Park Tung
Jason, Jinwei (Kevin) Zhang

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

[Jing Ge]

Hi Ron,

Thanks for sharing the insight. Agree that it is not doable to rewrite the
entire planner module with Java. That was the reason why it has been hidden
instead of replaced. I thought, since the community decided to walk away
from scala, we should at least not add any more new scala code. According
to your suggestion, it is not the fact. I think the community should
reconsider how to handle scala, since the more features we are developing
in those areas the more scala code we will have, which makes it even harder
(impossible) to migrate to java. This is beyond the scope of this
discussion. I will start a new thread to address it.

Best regards,
Jing


On Thu, Jun 8, 2023 at 5:20 AM liu ron  wrote:

> Hi, Ging
>
> Thanks for your valuable input about scala free.
>
> Firstly, reply to your question, using java to implement codegen is
> possible,  but we need to utilize some tools. I think the first alternative
> is to update our jdk version to 13, which provides text block feature[1]
> makes string format easier, and improves the multiple-line String
> readability and writability. However, we don't update the JDK version to 13
> in the short term future. The second alternative is to use a third library
> such as Freemarker and StringTemplate, but this is not easy work, we need
> to introduce extra dependency in table planner, and makes our
> implementation more complicated.
>
> We use a lot of scala code in the planner module, one of the main purposes
> is that codegen is more friendly, and many of the operators are also
> implemented through codegen. In the foreseeable future, we do not have the
> time and manpower to remove the scala code from the planner module, so
> scala-free is unlikely. From the point of view of development friendliness
> and development cost, scala is currently a relatively better solution for
> codegen. Suppose we need to completely rewrite the planner module in java
> in the future, I think it is better to consider what tools are used to
> support codegen in a unified way at that time, and I can't give a suitable
> tool at the moment.
>
> In summary, I don't think it is feasible to implement my FLIP with
> scala-free at this time.
>
> [1]: https://openjdk.org/jeps/378
>
> Best,
> Ron
>
>
> liu ron  于2023年6月8日周四 10:51写道：
>
> > Hi, Atiozi
> >
> > Thanks for your feedback.
> >
> > > Traverse the ExecNode DAG and create a FusionExecNode  for physical
> > operators that can be fused together.
> > which kind of operators can be fused together ? are the operators in an
> > operator chain? Is this optimization aligned to spark's whole stage
> codegen
> > ?
> > In theory, all kinds of operators can be fused together, our final goal
> is
> > to support all operators in batch mode, OperatorChain is just one case.
> Due
> > to this work effort is relatively large, so we need to complete it step
> by
> > step. Our OFCG not only achieves the ability of spark's whole stage
> > codegen, but also do more better than them.
> >
> > > does the "support codegen" means fusion codegen? but why we generate a
> > FusionTransformation when the member operator does not support codegen,
> IMO
> > it should
> > fallback to the current behavior.
> >
> > yes, it means the fusion codegen. In FLIP, I propose two operator fusion
> > mechanisms, one is like OperatorChain for single input operator, another
> is
> > MultipleInput fusion. For the former, our design mechanism is to fuse all
> > operators together at the ExecNode layer only if they all support fusion
> > codegen, or else go over the default OperatorChain. For the latter, in
> > order not to break the existing MultipleInput optimization purpose, so
> when
> > there are member operators that do not support fusion codegen,  we will
> > fall back to the current behavior[1], which means that a
> > FusionTransformation is created. here FusionTransformation is just a
> > surrogate for MultipleInput case, it actually means
> > MultipleInputTransformation, which fuses multiple physical operators.
> > Sorry, the description in the flow is not very clear and caused your
> > confusion.
> >
> > > In the end, I share the same idea with Lincoln about performance
> > benchmark.
> > Currently flink community's flink-benchmark only covers like schedule,
> > state, datastream operator's performance.
> > A good benchmark harness for sql operator will benefit the sql optimizer
> > topic and observation
> >
> > For the performance benchmark, I agree with you. As I stated earlier, I
> > think this is a new scope of work, we sh

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

[Jing Ge]

Hi Shammon,

If we take a look at the JDK Event design as a reference, we can even add
an Object into the event [1]. Back to the CatalogModificationEvent,
everything related to the event could be defined in the Event. If we want
to group some information into the Context, we could also consider adding
the CatalogModificationContext into the Event and make the onEvent() method
cleaner with only one input parameter CatalogModificationEvent, because the
interface CatalogModificationListener is the most often used interface for
users. Just my two cents.

Best regards,
Jing

[1]
http://www.java2s.com/example/java-src/pkg/java/util/eventobject-85298.html

On Thu, Jun 8, 2023 at 7:50 AM Shammon FY  wrote:

> Hi,
>
> To @Jing Ge
> > Thanks for the clarification. Just out of curiosity, if the context is
> not part of the event, why should it be the input parameter of each onEvent
> call?
>
> I think it's quite strange to put some information in an Event, such as a
> factory identifier for catalog, but they will be used by the listener.  I
> place it in the context class and I think it is more suitable than directly
> placing it in the event class.
>
> To @Mason
> > 1. I'm also curious about default implementations. Would atlas/datahub be
> supported by default?
>
> We won't do that and external systems such as atlas/datahub need to
> implement the listener themselves.
>
> > 2. The FLIP title is confusing to me, especially in distinguishing it
> from FLIP-314. Would a better FLIP title be "Support Catalog Metadata
> Listener" or something alike?
>
> Thanks, I think  "Support Catalog Modification Listener" will be
> more suitable, I'll update the title to it.
>
>
> Best,
> Shammon FY
>
>
> On Thu, Jun 8, 2023 at 12:25 PM Mason Chen  wrote:
>
> > Hi Shammon,
> >
> > FLIP generally looks good and I'm excited to see this feature.
> >
> > 1. I'm also curious about default implementations. Would atlas/datahub be
> > supported by default?
> > 2. The FLIP title is confusing to me, especially in distinguishing it
> from
> > FLIP-314. Would a better FLIP title be "Support Catalog Metadata
> Listener"
> > or something alike?
> >
> > Best,
> > Mason
> >
> > On Tue, Jun 6, 2023 at 3:33 AM Jing Ge 
> wrote:
> >
> > > Hi Shammon,
> > >
> > > Thanks for the clarification. Just out of curiosity, if the context is
> > not
> > > part of the event, why should it be the input parameter of each onEvent
> > > call?
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Tue, Jun 6, 2023 at 11:58 AM Leonard Xu  wrote:
> > >
> > > > Thanks Shammon for the timely update, the updated FLIP looks good to
> > me.
> > > >
> > > > Hope to see the vote thread and following FLIP-314 discussion thread.
> > > >
> > > > Best,
> > > > Leonard
> > > >
> > > > > On Jun 6, 2023, at 5:04 PM, Shammon FY  wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > Thanks for all the feedback.
> > > > >
> > > > > For @Jing Ge,
> > > > > I forget to update the demo code in the FLIP, the method is
> > > > > `onEvent(CatalogModificationEvent, CatalogModificationContext)` and
> > > there
> > > > > is no `onEvent(CatalogModificationEvent)`. I have updated the code.
> > > > Context
> > > > > contains some additional information that is not part of an Event,
> > but
> > > > > needs to be used in the listener, so we separate it from the event.
> > > > >
> > > > > For @Panagiotis,
> > > > > I think `ioExecutor` make sense to me and I have added it in
> > > > > `ContextModificationContext`, thanks
> > > > >
> > > > > For @Leonard,
> > > > > Thanks for your input.
> > > > > 1. I have updated `CatalogModificationContext` as an interface, as
> > well
> > > > as
> > > > > Context in CatalogModificationListenerFactory
> > > > > 2. Configuration sounds good to me, I have updated the method name
> > and
> > > > > getConfiguration in Context
> > > > >
> > > > > For @David,
> > > > > Yes, you're right. The listener will only be used on the client
> side
> > > and
> > > > > won't introduce a new code path for running per-job/per-session
> jobs.
> > > The
>

Re: [VOTE] FLIP-315: Support Operator Fusion Codegen for Flink SQL

[Jing Ge]

+1

Best Regards,
Jing

On Wed, Jun 7, 2023 at 10:52 AM weijie guo 
wrote:

> +1 (binding)
>
> Best regards,
>
> Weijie
>
>
> Jingsong Li  于2023年6月7日周三 15:59写道：
>
> > +1
> >
> > On Wed, Jun 7, 2023 at 3:03 PM Benchao Li  wrote:
> > >
> > > +1, binding
> > >
> > > Jark Wu  于2023年6月7日周三 14:44写道：
> > >
> > > > +1 (binding)
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > > 2023年6月7日 14:20，liu ron  写道：
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > Thanks for all the feedback about FLIP-315: Support Operator Fusion
> > > > Codegen
> > > > > for Flink SQL[1].
> > > > > [2] is the discussion thread.
> > > > >
> > > > > I'd like to start a vote for it. The vote will be open for at least
> > 72
> > > > > hours (until June 12th, 12:00AM GMT) unless there is an objection
> or
> > an
> > > > > insufficient number of votes.
> > > > >
> > > > > [1]:
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-315+Support+Operator+Fusion+Codegen+for+Flink+SQL
> > > > > [2]:
> > https://lists.apache.org/thread/9cnqhsld4nzdr77s2fwf00o9cb2g9fmw
> > > > >
> > > > > Best,
> > > > > Ron
> > > >
> > > >
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> >
>

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

[Jing Ge]

Hi Ron,

Thanks for raising the proposal. It is a very attractive idea! Since the
FLIP is a relatively complex one which contains three papers and a design
doc. It deserves more time for the discussion to make sure everyone is on
the same page. I have a NIT question which will not block your voting
process. Previously, it took the community a lot of effort to make Flink
kinds of scala free. Since the code base of the table module is too big,
instead of porting to Java, all scala code has been hidden. Furthermore,
there are ongoing efforts to remove Scala code from Flink. As you can see,
the community tries to limit (i.e. get rid of) scala code as much as
possible. I was wondering if it is possible for you to implement the FLIP
with scala free code?

Best regards,
Jing

[1] https://flink.apache.org/2022/02/22/scala-free-in-one-fifteen/

On Wed, Jun 7, 2023 at 5:33 PM Aitozi  wrote:

> Hi Ron:
> Sorry for the late reply after the voting process. I just want to ask
>
> > Traverse the ExecNode DAG and create a FusionExecNode  for physical
> operators that can be fused together.
> which kind of operators can be fused together ? are the operators in an
> operator chain? Is this optimization aligned to spark's whole stage codegen
> ?
>
> > If any member operator does not support codegen, generate a
> Transformation DAG based on the topological relationship of member ExecNode
>  and jump to step 8.
> step8: Generate a FusionTransformation, setting the parallelism and managed
> memory for the fused operator.
>
> does the "support codegen" means fusion codegen? but why we generate a
> FusionTransformation when the member operator does not support codegen, IMO
> it should
> fallback to the current behavior.
>
> In the end, I share the same idea with Lincoln about performance benchmark.
> Currently flink community's flink-benchmark only covers like schedule,
> state, datastream operator's performance.
> A good benchmark harness for sql operator will benefit the sql optimizer
> topic and observation
>
> Thanks,
> Atiozi.
>
>
> liu ron  于2023年6月6日周二 19:30写道：
>
> > Hi dev
> >
> > Thanks for all the feedback, it seems that here are no more comments, I
> > will
> > start a vote on FLIP-315 [1] later. Thanks again.
> >
> > [1]:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-315+Support+Operator+Fusion+Codegen+for+Flink+SQL
> >
> > Best,
> > Ron
> >
> > liu ron  于2023年6月5日周一 16:01写道：
> >
> > > Hi, Yun, Jinsong, Benchao
> > >
> > > Thanks for your valuable input about this FLIP.
> > >
> > > First of all, let me emphasize that from the technical implementation
> > > point of view, this design is feasible in both stream and batch
> > scenarios,
> > > so I consider both stream and batch mode in FLIP. In the stream
> scenario,
> > > for stateful operator, according to our business experience, basically
> > the
> > > bottleneck is on the state access, so the optimization effect of OFCG
> for
> > > the stream will not be particularly obvious, so we will not give
> priority
> > > to support it currently. On the contrary, in the batch scenario, where
> > CPU
> > > is the bottleneck, this optimization is gainful.
> > >
> > > Taking the above into account, we are able to support both stream and
> > > batch mode optimization in this design, but we will give priority to
> > > supporting batch operators. As benchao said, when we find a suitable
> > > streaming business scenario in the future, we can consider doing this
> > > optimization. Back to Yun issue, the design will break state
> > compatibility
> > > in stream mode as[1] and the version upgrade will not support this
> OFCG.
> > As
> > > mentioned earlier, we will not support this feature in stream mode in
> the
> > > short term.
> > >
> > > Also thanks to Benchao's suggestion, I will state the current goal of
> > that
> > > optimization in the FLIP, scoped to batch mode.
> > >
> > > Best,
> > > Ron
> > >
> > > liu ron  于2023年6月5日周一 15:04写道：
> > >
> > >> Hi, Lincoln
> > >>
> > >> Thanks for your appreciation of this design. Regarding your question:
> > >>
> > >> > do we consider adding a benchmark for the operators to intuitively
> > >> understand the improvement brought by each improvement?
> > >>
> > >> I think it makes sense to add a benchmark, Spark also has this
> benchmark
> > >> framework. But I think it is another story to introduce a benchmark
> > >> framework in Flink, we need to start a new discussion to this work.
> > >>
> > >> > for the implementation plan, mentioned in the FLIP that 1.18 will
> > >> support Calc, HashJoin and HashAgg, then what will be the next step?
> and
> > >> which operators do we ultimately expect to cover (all or specific
> ones)?
> > >>
> > >> Our ultimate goal is to support all operators in batch mode, but we
> > >> prioritize them according to their usage. Operators like Calc,
> HashJoin,
> > >> HashAgg, etc. are more commonly used, so we will support them first.
> > Later
> > >> we support the rest of the operators step

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

[Jing Ge]

Hi Mason,

It is a very practical feature that many users are keen to use. Thanks to
the previous discussion, the FLIP now looks informative. Thanks for your
proposal. One small suggestion is that the attached images are quite small
to read if we don't click and enlarge them. Besides that, It is difficult
to read the text on the current sequence diagram because it has a
transparent background. Would you like to replace it with a white
background?

Exactly-one is one of the key features of Kafka connector. I have the same
concern as Qingsheng. Since you have answered questions about it
previously, would you like to create an extra section in your FLIP to
explicitly describe scenarios when exactly-one is supported and when it is
not?

Best regards,
Jing

On Mon, Jun 5, 2023 at 11:41 PM Mason Chen  wrote:

> Hi all,
>
> I'm working on FLIP-246 again, for the Multi Cluster Kafka Source
> contribution. The document has been updated with some more context about
> how it can solve the Kafka topic removal scenario and a sequence diagram to
> illustrate how the components interact.
>
> Looking forward to any feedback!
>
> Best,
> Mason
>
> On Wed, Oct 12, 2022 at 11:12 PM Mason Chen 
> wrote:
>
> > Hi Ryan,
> >
> > Thanks for the additional context! Yes, the offset initializer would need
> > to take a cluster as a parameter and the MultiClusterKafkaSourceSplit can
> > be exposed in an initializer.
> >
> > Best,
> > Mason
> >
> > On Thu, Oct 6, 2022 at 11:00 AM Ryan van Huuksloot <
> > ryan.vanhuuksl...@shopify.com> wrote:
> >
> >> Hi Mason,
> >>
> >> Thanks for the clarification! In regards to the addition to the
> >> OffsetInitializer of this API - this would be an awesome addition and I
> >> think this entire FLIP would be a great addition to the Flink.
> >>
> >> To provide more context as to why we need particular offsets, we use
> >> Hybrid Source to currently backfill from buckets prior to reading from
> >> Kafka. We have a service that will tell us what offset has last been
> loaded
> >> into said bucket which we will use to initialize the KafkaSource
> >> OffsetsInitializer. We couldn't use a timestamp here and the offset
> would
> >> be different for each Cluster.
> >>
> >> In pseudocode, we'd want the ability to do something like this with
> >> HybridSources - if this is possible.
> >>
> >> ```scala
> >> val offsetsMetadata: Map[TopicPartition, Long] = // Get current offsets
> >> from OffsetReaderService
> >> val multiClusterArchiveSource: MultiBucketFileSource[T] = // Data is
> read
> >> from different buckets (multiple topics)
> >> val multiClusterKafkaSource: MultiClusterKafkaSource[T] =
> >> MultiClusterKafkaSource.builder()
> >>   .setKafkaMetadataService(new KafkaMetadataServiceImpl())
> >>   .setStreamIds(List.of("my-stream-1", "my-stream-2"))
> >>   .setGroupId("myConsumerGroup")
> >>
> >>
> .setDeserializer(KafkaRecordDeserializationSchema.valueOnly(StringDeserializer.class))
> >>   .setStartingOffsets(offsetsMetadata)
> >>   .setProperties(properties)
> >>   .build()
> >> val source =
> >>
> HybridSource.builder(multiClusterArchiveSource).addSource(multiClusterKafkaSource).build()
> >> ```
> >>
> >> Few notes:
> >> - TopicPartition won't work because the topic may be the same name as
> >> this is something that is supported IIRC
> >> - I chose to pass a map into starting offsets just for demonstrative
> >> purposes, I would be fine with whatever data structure would work best
> >>
> >> Ryan van Huuksloot
> >> Data Developer | Production Engineering | Streaming Capabilities
> >> [image: Shopify]
> >> <
> https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email>
> >>
> >>
> >> On Mon, Oct 3, 2022 at 11:29 PM Mason Chen 
> >> wrote:
> >>
> >>> Hi Ryan,
> >>>
> >>> Just copying your message over to the email chain.
> >>>
> >>> Hi Mason,
>  First off, thanks for putting this FLIP together! Sorry for the delay.
>  Full disclosure Mason and I chatted a little bit at Flink Forward
> 2022 but
>  I have tried to capture the questions I had for him then.
>  I'll start the conversation with a few questions:
>  1. The concept of streamIds is not clear to me in the proposal and
>  could use some more information. If I understand correctly, they will
> be
>  used in the MetadataService to link KafkaClusters to ones you want to
> use?
>  If you assign stream ids using `setStreamIds`, how can you dynamically
>  increase the number of clusters you consume if the list of StreamIds
> is
>  static? I am basing this off of your example .setStreamIds(List.of(
>  "my-stream-1", "my-stream-2")) so I could be off base with my
>  assumption. If you don't mind clearing up the intention, that would be
>  great!
>  2. How would offsets work if you wanted to use this
>  MultiClusterKafkaSource with a file based backfill? In the case I am
>  thinking of, you have a bucket backed archive of Kafka data per
> cluster.
>  and you want to pick up from

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

[Jing Ge]

Hi Shammon,

Thanks for the clarification. Just out of curiosity, if the context is not
part of the event, why should it be the input parameter of each onEvent
call?

Best regards,
Jing

On Tue, Jun 6, 2023 at 11:58 AM Leonard Xu  wrote:

> Thanks Shammon for the timely update, the updated FLIP looks good to me.
>
> Hope to see the vote thread and following FLIP-314 discussion thread.
>
> Best,
> Leonard
>
> > On Jun 6, 2023, at 5:04 PM, Shammon FY  wrote:
> >
> > Hi,
> >
> > Thanks for all the feedback.
> >
> > For @Jing Ge,
> > I forget to update the demo code in the FLIP, the method is
> > `onEvent(CatalogModificationEvent, CatalogModificationContext)` and there
> > is no `onEvent(CatalogModificationEvent)`. I have updated the code.
> Context
> > contains some additional information that is not part of an Event, but
> > needs to be used in the listener, so we separate it from the event.
> >
> > For @Panagiotis,
> > I think `ioExecutor` make sense to me and I have added it in
> > `ContextModificationContext`, thanks
> >
> > For @Leonard,
> > Thanks for your input.
> > 1. I have updated `CatalogModificationContext` as an interface, as well
> as
> > Context in CatalogModificationListenerFactory
> > 2. Configuration sounds good to me, I have updated the method name and
> > getConfiguration in Context
> >
> > For @David,
> > Yes, you're right. The listener will only be used on the client side and
> > won't introduce a new code path for running per-job/per-session jobs. The
> > listener will be created in `TableEnvironment` and `SqlGateway` which
> can a
> > `CatalogManager` with the listener.
> >
> >
> > Best,
> > Shammon FY
> >
> >
> > On Tue, Jun 6, 2023 at 3:33 PM David Morávek 
> > wrote:
> >
> >> Hi,
> >>
> >> Thanks for the FLIP! Data lineage is an important problem to tackle.
> >>
> >> Can you please expand on how this is planned to be wired into the
> >> JobManager? As I understand, the listeners will be configured globally
> (per
> >> cluster), so this won't introduce a new code path for running per-job /
> >> per-session user code. Is that correct?
> >>
> >> Best,
> >> D
> >>
> >> On Tue, Jun 6, 2023 at 9:17 AM Leonard Xu  wrote:
> >>
> >>> Thanks Shammon for driving this FLIP forward, I’ve several comments
> about
> >>> the updated FLIP.
> >>>
> >>> 1. CatalogModificationContext is introduced as a class instead of an
> >>> interface, is it a typo?
> >>>
> >>> 2. The FLIP defined multiple  Map config();  methods in
> >>> some Context classes, Could we use  Configuration
> >> getConfiguration()；Class
> >>> org.apache.flink.configuration.Configuration is recommend as it’s
> public
> >>> API and offers more useful methods as well.
> >>>
> >>> 3. The Context of CatalogModificationListenerFactory should be an
> >>> interface too, and getUserClassLoder()
> >>> would be more aligned with flink’s naming style.
> >>>
> >>>
> >>> Best,
> >>> Leonard
> >>>
> >>>> On May 26, 2023, at 4:08 PM, Shammon FY  wrote:
> >>>>
> >>>> Hi devs,
> >>>>
> >>>> We would like to bring up a discussion about FLIP-294: Support
> >> Customized
> >>>> Job Meta Data Listener[1]. We have had several discussions with Jark
> >> Wu,
> >>>> Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions
> >> and
> >>>> interfaces, and thanks for their valuable advice.
> >>>> The overall job and connector information is divided into metadata and
> >>>> lineage, this FLIP focuses on metadata and lineage will be discussed
> in
> >>>> another FLIP in the future. In this FLIP we want to add a customized
> >>>> listener in Flink to report catalog modifications to external metadata
> >>>> systems such as datahub[2] or atlas[3]. Users can view the specific
> >>>> information of connectors such as source and sink for Flink jobs in
> >> these
> >>>> systems, including fields, watermarks, partitions, etc.
> >>>>
> >>>> Looking forward to hearing from you, thanks.
> >>>>
> >>>>
> >>>> [1]
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener
> >>>> [2] https://datahub.io/
> >>>> [3] https://atlas.apache.org/#/
> >>>
> >>>
> >>
>
>

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

[Jing Ge]

Hi Shammon,

Thanks for driving it! It is a really interesting proposal. Looking forward
to the follow-up FLIP for the lineage feature, users will love it :-)

There are some inconsistencies in the content. In the very below example,
listener.onEvent(CatalogModificationEvent) is called, while in the
CatalogModificationListener interface definition, only
onEvent(CatalogModificationEvent, CatalogModificationContext) has been
defined.  I was wondering(NIT):

1. should there be another overloading method
onEvent(CatalogModificationEvent) alongside
onEvent(CatalogModificationEvent, CatalogModificationContext) ?
2. Since onEvent(CatalogModificationEvent) could be used, do we really need
CatalogModificationContext? API design example as reference: [1]

Best regards,
Jing


[1]
http://www.java2s.com/example/java-src/pkg/java/awt/event/actionlistener-add27.html

On Tue, Jun 6, 2023 at 7:43 AM Shammon FY  wrote:

> Hi devs:
>
> Thanks for all the feedback, and if there are no more comments, I will
> start a vote on FLIP-294 [1] later. Thanks again.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener
>
> Best,
> Shammon FY
>
> On Tue, Jun 6, 2023 at 1:40 PM Shammon FY  wrote:
>
> > Hi Martijn,
> >
> > Thanks for your attention, I will soon initiate a discussion about
> > FLIP-314.
> >
> > Best,
> > Shammon FY
> >
> >
> > On Fri, Jun 2, 2023 at 2:55 AM Martijn Visser 
> > wrote:
> >
> >> Hi Shammon,
> >>
> >> Just wanted to chip-in that I like the overall FLIP. Will be interesting
> >> to
> >> see the follow-up discussion on FLIP-314.
> >>
> >> Best regards,
> >>
> >> Martijn
> >>
> >> On Thu, Jun 1, 2023 at 5:45 AM yuxia 
> wrote:
> >>
> >> > Thanks for explanation. Make sense to me.
> >> >
> >> > Best regards,
> >> > Yuxia
> >> >
> >> > - 原始邮件 -
> >> > 发件人: "Shammon FY" 
> >> > 收件人: "dev" 
> >> > 发送时间: 星期四, 2023年 6 月 01日 上午 10:45:12
> >> > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
> >> >
> >> > Thanks yuxia, you're right and I'll add the new database to
> >> > AlterDatabaseEvent.
> >> >
> >> > I added `ignoreIfNotExists` for AlterDatabaseEvent because it is a
> >> > parameter in the `Catalog.alterDatabase` method. Although this value
> is
> >> > currently always false in `AlterDatabaseOperation`, I think it's
> better
> >> > to stay consistent with `Catalog.alterDatabase`. What do you think?
> >> >
> >> > Best,
> >> > Shammon FY
> >> >
> >> > On Thu, Jun 1, 2023 at 10:25 AM yuxia 
> >> wrote:
> >> >
> >> > > Hi, Shammon.
> >> > > I mean do we need to contain the new database after alter in
> >> > > AlterDatabaseEvent?  So that the listener can know what has been
> >> modified
> >> > > for the database. Or the listener don't need to care about the
> actual
> >> > > modification.
> >> > > Also, I'm wondering whether AlterDatabaseEven need to include
> >> > > ignoreIfNotExists method since alter database operation don't have
> >> such
> >> > > syntax like 'alter database if exists xxx'.
> >> > >
> >> > > Best regards,
> >> > > Yuxia
> >> > >
> >> > > - 原始邮件 -
> >> > > 发件人: "Shammon FY" 
> >> > > 收件人: "dev" 
> >> > > 发送时间: 星期三, 2023年 5 月 31日 下午 2:55:26
> >> > > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data
> Listener
> >> > >
> >> > > Hi yuxia
> >> > >
> >> > > Thanks for your input. The `AlterDatabaseEvent` extends
> >> > > `DatabaseModificationEvent` which has the original database.
> >> > >
> >> > > Best,
> >> > > Shammon FY
> >> > >
> >> > > On Wed, May 31, 2023 at 2:24 PM yuxia 
> >> > wrote:
> >> > >
> >> > > > Thanks Shammon for driving it.
> >> > > > The FLIP generally looks good to me. I only have one question.
> >> > > > WRT AlterDatabaseEvent, IIUC, it'll contain the origin database
> name
> >> > and
> >> > > > the new CatalogDatabase after modified. Is it enough only pass the
> >> > origin
> >> > > > database name? Will it be better to contain the origin
> >> CatalogDatabase
> >> > so
> >> > > > that listener have ways to know what changes?
> >> > > >
> >> > > > Best regards,
> >> > > > Yuxia
> >> > > >
> >> > > > - 原始邮件 -
> >> > > > 发件人: "ron9 liu" 
> >> > > > 收件人: "dev" 
> >> > > > 发送时间: 星期三, 2023年 5 月 31日 上午 11:36:04
> >> > > > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data
> >> Listener
> >> > > >
> >> > > > Hi, Shammon
> >> > > >
> >> > > > Thanks for driving this FLIP, It will enforce the Flink metadata
> >> > > capability
> >> > > > from the platform produce perspective. The overall design looks
> >> good to
> >> > > me,
> >> > > > I just have some small question:
> >> > > > 1. Regarding CatalogModificationListenerFactory#createListener
> >> method,
> >> > I
> >> > > > think it would be better to pass Context as its parameter instead
> of
> >> > two
> >> > > > specific Object. In this way, we can easily extend it in the
> future
> >> and
> >> > > > there will be no compatibility problems. Refer to
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.co

Re: [DISCUSS] FLIP-307: Flink connector Redshift

[Jing Ge]

Hi Samrat,

Thanks for the feedback. I would suggest adding that information into the
FLIP.

+1 Looking forward to your PR :-)

Best regards,
Jing

On Sat, Jun 3, 2023 at 9:19 PM Samrat Deb  wrote:

> Hi Jing Ge,
>
> >>> Do you already have any prototype? I'd like to join the reviews.
> The prototype is in progress. I will raise the dedicated PR for review soon
> also notify in this thread as well .
>
> >>> Will the Redshift connector provide additional features
> beyond the mediator/wrapper of the jdbc connector?
>
> Here are the additional features that the Flink connector for AWS Redshift
> can provide on top of using JDBC:
>
> 1. Integration with AWS Redshift Workload Management (WLM): AWS Redshift
> allows you to configure WLM[1] to manage query prioritization and resource
> allocation. The Flink connector for Redshift will be agnostic to the
> configured WLM and utilize it for scaling in and out for the sink. This
> means that the connector can leverage the WLM capabilities of Redshift to
> optimize the execution of queries and allocate resources efficiently based
> on your defined workload priorities.
>
> 2. Abstraction of AWS Redshift Quotas and Limits: AWS Redshift imposes
> certain quotas and limits[2] on various aspects such as the number of
> clusters, concurrent connections, queries per second, etc. The Flink
> connector for Redshift will provide an abstraction layer for users,
> allowing them to work with Redshift without having to worry about these
> specific limits. The connector will handle the management of connections
> and queries within the defined quotas and limits, abstracting away the
> complexity and ensuring compliance with Redshift's restrictions.
>
> These features aim to simplify the integration of Flink with AWS Redshift,
> providing optimized resource utilization and transparent handling of
> Redshift-specific limitations.
>
> Bests,
> Samrat
>
> [1]
>
> https://docs.aws.amazon.com/redshift/latest/dg/cm-c-implementing-workload-management.html
> [2]
>
> https://docs.aws.amazon.com/redshift/latest/mgmt/amazon-redshift-limits.html
>
> On Sat, Jun 3, 2023 at 11:40 PM Samrat Deb  wrote:
>
> > Hi Ahmed,
> >
> > >>> please let me know If you need any collaboration regarding
> integration
> > with
> > AWS connectors credential providers or regarding FLIP-171 I would be more
> > than happy to assist.
> >
> > Sure, I will reach out incase of any hands required.
> >
> >
> >
> > On Fri, Jun 2, 2023 at 6:12 PM Jing Ge 
> wrote:
> >
> >> Hi Samrat,
> >>
> >> Excited to see your proposal. Supporting data warehouses is one of the
> >> major tracks for Flink. Thanks for driving it! Happy to see that we
> >> reached
> >> consensus to prioritize the Sink over Source in the previous discussion.
> >> Do
> >> you already have any prototype? I'd like to join the reviews.
> >>
> >> Just out of curiosity, speaking of JDBC mode, according to the FLIP, it
> >> should be doable to directly use the jdbc connector with Redshift, if I
> am
> >> not mistaken. Will the Redshift connector provide additional features
> >> beyond the mediator/wrapper of the jdbc connector?
> >>
> >> Best regards,
> >> Jing
> >>
> >> On Thu, Jun 1, 2023 at 8:22 PM Ahmed Hamdy 
> wrote:
> >>
> >> > Hi Samrat
> >> >
> >> > Thanks for putting up this FLIP. I agree regarding the importance of
> the
> >> > use case.
> >> > please let me know If you need any collaboration regarding integration
> >> with
> >> > AWS connectors credential providers or regarding FLIP-171 I would be
> >> more
> >> > than happy to assist.
> >> > I also like Leonard's proposal for starting with DataStreamSink and
> >> > TableSink, It would be great to have some milestones delivered as soon
> >> as
> >> > ready.
> >> > best regards
> >> > Ahmed Hamdy
> >> >
> >> >
> >> > On Wed, 31 May 2023 at 11:15, Samrat Deb 
> wrote:
> >> >
> >> > > Hi Liu Ron,
> >> > >
> >> > > > 1. Regarding the  `read.mode` and `write.mode`, you say here
> >> provides
> >> > two
> >> > > modes, respectively, jdbc and `unload or copy`, What is the default
> >> value
> >> > > for `read.mode` and `write.mode?
> >> > >
> >> > > I have made an effort to make the configuration options `read.m

Re: [DISCUSS] FLIP-307: Flink connector Redshift

[Jing Ge]

Hi Samrat,

Excited to see your proposal. Supporting data warehouses is one of the
major tracks for Flink. Thanks for driving it! Happy to see that we reached
consensus to prioritize the Sink over Source in the previous discussion. Do
you already have any prototype? I'd like to join the reviews.

Just out of curiosity, speaking of JDBC mode, according to the FLIP, it
should be doable to directly use the jdbc connector with Redshift, if I am
not mistaken. Will the Redshift connector provide additional features
beyond the mediator/wrapper of the jdbc connector?

Best regards,
Jing

On Thu, Jun 1, 2023 at 8:22 PM Ahmed Hamdy  wrote:

> Hi Samrat
>
> Thanks for putting up this FLIP. I agree regarding the importance of the
> use case.
> please let me know If you need any collaboration regarding integration with
> AWS connectors credential providers or regarding FLIP-171 I would be more
> than happy to assist.
> I also like Leonard's proposal for starting with DataStreamSink and
> TableSink, It would be great to have some milestones delivered as soon as
> ready.
> best regards
> Ahmed Hamdy
>
>
> On Wed, 31 May 2023 at 11:15, Samrat Deb  wrote:
>
> > Hi Liu Ron,
> >
> > > 1. Regarding the  `read.mode` and `write.mode`, you say here provides
> two
> > modes, respectively, jdbc and `unload or copy`, What is the default value
> > for `read.mode` and `write.mode?
> >
> > I have made an effort to make the configuration options `read.mode` and
> > `write.mode` mandatory for the "flink-connector-redshift" according to
> > FLIP[1]. The rationale behind this decision is to empower users who are
> > familiar with their Redshift setup and have specific expectations for the
> > sink. By making these configurations mandatory, users can have more
> control
> > and flexibility in configuring the connector to meet their requirements.
> >
> > However, I am open to receiving feedback on whether it would be
> beneficial
> > to make the configuration options non-mandatory and set default values
> for
> > them. If you believe there are advantages to having default values or any
> > other suggestions, please share your thoughts. Your feedback is highly
> > appreciated.
> >
> > >  2. For Source, does it both support batch read and streaming read?
> >
> > Redshift currently does not provide native support for streaming reads,
> > although it does support streaming writes[2]. As part of the plan, I
> intend
> > to conduct a proof of concept and benchmarking to explore the
> possibilities
> > of implementing streaming reads using the Flink JDBC connector, as
> Redshift
> > is JDBC compatible.
> > However, it is important to note that, in the initial phase of
> > implementation, the focus will primarily be on supporting batch reads
> > rather than streaming reads. This approach will allow us to deliver a
> > robust and reliable solution for batch processing in phase 2 of the
> > implementation.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
> > [2]
> >
> >
> https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-streaming-ingestion.html
> >
> > Bests,
> > Samrat
> >
> > On Wed, May 31, 2023 at 8:03 AM liu ron  wrote:
> >
> > > Hi, Samrat
> > >
> > > Thanks for driving this FLIP. It looks like supporting
> > > flink-connector-redshift is very useful to Flink. I have two question:
> > > 1. Regarding the  `read.mode` and `write.mode`, you say here provides
> two
> > > modes, respectively, jdbc and `unload or copy`, What is the default
> value
> > > for `read.mode` and `write.mode?
> > > 2. For Source, does it both support batch read and streaming read?
> > >
> > >
> > > Best,
> > > Ron
> > >
> > > Samrat Deb  于2023年5月30日周二 17:15写道：
> > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
> > > >
> > > > [note] Missed the trailing link for previous mail
> > > >
> > > >
> > > >
> > > > On Tue, May 30, 2023 at 2:43 PM Samrat Deb 
> > > wrote:
> > > >
> > > > > Hi Leonard,
> > > > >
> > > > > > and I’m glad to help review the design as well as the code
> review.
> > > > > Thank you so much. It would be really great and helpful to bring
> > > > > flink-connector-redshift for flink users :) .
> > > > >
> > > > > I have divided the implementation in 3 phases in the `Scope`
> > > Section[1].
> > > > > 1st phase is to
> > > > >
> > > > >- Integrate with Flink Sink API (*FLIP-171*
> > > > ><
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
> > > >
> > > > >)
> > > > >
> > > > >
> > > > > > About the implementation phases, How about prioritizing support
> for
> > > the
> > > > > Datastream Sink API and TableSink API in the first phase?
> > > > > I can completely resonate with you to prioritize support for
> > Datastream
> > > > > Sink API and TableSink API in the first phase.
> > > > > I will update the FLIP[1] as you have suggested.
> > > > >
> > > > > > It

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

[Jing Ge]

Hi Aitozi,

Thanks for the update. Just out of curiosity, what is the difference
between the RPC call or query you mentioned and the lookup in a very
general way? Since Lateral join is used in the FLIP. Is there any special
thought for that? Sorry for asking so many questions. The FLIP contains
limited information to understand the motivation.

Best regards,
Jing

On Fri, Jun 2, 2023 at 3:48 AM Aitozi  wrote:

> Hi Jing,
> I have updated the proposed changes to the FLIP. IMO, lookup has its
> clear
> async call requirement is due to its IO heavy operator. In our usage, sql
> users have
> logic to do some RPC call or query the third-party service which is also IO
> intensive.
> In these case, we'd like to leverage the async function to improve the
> throughput.
>
> Thanks,
> Aitozi.
>
> Jing Ge  于2023年6月1日周四 22:55写道：
>
> > Hi Aitozi,
> >
> > Sorry for the late reply. Would you like to update the proposed changes
> > with more details into the FLIP too?
> > I got your point. It looks like a rational idea. However, since lookup
> has
> > its clear async call requirement, are there any real use cases that
> > need this change? This will help us understand the motivation. After all,
> > lateral join and temporal lookup join[1] are quite different.
> >
> > Best regards,
> > Jing
> >
> >
> > [1]
> >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> >
> > On Wed, May 31, 2023 at 8:53 AM Aitozi  wrote:
> >
> > > Hi Jing,
> > > What do you think about it? Can we move forward this feature?
> > >
> > > Thanks,
> > > Aitozi.
> > >
> > > Aitozi  于2023年5月29日周一 09:56写道：
> > >
> > > > Hi Jing,
> > > > > "Do you mean to support the AyncTableFunction beyond the
> > > > LookupTableSource?"
> > > > Yes, I mean to support the AyncTableFunction beyond the
> > > LookupTableSource.
> > > >
> > > > The "AsyncTableFunction" is the function with ability to be executed
> > > async
> > > > (with AsyncWaitOperator).
> > > > The async lookup join is a one of usage of this. So, we don't have to
> > > bind
> > > > the AyncTableFunction with LookupTableSource.
> > > > If User-defined AsyncTableFunction is supported, user can directly
> use
> > > > lateral table syntax to perform async operation.
> > > >
> > > > > "It would be better if you could elaborate the proposed changes wrt
> > the
> > > > CorrelatedCodeGenerator with more details"
> > > >
> > > > In the proposal, we use lateral table syntax to support the async
> table
> > > > function. So the planner will also treat this statement to a
> > > > CommonExecCorrelate node. So the runtime code should be generated in
> > > > CorrelatedCodeGenerator.
> > > > In CorrelatedCodeGenerator, we will know the TableFunction's Kind of
> > > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > > > For  `FunctionKind.ASYNC_TABLE` we can generate a AsyncWaitOperator
> to
> > > > execute the async table function.
> > > >
> > > >
> > > > Thanks,
> > > > Aitozi.
> > > >
> > > >
> > > > Jing Ge  于2023年5月29日周一 03:22写道：
> > > >
> > > >> Hi Aitozi,
> > > >>
> > > >> Thanks for the clarification. The naming "Lookup" might suggest
> using
> > it
> > > >> for table look up. But conceptually what the eval() method will do
> is
> > to
> > > >> get a collection of results(Row, RowData) from the given keys. How
> it
> > > will
> > > >> be done depends on the implementation, i.e. you can implement your
> own
> > > >> Source[1][2]. The example in the FLIP should be able to be handled
> in
> > > this
> > > >> way.
> > > >>
> > > >> Do you mean to support the AyncTableFunction beyond the
> > > LookupTableSource?
> > > >> It would be better if you could elaborate the proposed changes wrt
> the
> > > >> CorrelatedCodeGenerator with more details. Thanks!
> > > >>
> > > >> Best regards,
> > > >> Jing
> > > >>
> > > >> [1]
> > > >>
> >

Re: [DISCUSS] Update Flink Roadmap

[Jing Ge]

Hi Jark,

Fair enough. Let's do it like you suggested. Thanks!

Best regards,
Jing

On Thu, Jun 1, 2023 at 6:00 PM Jark Wu  wrote:

> Hi Jing,
>
> This thread is for discussing the roadmap for versions 1.18, 2.0, and even
> more.
> One of the outcomes of this discussion will be an updated version of the
> current roadmap.
> Let's work together on refining the roadmap in this thread.
>
> Best,
> Jark
>
> On Thu, 1 Jun 2023 at 23:25, Jing Ge  wrote:
>
> > Hi Jark,
> >
> > Thanks for driving it! For point 2, since we are developing 1.18 now,
> > does it make sense to update the roadmap this time while we are releasing
> > 1.18? This discussion thread will be focusing on the Flink 2.0 roadmap,
> as
> > you mentioned previously. WDYT?
> >
> > Best regards,
> > Jing
> >
> > On Thu, Jun 1, 2023 at 3:31 PM Jark Wu  wrote:
> >
> > > Hi all,
> > >
> > > Martijn and I would like to initiate a discussion on the Flink roadmap,
> > > which should cover the project's long-term roadmap and the regular
> update
> > > mechanism.
> > >
> > > Xintong has already started a discussion about Flink 2.0 planning. One
> of
> > > the points raised in that discussion is that we should have a
> high-level
> > > discussion of the roadmap to present where the project is heading
> (which
> > > doesn't necessarily need to block the Flink 2.0 planning). Moreover,
> the
> > > roadmap on the Flink website [1] hasn't been updated for half a year,
> and
> > > the last update was for the feature radar for the 1.15 release. It has
> > been
> > > 2 years since the community discussed Flink's overall roadmap.
> > >
> > > I would like to raise two topics for discussion:
> > >
> > > 1. The new roadmap. This should be an updated version of the current
> > > roadmap[1].
> > > 2. A mechanism to regularly discuss and update the roadmap.
> > >
> > > To make the first topic discussion more efficient, Martijn and I
> > volunteer
> > > to summarize the ongoing big things of different components and
> present a
> > > roadmap draft to the community in the next few weeks. This should be a
> > good
> > > starting point for a more detailed discussion.
> > >
> > > Regarding the regular update mechanism, there was a proposal in a
> thread
> > > [2] three years ago to make the release manager responsible for
> updating
> > > the roadmap. However, it appears that this was not documented as a
> > release
> > > management task [3], and the roadmap update wasn't performed for
> releases
> > > 1.16 and 1.17.
> > >
> > > In my opinion, making release managers responsible for keeping the
> > roadmap
> > > up to date is a good idea. Specifically, release managers of release X
> > can
> > > kick off the roadmap update at the beginning of release X, which can
> be a
> > > joint task with collecting a feature list [4]. Additionally, release
> > > managers of release X-1 can help verify and remove the accomplished
> items
> > > from the roadmap and update the feature radar.
> > >
> > > What do you think? Do you have other ideas?
> > >
> > > Best,
> > > Jark & Martijn
> > >
> > > [1]: https://flink.apache.org/roadmap.html
> > > [2]: https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> > > [3]:
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> > > [4]: https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> > >
> >
>

Re: [DISCUSS] Update Flink Roadmap

[Jing Ge]

Hi Jark,

Thanks for driving it! For point 2, since we are developing 1.18 now,
does it make sense to update the roadmap this time while we are releasing
1.18? This discussion thread will be focusing on the Flink 2.0 roadmap, as
you mentioned previously. WDYT?

Best regards,
Jing

On Thu, Jun 1, 2023 at 3:31 PM Jark Wu  wrote:

> Hi all,
>
> Martijn and I would like to initiate a discussion on the Flink roadmap,
> which should cover the project's long-term roadmap and the regular update
> mechanism.
>
> Xintong has already started a discussion about Flink 2.0 planning. One of
> the points raised in that discussion is that we should have a high-level
> discussion of the roadmap to present where the project is heading (which
> doesn't necessarily need to block the Flink 2.0 planning). Moreover, the
> roadmap on the Flink website [1] hasn't been updated for half a year, and
> the last update was for the feature radar for the 1.15 release. It has been
> 2 years since the community discussed Flink's overall roadmap.
>
> I would like to raise two topics for discussion:
>
> 1. The new roadmap. This should be an updated version of the current
> roadmap[1].
> 2. A mechanism to regularly discuss and update the roadmap.
>
> To make the first topic discussion more efficient, Martijn and I volunteer
> to summarize the ongoing big things of different components and present a
> roadmap draft to the community in the next few weeks. This should be a good
> starting point for a more detailed discussion.
>
> Regarding the regular update mechanism, there was a proposal in a thread
> [2] three years ago to make the release manager responsible for updating
> the roadmap. However, it appears that this was not documented as a release
> management task [3], and the roadmap update wasn't performed for releases
> 1.16 and 1.17.
>
> In my opinion, making release managers responsible for keeping the roadmap
> up to date is a good idea. Specifically, release managers of release X can
> kick off the roadmap update at the beginning of release X, which can be a
> joint task with collecting a feature list [4]. Additionally, release
> managers of release X-1 can help verify and remove the accomplished items
> from the roadmap and update the feature radar.
>
> What do you think? Do you have other ideas?
>
> Best,
> Jark & Martijn
>
> [1]: https://flink.apache.org/roadmap.html
> [2]: https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> [3]:
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> [4]: https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

[Jing Ge]

Hi Aitozi,

Sorry for the late reply. Would you like to update the proposed changes
with more details into the FLIP too?
I got your point. It looks like a rational idea. However, since lookup has
its clear async call requirement, are there any real use cases that
need this change? This will help us understand the motivation. After all,
lateral join and temporal lookup join[1] are quite different.

Best regards,
Jing


[1]
https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54

On Wed, May 31, 2023 at 8:53 AM Aitozi  wrote:

> Hi Jing,
> What do you think about it? Can we move forward this feature?
>
> Thanks,
> Aitozi.
>
> Aitozi  于2023年5月29日周一 09:56写道：
>
> > Hi Jing,
> > > "Do you mean to support the AyncTableFunction beyond the
> > LookupTableSource?"
> > Yes, I mean to support the AyncTableFunction beyond the
> LookupTableSource.
> >
> > The "AsyncTableFunction" is the function with ability to be executed
> async
> > (with AsyncWaitOperator).
> > The async lookup join is a one of usage of this. So, we don't have to
> bind
> > the AyncTableFunction with LookupTableSource.
> > If User-defined AsyncTableFunction is supported, user can directly use
> > lateral table syntax to perform async operation.
> >
> > > "It would be better if you could elaborate the proposed changes wrt the
> > CorrelatedCodeGenerator with more details"
> >
> > In the proposal, we use lateral table syntax to support the async table
> > function. So the planner will also treat this statement to a
> > CommonExecCorrelate node. So the runtime code should be generated in
> > CorrelatedCodeGenerator.
> > In CorrelatedCodeGenerator, we will know the TableFunction's Kind of
> > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > For  `FunctionKind.ASYNC_TABLE` we can generate a AsyncWaitOperator to
> > execute the async table function.
> >
> >
> > Thanks,
> > Aitozi.
> >
> >
> > Jing Ge  于2023年5月29日周一 03:22写道：
> >
> >> Hi Aitozi,
> >>
> >> Thanks for the clarification. The naming "Lookup" might suggest using it
> >> for table look up. But conceptually what the eval() method will do is to
> >> get a collection of results(Row, RowData) from the given keys. How it
> will
> >> be done depends on the implementation, i.e. you can implement your own
> >> Source[1][2]. The example in the FLIP should be able to be handled in
> this
> >> way.
> >>
> >> Do you mean to support the AyncTableFunction beyond the
> LookupTableSource?
> >> It would be better if you could elaborate the proposed changes wrt the
> >> CorrelatedCodeGenerator with more details. Thanks!
> >>
> >> Best regards,
> >> Jing
> >>
> >> [1]
> >>
> >>
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> >> [2]
> >>
> >>
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> >>
> >> On Sat, May 27, 2023 at 9:48 AM Aitozi  wrote:
> >>
> >> > Hi Jing,
> >> > Thanks for your response. As stated in the FLIP, the purpose of
> this
> >> > FLIP is meant to support
> >> > user-defined async table function. As described in flink document [1]
> >> >
> >> > Async table functions are special functions for table sources that
> >> perform
> >> > > a lookup.
> >> > >
> >> >
> >> > So end user can not directly define and use async table function now.
> An
> >> > user case is reported in [2]
> >> >
> >> > So, in conclusion, no new interface is introduced, but we extend the
> >> > ability to support user-defined async table function.
> >> >
> >> > [1]:
> >> >
> >> >
> >>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> >> > [2]: https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> >> >
> >> > Thanks.
> >> > Aitozi.
> >> >
> >> >
> >> > Jing Ge  于2023年5月27日周六 06:40写道：
> >> >
>

Re: [VOTE] Release flink-connector-pulsar 3.0.1, release candidate #1

[Jing Ge]

+1(non-binding)

- verified sign
- verified hash
- checked repos
- checked tag. NIT: the tag link should be:
https://github.com/apache/flink-connector-pulsar/releases/tag/v3.0.1-rc1
- reviewed PR. NIT: left a comment.

Best regards,
Jing

On Wed, May 31, 2023 at 11:16 PM Neng Lu  wrote:

> +1
>
> I verified
>
> + the release now can communicate with Pulsar using OAuth2 auth plugin
> + build from source and run unit tests with JDK 17 on macOS M1Max
>
>
> On Wed, May 31, 2023 at 4:24 AM Zili Chen  wrote:
>
> > +1
> >
> > I verified
> >
> > + LICENSE and NOTICE present
> > + Checksum and GPG sign matches
> > + No unexpected binaries in the source release
> > + Build from source and run unit tests with JDK 17 on macOS M1
> >
> > On 2023/05/25 16:18:51 Leonard Xu wrote:
> > > Hey all,
> > >
> > > Please review and vote on the release candidate #1 for the version
> 3.0.1
> > of the
> > > Apache Flink Pulsar Connector as follows:
> > >
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > > JIRA release notes [1],
> > > The official Apache source release to be deployed to dist.apache.org
> > [2], which are signed with the key with
> > fingerprint5B2F6608732389AEB67331F5B197E1F1108998AD [3],
> > > All artifacts to be deployed to the Maven Central Repository [4],
> > > Source code tag v3.0.1-rc1 [5],
> > > Website pull request listing the new release [6].
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> > >
> > >
> > > Best,
> > > Leonard
> > >
> > > [1]
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352640
> > > [2]
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-pulsar-3.0.1-rc1/
> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1641/
> > > [5] https://github.com/apache/flink-connector-pulsar/tree/v3.0.1-rc1
> > > [6] https://github.com/apache/flink-web/pull/655
> > >
> > >
> >
>

Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode

[Jing Ge]

Hi Feng,

Thanks for the proposal! Very interesting feature. Would you like to update
your thoughts described in your previous email about why SupportsTimeTravel
has been rejected into the FLIP? This will help readers understand the
context (in the future).

Since we always directly add overload methods into Catalog according to new
requirements, which makes the interface bloated. Just out of curiosity,
does it make sense to introduce some DSL design? Like
Catalog.getTable(tablePath).on(timeStamp),
Catalog.getTable(tablePath).current() for the most current version, and
more room for further extension like timestamp range, etc. I haven't read
all the source code yet and I'm not sure if it is possible. But a
design like this will keep the Catalog API lean and the API/DSL will be
self described and easier to use.

Best regards,
Jing


On Wed, May 31, 2023 at 12:08 PM Krzysztof Chmielewski <
krzysiek.chmielew...@gmail.com> wrote:

> Ok after second though I'm retracting my previous statement about Catalog
> changes you proposed.
> I do see a benefit for Delta connector actually with this change and see
> why this could be coupled with Catalog.
>
> Delta Connector SQL support, also ships a Delta Catalog implementation for
> Flink.
> For Delta Catalog, table schema information is fetched from underlying
> _delta_log and not stored in metastore. For time travel we actually had a
> problem, that if we would like to timetravel back to some old version,
> where schema was slightly different, then we would have a conflict since
> Catalog would return current schema and not how it was for version X.
>
> With your change, our Delta Catalog can actually fetch schema for version X
> and send it to DeltaTableFactory. Currency, Catalog can fetch only current
> version. What we would also need however is version (number/timestamp) for
> this table passed to DynamicTableFactory so we could properly set Delta
> standalone library.
>
> Regards,
> Krzysztof
>
> śr., 31 maj 2023 o 10:37 Krzysztof Chmielewski <
> krzysiek.chmielew...@gmail.com> napisał(a):
>
> > Hi,
> > happy to see such a feature.
> > Small note from my end regarding Catalog changes.
> >
> > TL;DR
> > I don't think it is necessary to delegate this feature to the catalog. I
> > think that since "timetravel" is per job/query property, its should not
> be
> > coupled with the Catalog or table definition. In my opinion this is
> > something that DynamicTableFactory only has to know about. I would rather
> > see this feature as it is - SQL syntax enhancement but delegate clearly
> to
> > DynamicTableFactory.
> >
> > I've implemented timetravel feature for Delta Connector  [1]  using
> > current Flink API.
> > Docs are pending code review, but you can find them here [2] and examples
> > are available here [3]
> >
> > The timetravel feature that I've implemented is based on Flink Query
> > hints.
> > "SELECT * FROM sourceTable /*+ OPTIONS('versionAsOf' = '1') */"
> >
> > The " versionAsOf" (we also have 'timestampAsOf') parameter is handled
> not
> > by Catalog but by DyntamicTableFactory implementation for Delta
> connector.
> > The value of this property is passed to Delta standalone lib API that
> > returns table view for given version.
> >
> > I'm not sure how/if proposed change could benefit Delta connector
> > implementation for this feature.
> >
> > Thanks,
> > Krzysztof
> >
> > [1]
> >
> https://github.com/delta-io/connectors/tree/flink_table_catalog_feature_branch/flink
> > [2] https://github.com/kristoffSC/connectors/tree/FlinkSQL_PR_15-docs
> > [3]
> >
> https://github.com/delta-io/connectors/tree/flink_table_catalog_feature_branch/examples/flink-example/src/main/java/org/example/sql
> >
> > śr., 31 maj 2023 o 06:03 liu ron  napisał(a):
> >
> >> Hi, Feng
> >>
> >> Thanks for driving this FLIP, Time travel is very useful for Flink
> >> integrate with data lake system. I have one question why the
> >> implementation
> >> of TimeTravel is delegated to Catalog? Assuming that we use Flink to
> query
> >> Hudi table with the time travel syntax, but we don't use the
> HudiCatalog,
> >> instead, we register the hudi table to InMemoryCatalog,  can we support
> >> time travel for Hudi table in this case?
> >> In contrast, I think time travel should bind to connector instead of
> >> Catalog, so the rejected alternative should be considered.
> >>
> >> Best,
> >> Ron
> >>
> >> yuxia  于2023年5月30日周二 09:40写道：
> >>
> >> > Hi, Feng.
> >> > Notice this FLIP only support batch mode for time travel.  Would it
> also
> >> > make sense to support stream mode to a read a snapshot of the table
> as a
> >> > bounded stream?
> >> >
> >> > Best regards,
> >> > Yuxia
> >> >
> >> > - 原始邮件 -
> >> > 发件人: "Benchao Li" 
> >> > 收件人: "dev" 
> >> > 发送时间: 星期一, 2023年 5 月 29日 下午 6:04:53
> >> > 主题: Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode
> >> >
> >> > # Can Calcite support this syntax ` VERSION AS OF`  ?
> >> >
> >> > This also depends on whether this is defined in standar

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

[Jing Ge]

e could switch between fast and
> > slow checkpointing intervals based on the information if the job is
> backpressured or not. My thinking is as follows:
> >
> > As a user, I would like to have my regular fast checkpointing interval
> for low latency, but the moment my system is not keeping up, if the
> backpressure
> > builds up, or simply we have a huge backlog to reprocess, latency doesn't
> matter anymore. Only throughput matters. So I would like the checkpointing
> to slow down.
> >
> > I think this should cover pretty well most of the cases, what do you
> think? If this backpressured based behaviour is still not enough, I would
> still say
> > that we should provide plugable checkpoint triggering controllers that
> would work based on metrics.
>
> > change the checkpointing interval based on the "backlog signal",
>
> What's wrong with the job being backpressured? If job is backpressured, we
> don't care about individual records latency, only about increasing
> the throughput to get out of the backpressure situation ASAP.
>
> > In the mentioned use-case, users want to have two different checkpointing
> > intervals at different phases of the HybridSource. We should provide an
> API
> > for users to express the extra checkpointing interval in addition to the
> > existing execution.checkpointing.interval. What would be the definition
> of
> > that API with this alternative approach?
>
> I think my proposal with `BacklogDynamicCheckpointTrigger` or
> `BackpressureDetectingCheckpointTrigger` would solve your motivating use
> case
> just as well.
>
> 1. In the catch up phase (reading the bounded source):
>   a) if we are under backpressure (common case), system would fallback to
> the less frequent checkpointing interval
>   b) if there is no backpressure (I hope a rare case, there is a backlog,
> but the source is too slow), Flink cluster has spare resources to actually
> run more
>   frequent checkpointing interval. No harm should be done. But arguably
> using a less frequent checkpointing interval here should be more desirable.
>
> 2. In the continuous processing phase (unbounded source)
>   a) if we are under backpressure, as I mentioned above, no one cares about
> checkpointing interval and the frequency of committing records to the
>   output, as e2e latency is already high due to the backlog in the
> sources
>   b) if there is no backpressure, that's the only case where the user
> actually cares about the frequency of committing records to the output, we
> are
>   using the more frequent checkpointing interval.
>
> 1b) I think is mostly harmless, and I think could be solved with some extra
> effort
> 2a) and 2b) are not solved by your proposal
> 2a) and 2b) are applicable to any source, not just HybridSource, which is
> also not covered by your proposal.
>
> Best,
> Piotrek
>
>
> czw., 25 maj 2023 o 17:29 Jing Ge  napisał(a):
>
> > Hi Dong, Hi Piotr,
> >
> > Thanks for the clarification.
> >
> > @Dong
> >
> > According to the code examples in the FLIP, I thought we are focusing on
> > the HybridSource scenario. With the current HybridSource implementation,
> we
> > don't even need to know the boundedness of sources in the HybridSource,
> > since all sources except the last one must be bounded[1], i.e. only the
> > last source is unbounded. This makes it much easier to set different
> > intervals to sources with different boundedness.
> >
> > Boundedness in Flink is a top level concept. I think it should be ok to
> > introduce a top level config for the top level concept. I am not familiar
> > with MySQL CDC. For those specific cases, you are right, your proposal
> can
> > provide the feature with minimal changes, like I mentioned previously, it
> > is a thoughtful design.  +1
> >
> > @Piotr
> >
> > > For example join (windowed/temporal) of two tables backed by a hybrid
> > > source? I could easily see a scenario where one table with little data
> > > catches up much more quickly.
> >
> > I am confused. I thought we were talking about HybridSource which "solves
> > the problem of sequentially reading input from heterogeneous sources to
> > produce a single input stream."[2]
> > I could not find any join within a HybridSource. So, your might mean
> > something else the join example and it should be out of the scope, if I
> am
> > not mistaken.
> >
> > > About the (un)boundness of the input stream. I'm not sure if that
> should
> > > actually matter. Actually the same issue, with