[jira] [Created] (FLINK-29826) flink-sql-connector-hbase-1.4 netty class not found, because of shade not work

[xuyong (Jira)]

xuyong created FLINK-29826:
--

 Summary: flink-sql-connector-hbase-1.4 netty class not found, 
because of shade not work
 Key: FLINK-29826
 URL: https://issues.apache.org/jira/browse/FLINK-29826
 Project: Flink
  Issue Type: Bug
  Components: Connectors / HBase
Affects Versions: 1.15.2
 Environment: flink 1.15.1

hbase 1.4
Reporter: xuyong
 Attachments: 2022-11-01 22-55-12屏幕截图-1.png, 2022-11-01 
23-06-41屏幕截图.png, 2022-11-01 23-18-00屏幕截图.png

flink-connectors/flink-sql-connector-hbase-1.4/pom.xml 
,flink-connectors/flink-sql-connector-hbase-2.2/pom.xml shade plugin only 
include io.netty:netty-all artifactId. this jar does not 
contains classes. it noly has META-INF.this cause somebody use 
flink-sql-connector-hbase-1.4 dependency netty classe not found .

I also tried flink-sql-connector-hbase-2.2, it workes perfected. Because 
hbase-client itself shaded netty. So this does not affect this connector.

When I change io.netty:netty-all to 
io.netty:* and repackage this, it workes.

Please fix it, thanks



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29827) [Connector][AsyncSinkWriter] Checkpointed states block writer from sending records

[Hoang Tri Tam (Jira)]

Hoang Tri Tam created FLINK-29827:
-

 Summary: [Connector][AsyncSinkWriter] Checkpointed states block 
writer from sending records
 Key: FLINK-29827
 URL: https://issues.apache.org/jira/browse/FLINK-29827
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Common
Affects Versions: 1.15.2
Reporter: Hoang Tri Tam


Hi every one,

Recently we discovered an issue which blocks Sink operators from sending 
records to client's endpoint.

To *reproduce* the issue, we started our Flink app from an existing savepoint, 
in which some Sink operators hold some buffered records. For instance, app 
employs KinesisStreamSink with a parallelism of 4. 2 of them has no buffered 
records, the other 2 start with existing states of some records, which are 
leftover from the previous run. 

{*}Behavior{*}: during runtime, we sent records (let's say 200) to this sink in 
rebalance mode. But only 100 of them (50%) were dispatched from the sink 
operators.

After {*}investigation{*}, we found that the implementation AsyncSinkWriter 
invokes submitRequestEntries() to send the records to their destination. This 
invocation is performed when a callback is performed, a flush(true) or 
forced-flush is called, or when the buffered is full (either in size or in 
quantity).

The case falls in the first scenario: the _callback is not registered_ {_}when 
the writer starts with some existing buffered records{_}, initialized from 
savepoint. Hence in our case, those operators were holding records till their 
buffers become full, while other operators still perform the usual sending.

Impacted {*}scope{*}: flink-1.15.2 or later version, for any Sink that 
implements AsyncSinkWriter.

We currently treat this as an abnormal behavior of Flink, but please let me 
know if this behavior is intended by design.

Thanks in advance.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Remove FlinkKafkaConsumer and FlinkKafkaProducer in the master for 1.17 release

[Martijn Visser]

Hi all,

@Piotr
When FlinkKafkaConsumer was marked as deprecated, the migration steps were
included in the release notes:
https://nightlies.apache.org/flink/flink-docs-master/release-notes/flink-1.14/#flink-24055httpsissuesapacheorgjirabrowseflink-24055
Scrolling up a bit shows how move from FlinkKafkaProducer to KafkaSink

Given the importance of the Kafka connector, I think it's smart to write a
dedicated blog on how to migrate.
That could also include the differences between the two implementations, as
David just outlined.

@Jing
SinkV2 was released with Flink 1.15, so with 1.16 released there will be 2
release cycles with that API at PublicEvolving. I don't think there were
breaking changes when the bug fixes were applied, because those bug fixes
were also applied in a patch version (1.15.3) which can't be done unless
they weren't breaking the API. Unless there will be work planned for 1.17
on the Sink, I think it can be moved to Public too.

Best regards,

Martijn

On Tue, Nov 1, 2022 at 7:21 AM David Anderson  wrote:

> >
> > [H]ow one can migrate from the
> > FlinkKafkaConsumer/FlinkKafkaProducer to KafkaSource/KafkaSink, while
> > preserving exactly-once guarantees etc?
>
>
> The responses from Fabian Paul in [1] and [2] address the question of how
> to handle the migration in terms of managing the state (where the short
> answer is "arrange for Kafka to be the source of truth").
>
> Those threads don't get into the differences in behavior between the two
> implementations. Here I'm thinking about (1) the loss of
> DeserializationSchema#isEndOfStream, and the fact that you can no longer
> dynamically determine when the input stream has finished, and (2) the
> change to how empty partitions are handled on startup (they used to be
> marked idle automatically, whereas now you must use withIdleness in the
> WatermarkStrategy).
>
> [1] https://www.mail-archive.com/user@flink.apache.org/msg44618.html
> [2] https://www.mail-archive.com/user@flink.apache.org/msg45864.html
>
> On Mon, Oct 31, 2022 at 7:32 PM Piotr Nowojski 
> wrote:
>
> > Hi,
> >
> > Maybe a stupid question, but how one can migrate from the
> > FlinkKafkaConsumer/FlinkKafkaProducer to KafkaSource/KafkaSink, while
> > preserving exactly-once guarantees etc? Is it possible? I've tried a
> quick
> > search and couldn't find it, but maybe I was looking in wrong places.
> >
> > Best,
> > Piotrek
> >
> > pon., 31 paź 2022 o 16:40 Jing Ge  napisał(a):
> >
> > > Thanks Martijn. What you said makes a lot of sense. I figure we should
> do
> > > it in 2 steps.
> > >
> > >  Step 1 (with 1.17):
> > > - Remove FlinkKafkaConsumer.
> > > - Graduate Kafka Source from @PublicEvolving to @Public.
> > > - Update doc and leave hints for customers as the reference.
> > >
> > > According to [1], the Kafka Sink should also be graduated with 1.17,
> i.e.
> > > after 1.15 and 1.16 two release cycles. But since the design change
> from
> > > SinkV1 to SinkV2 were significant and there were many change requests
> > since
> > > then, we'd better give the sink one more release cycle time to become
> > more
> > > stable. The other reason for giving the Sink more time is that the
> > > experimental phase was only covered by one release cycle instead of two
> > as
> > > [1] suggested.
> > >
> > > Step 2 (with 1.18 ):
> > > - Remove FlinkKafkaProducer.
> > > - Graduate Kafka Sink from @PublicEvolving to @Public.
> > > - Update doc and leave hints for customers as the reference.
> > >
> > > Best regards,
> > > Jing
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process
> > >
> > > On Thu, Oct 27, 2022 at 3:01 PM Martijn Visser <
> martijnvis...@apache.org
> > >
> > > wrote:
> > >
> > > > Hi Jing,
> > > >
> > > > Thanks for opening the discussion. I see no issue with removing the
> > > > FlinkKafkaConsumer, since it has been marked as deprecated and the
> > Source
> > > > API (which is used by the KafkaSource) is marked as @Public (at least
> > the
> > > > Base implementation)
> > > >
> > > > The successor of the FlinkKafkaProducer is the KafkaSink, which is
> > using
> > > > the Sink V2 API which is still marked as @PublicEvolving (Base
> > > > implementation). I think that we should only remove the
> > > FlinkKafkaProducer
> > > > if we also mark the Sink V2 as @Public. I don't think that should be
> a
> > > > problem (since it's based on the first Sink implementation, which was
> > > > Experimental in 1.14 and got replaced with Sink V2 as PublicEvolving
> in
> > > > 1.15).
> > > >
> > > > Thanks,
> > > >
> > > > Martijn
> > > >
> > > > On Thu, Oct 27, 2022 at 2:06 PM Jing Ge  wrote:
> > > >
> > > > > Hi Dev,
> > > > >
> > > > > I'd like to start a discussion about removing FlinkKafkaConsumer
> and
> > > > > FlinkKafkaProducer in 1.17.
> > > > >
> > > > > Back in the past, it was originally announced to remove it with
> Flink
> > > > 1.15
> > > > > after Flink 1.14 had been released[1]. And then postp

[jira] [Created] (FLINK-29828) Support to read data from table with given snapshot id

[Shammon (Jira)]

Shammon created FLINK-29828:
---

 Summary: Support to read data from table with given snapshot id
 Key: FLINK-29828
 URL: https://issues.apache.org/jira/browse/FLINK-29828
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Affects Versions: table-store-0.3.0, table-store-0.2.2
Reporter: Shammon


Support reading data from table with given snapshot id from table, users can 
query table data as follows:

`SELECT t1.word, t1.cnt, t2.cnt FROM word_count$snapshot$10 t1 JOIN 
word_count$snapshot$11 t2 ON t1.word = t2.word and t1.cnt != t2.cnt`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29829) align explain results in different platforms

[Yubin Li (Jira)]

Yubin Li created FLINK-29829:


 Summary: align explain results in different platforms
 Key: FLINK-29829
 URL: https://issues.apache.org/jira/browse/FLINK-29829
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: Yubin Li


Delete last line separator during explain results generation in design, but in 
fact, just delete the last character, it will result in results in  the output 
in Windows has one more line than Linux.

 
{code:java}
//Windows
LegacySink(name=[`default_catalog`.`default_database`.`appendSink2`], 
fields=[a, b])
+- GroupWindowAggregate(groupBy=[id1], window=[SlidingGroupWindow('w$, rowtime, 
6000, 12000)], select=[id1, LISTAGG(text, $f3) AS EXPR$1])
   +- Exchange(distribution=[hash[id1]])
      +- Calc(select=[id1, rowtime, text, '*' AS $f3])
         +- Reused(reference_id=[1])


{code}
 

 
{code:java}
//linux
LegacySink(name=[`default_catalog`.`default_database`.`appendSink2`], 
fields=[a, b])
+- GroupWindowAggregate(groupBy=[id1], window=[SlidingGroupWindow('w$, rowtime, 
6000, 12000)], select=[id1, LISTAGG(text, $f3) AS EXPR$1])
   +- Exchange(distribution=[hash[id1]])
      +- Calc(select=[id1, rowtime, text, '*' AS $f3])
         +- Reused(reference_id=[1])
 {code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: SQL Gateway and SQL Client

[Alexey Leonov-Vendrovskiy]

Thanks Shengkai.

>  We don't have any detailed plan about this. But I know Livy has a
similar feature.
How can we check with Livy on this?

Back to the SQL client: is this the master issue tracking enabling SQL
Client <-> Gateway connection?
https://issues.apache.org/jira/browse/FLINK-29751 (Migrate SQL Client Local
Mode to use sql gateway)
It currently shows as unassigned. Is this really the case and no one is
working on enabling the connection from SQL Client to SQL Gateway.
Is the gateway mode planned to be supported for SQL Client in 1.17?

Regards,
Alexey

On Thu, Oct 27, 2022 at 11:07 PM Shengkai Fang  wrote:

> Hi.
>
> > Is there a possibility for us to get engaged and at least introduce
> initial changes to support authentication/authorization?
>
> Yes. You can write a FLIP about the design and change. We can discuss this
> in the dev mail. If the FLIP passes, we can develop it together.
>
> > Another question about persistent Gateway: did you have any specific
> thoughts about it or some draft design?
>
> We don't have any detailed plan about this. But I know Livy has a similar
> feature.
>
> Best,
> Shengkai
>
>
> Alexey Leonov-Vendrovskiy  于2022年10月27日周四 15:12写道：
>
>> Apologies from the delayed response on my side.
>>
>>  I think the authentication module is not part of our plan in 1.17
>>> because of the busy work. I think we'll start the design at the end of the
>>> release-1.17.
>>
>>
>> Is there a possibility for us to get engaged and at least introduce
>> initial changes to support authentication/authorization? Specifically,
>> changes in the API and in SQL Client.
>>
>> We expect the following authentication flow:
>>
>> On the SQL gateway we want to be able to use a delegation token.
>> SQL client should be able to supply an API key.
>> The SQL Gateway *would not *be submitting jobs on behalf of the client.
>>
>> Ideally it would be nice to introduce some interfaces in the SQL Gateway
>> that would allow implementing custom authentication and authorization.
>>
>> Another question about persistent Gateway: did you have any specific
>> thoughts about it or some draft design?
>>
>> Thanks,
>> Alexey
>>
>>
>> On Fri, Oct 21, 2022 at 1:13 AM Shengkai Fang  wrote:
>>
>>> Sorry for the late response.
>>>
>>> In the next version(Flink 1.17), we plan to support the SQL Client to
>>> submit the statement to the Flink SQL Gateway. The FLINK-29486
>>>  is the first step
>>> to remove the usage of the `Parser` in the client side, which needs to read
>>> the table schema during the converting sql node to operation. I think the 
>>> authentication
>>> module is not part of our plan in 1.17 because of the busy work. I think
>>> we'll start the design at the end of the release-1.17.
>>> But could you share more details about the requirements of the
>>> authentication?
>>> - Do you use the kerberos or delegation token or password to do the
>>> authentication?
>>> - After the authentication, do you need the sql gateway to submit the
>>> job on behalf of the client?
>>> - ...
>>>
>>> For detailed implementation, I think Hive and Presto are good
>>> examples to dig in.  If you have some thoughts about the authentication 
>>> module,
>>> please let me know.
>>>
>>> Best,
>>> Shengkai
>>>
>>> Alexey Leonov-Vendrovskiy  于2022年10月19日周三 00:37写道：
>>>
 Thank you for the response, Yuxia!

 Shengkai, I would like to learn more about nearest and a bit more
 distant plans about development of the SQL Gateway and the SQL Client.
 Do you have a description of the work planned or maybe can share
 general thoughts about the Authentication module, or Persistent Gateway.
 How can the authentication part be addressed on the SQL Client side?

 Regards,
 -Alexey


 On Wed, Oct 12, 2022 at 11:24 PM yuxia 
 wrote:

> > In what Flink’s release the connection from SQL Client to the
> Gateway is
> expected to be added?
> Flink 1.17
>
> > “Authentication module” (2) and “Persistent Gateway” (4) as
> possible future work. Were there any recent discussions on these
> subjects?
> No recent discussions on these subjects, but I think it'll come in
> Flink 1.17
>
> > Another related topic: are there ideas around making SQL Gateway a
> multi-tenant
> component?
> Yes.
>
> Shengkaiis the maintainer of SQL Client and SQL gateway, maybe he can
> provide more information.
>
>
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "Alexey Leonov-Vendrovskiy" 
> 收件人: "dev" 
> 发送时间: 星期四, 2022年 10 月 13日 下午 12:33:08
> 主题: SQL Gateway and SQL Client
>
> Hi all,
>
> I’m Alexey from Confluent. This is my first email in this discussion
> list.
> I’m rather new to Flink, and to local customs of communication. I want
> to
> dive deeper and hopefully get more involved over time.
>
> Currently I h

[jira] [Created] (FLINK-29830) PulsarSinkITCase$DeliveryGuaranteeTest.writeRecordsToPulsar failed

[Martijn Visser (Jira)]

Martijn Visser created FLINK-29830:
--

 Summary: 
PulsarSinkITCase$DeliveryGuaranteeTest.writeRecordsToPulsar failed
 Key: FLINK-29830
 URL: https://issues.apache.org/jira/browse/FLINK-29830
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Pulsar
Affects Versions: 1.17.0
Reporter: Martijn Visser


{code:java}
Nov 01 01:28:03 [ERROR] Failures: 
Nov 01 01:28:03 [ERROR]   
PulsarSinkITCase$DeliveryGuaranteeTest.writeRecordsToPulsar:140 
Nov 01 01:28:03 Actual and expected should have same size but actual size is:
Nov 01 01:28:03   0
Nov 01 01:28:03 while expected size is:
Nov 01 01:28:03   115
Nov 01 01:28:03 Actual was:
Nov 01 01:28:03   []
Nov 01 01:28:03 Expected was:
Nov 01 01:28:03   ["AT_LEAST_ONCE-isxrFGAL-0-kO65unDUKX",
Nov 01 01:28:03 "AT_LEAST_ONCE-isxrFGAL-1-4tBNu1UmeR",
Nov 01 01:28:03 "AT_LEAST_ONCE-isxrFGAL-2-9PTnEahlNU",
Nov 01 01:28:03 "AT_LEAST_ONCE-isxrFGAL-3-GjWqEp21yz",
Nov 01 01:28:03 "AT_LEAST_ONCE-isxrFGAL-4-jnbJr9C0w8",
Nov 01 01:28:03 "AT_LEAST_ONCE-isxrFGAL-5-e8Wacz5yDO",
Nov 01 01:28:03 "AT_LEAST_ONCE-isxrFGAL-6-9cW53j3Zcf",
Nov 01 01:28:03 "AT_LEAST_ONCE-isxrFGAL-7-jk8z3m2Aa5",
Nov 01 01:28:03 "AT_LEAST_ONCE-isxrFGAL-8-VU56KmMeiz",
Nov 01 01:28:03 "AT_LEAST_ONCE-isxrFGAL-9-uvMdFxxDAj",
Nov 01 01:28:03 "AT_LEAST_ONCE-isxrFGAL-10-FQyWfwJFbH",
...
{code}

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42680&view=logs&j=aa18c3f6-13b8-5f58-86bb-c1cffb239496&t=502fb6c0-30a2-5e49-c5c2-a00fa3acb203&l=37544



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29831) Multiple Hive tests are failing

[Martijn Visser (Jira)]

Martijn Visser created FLINK-29831:
--

 Summary: Multiple Hive tests are failing
 Key: FLINK-29831
 URL: https://issues.apache.org/jira/browse/FLINK-29831
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Affects Versions: 1.17.0
Reporter: Martijn Visser


{code:java}
Nov 01 01:56:03 [ERROR] 
org.apache.flink.table.module.hive.HiveModuleTest.testNumberOfBuiltinFunctions  
Time elapsed: 0.042 s  <<< FAILURE!

Nov 01 01:57:17 [ERROR] 
org.apache.flink.connectors.hive.HiveTableSourceStatisticsReportTest.testFlinkOrcFormatHiveTableSourceStatisticsReport
  Time elapsed: 12.846 s  <<< FAILURE!

Nov 01 01:57:17 [ERROR] 
org.apache.flink.connectors.hive.HiveTableSourceStatisticsReportTest.testMapRedOrcFormatHiveTableSourceStatisticsReport
  Time elapsed: 10.355 s  <<< FAILURE!

Nov 01 01:57:17 [ERROR]   
HiveTableSourceStatisticsReportTest.testFlinkOrcFormatHiveTableSourceStatisticsReport:124->assertHiveTableOrcFormatTableStatsEquals:368
 

Nov 01 01:57:17 [ERROR]   
HiveTableSourceStatisticsReportTest.testMapRedOrcFormatHiveTableSourceStatisticsReport:164->assertHiveTableOrcFormatTableStatsEquals:368
 

Nov 01 01:57:17 [ERROR]   
HiveModuleTest.testNumberOfBuiltinFunctions:53->verifyNumBuiltInFunctions:75 
{code}

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42680&view=logs&j=245e1f2e-ba5b-5570-d689-25ae21e5302f&t=d04c9862-880c-52f5-574b-a7a79fef8e0f&l=24818



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

[Guowei Ma]

Congratulations Danny!
Best,
Guowei


On Tue, Nov 1, 2022 at 2:20 PM weijie guo  wrote:

> Congratulations Danny!
>
> Best regards,
>
> Weijie
>
>
> Maximilian Michels  于2022年10月13日周四 21:41写道：
>
> > Congratulations Danny! Well deserved :)
> >
> > -Max
> >
> > On Thu, Oct 13, 2022 at 2:40 PM Yang Wang  wrote:
> >
> > > Congratulations Danny!
> > >
> > > Best,
> > > Yang
> > >
> > > Hang Ruan  于2022年10月13日周四 10:58写道：
> > >
> > > > Congratulations Danny!
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > Yun Gao  于2022年10月13日周四 10:56写道：
> > > >
> > > > > Congratulations Danny!
> > > > > Best,
> > > > > Yun Gao
> > > > > --
> > > > > From:yuxia 
> > > > > Send Time:2022 Oct. 12 (Wed.) 09:49
> > > > > To:dev 
> > > > > Subject:Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer
> > > > > Congratulations Danny!
> > > > > Best regards,
> > > > > Yuxia
> > > > > - 原始邮件 -
> > > > > 发件人: "Xingbo Huang" 
> > > > > 收件人: "dev" 
> > > > > 发送时间: 星期三, 2022年 10 月 12日 上午 9:44:22
> > > > > 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer
> > > > > Congratulations Danny!
> > > > > Best,
> > > > > Xingbo
> > > > > Sergey Nuyanzin  于2022年10月12日周三 01:26写道：
> > > > > > Congratulations, Danny
> > > > > >
> > > > > > On Tue, Oct 11, 2022, 15:18 Lincoln Lee 
> > > > wrote:
> > > > > >
> > > > > > > Congratulations Danny!
> > > > > > >
> > > > > > > Best,
> > > > > > > Lincoln Lee
> > > > > > >
> > > > > > >
> > > > > > > Congxian Qiu  于2022年10月11日周二 19:42写道：
> > > > > > >
> > > > > > > > Congratulations Danny!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Congxian
> > > > > > > >
> > > > > > > >
> > > > > > > > Leonard Xu  于2022年10月11日周二 18:03写道：
> > > > > > > >
> > > > > > > > > Congratulations Danny!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Leonard
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-29832) Refactor default database name in flink table store

[Shammon (Jira)]

Shammon created FLINK-29832:
---

 Summary: Refactor default database name in flink table store
 Key: FLINK-29832
 URL: https://issues.apache.org/jira/browse/FLINK-29832
 Project: Flink
  Issue Type: Bug
  Components: Table Store
Affects Versions: table-store-0.3.0, table-store-0.2.2
Reporter: Shammon
 Attachments: image-2022-11-01-16-40-47-539.png

`FlinkCatalogFactory` creates a default database named `default` in table 
store. The `default` is a keyword in SQL, and when we create a new database, we 
cant execute `use default` to switch to `default` database any more. We should 
refactor the default database name in table store.
 !image-2022-11-01-16-40-47-539.png|thumbnail! 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29833) Improve PyFlink support in Windows

[Huang Xingbo (Jira)]

Huang Xingbo created FLINK-29833:


 Summary: Improve PyFlink support in Windows
 Key: FLINK-29833
 URL: https://issues.apache.org/jira/browse/FLINK-29833
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Affects Versions: 1.15.2, 1.16.0
Reporter: Huang Xingbo
Assignee: Huang Xingbo


Many users are used to developing PyFlink jobs on Windows. It is necessary to 
improve the simplicity of PyFlink job development on Windows



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Remove FlinkKafkaConsumer and FlinkKafkaProducer in the master for 1.17 release

[Piotr Nowojski]

Thanks Martijn and David,

And what about the FlinkKafkaProducer? In order to migrate it, the user has
to make sure that the transactions stored on the state are committed, and
that there are no lingering/pending transactions (lingering transactions
are blocking further reads to be readable until they timeout). The first
one can be achieved by stop-with-savepoint. Regular savepoint or checkpoint
is not good enough, because transactions are committed on
notfiyCheckpointCompleted RPC, not on the checkpoint itself
(stop-with-savepoint is considered completed only once notifications are
received).  But what about the latter? What about the pending/lingering
transactions? Are they also closed in stop-with-savepoint?

Best,
Piotrek

wt., 1 lis 2022 o 08:51 Martijn Visser 
napisał(a):

> Hi all,
>
> @Piotr
> When FlinkKafkaConsumer was marked as deprecated, the migration steps were
> included in the release notes:
>
> https://nightlies.apache.org/flink/flink-docs-master/release-notes/flink-1.14/#flink-24055httpsissuesapacheorgjirabrowseflink-24055
> Scrolling up a bit shows how move from FlinkKafkaProducer to KafkaSink
>
> Given the importance of the Kafka connector, I think it's smart to write a
> dedicated blog on how to migrate.
> That could also include the differences between the two implementations, as
> David just outlined.
>
> @Jing
> SinkV2 was released with Flink 1.15, so with 1.16 released there will be 2
> release cycles with that API at PublicEvolving. I don't think there were
> breaking changes when the bug fixes were applied, because those bug fixes
> were also applied in a patch version (1.15.3) which can't be done unless
> they weren't breaking the API. Unless there will be work planned for 1.17
> on the Sink, I think it can be moved to Public too.
>
> Best regards,
>
> Martijn
>
> On Tue, Nov 1, 2022 at 7:21 AM David Anderson 
> wrote:
>
> > >
> > > [H]ow one can migrate from the
> > > FlinkKafkaConsumer/FlinkKafkaProducer to KafkaSource/KafkaSink, while
> > > preserving exactly-once guarantees etc?
> >
> >
> > The responses from Fabian Paul in [1] and [2] address the question of how
> > to handle the migration in terms of managing the state (where the short
> > answer is "arrange for Kafka to be the source of truth").
> >
> > Those threads don't get into the differences in behavior between the two
> > implementations. Here I'm thinking about (1) the loss of
> > DeserializationSchema#isEndOfStream, and the fact that you can no longer
> > dynamically determine when the input stream has finished, and (2) the
> > change to how empty partitions are handled on startup (they used to be
> > marked idle automatically, whereas now you must use withIdleness in the
> > WatermarkStrategy).
> >
> > [1] https://www.mail-archive.com/user@flink.apache.org/msg44618.html
> > [2] https://www.mail-archive.com/user@flink.apache.org/msg45864.html
> >
> > On Mon, Oct 31, 2022 at 7:32 PM Piotr Nowojski 
> > wrote:
> >
> > > Hi,
> > >
> > > Maybe a stupid question, but how one can migrate from the
> > > FlinkKafkaConsumer/FlinkKafkaProducer to KafkaSource/KafkaSink, while
> > > preserving exactly-once guarantees etc? Is it possible? I've tried a
> > quick
> > > search and couldn't find it, but maybe I was looking in wrong places.
> > >
> > > Best,
> > > Piotrek
> > >
> > > pon., 31 paź 2022 o 16:40 Jing Ge  napisał(a):
> > >
> > > > Thanks Martijn. What you said makes a lot of sense. I figure we
> should
> > do
> > > > it in 2 steps.
> > > >
> > > >  Step 1 (with 1.17):
> > > > - Remove FlinkKafkaConsumer.
> > > > - Graduate Kafka Source from @PublicEvolving to @Public.
> > > > - Update doc and leave hints for customers as the reference.
> > > >
> > > > According to [1], the Kafka Sink should also be graduated with 1.17,
> > i.e.
> > > > after 1.15 and 1.16 two release cycles. But since the design change
> > from
> > > > SinkV1 to SinkV2 were significant and there were many change requests
> > > since
> > > > then, we'd better give the sink one more release cycle time to become
> > > more
> > > > stable. The other reason for giving the Sink more time is that the
> > > > experimental phase was only covered by one release cycle instead of
> two
> > > as
> > > > [1] suggested.
> > > >
> > > > Step 2 (with 1.18 ):
> > > > - Remove FlinkKafkaProducer.
> > > > - Graduate Kafka Sink from @PublicEvolving to @Public.
> > > > - Update doc and leave hints for customers as the reference.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process
> > > >
> > > > On Thu, Oct 27, 2022 at 3:01 PM Martijn Visser <
> > martijnvis...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Hi Jing,
> > > > >
> > > > > Thanks for opening the discussion. I see no issue with removing the
> > > > > FlinkKafkaConsumer, since it has been marked as deprecated and the
> > > Source
> > > > > API (which is used by the KafkaSourc

[jira] [Created] (FLINK-29834) Clear static Jackson TypeFactory cache on CL release

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-29834:


 Summary: Clear static Jackson TypeFactory cache on CL release
 Key: FLINK-29834
 URL: https://issues.apache.org/jira/browse/FLINK-29834
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.17.0, 1.16.1


The Jackson TypeFactory contains a singleton instance that is at times used by 
Jackson, potentially containing user-classes for longer than necessary.

https://github.com/FasterXML/jackson-databind/issues/1363
BEAM-6460

We could clear this cache whenever a user code CL is being released.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[SUMMARY] Flink 1.17 Release Sync 11/1/2022

[Qingsheng Ren]

Hi devs and users,

I'd like to share some highlights about the 1.17 release sync on 11/1/2022.

- The release sync is scheduled for biweekly Tuesdays at 9am (Central
European Standard Time) / 4pm (China Standard Time).

- Retrospective for 1.16: a discussion [1] is opened by Matthias for
collecting feedback about the last release cycle.

- Release tracking page: @committers please update the page [2] in the
coming week

- CI stabilities / blockers: PR for FLINK-24119 is opened and waiting for a
green CI and review.

- Martijn is working on a handbook for release managers, which could also
benefit RMs in future releases.

- Tickets for new contributors: we encourage committers to add a "starter"
label in easy-to-fix tickets as described in Flink Jira Process [3], which
is helpful for new contributors to find the correct path.

The next release sync will be on November 15th, 2022.

Google Meet: https://meet.google.com/wcx-fjbt-hhz
Dial-in: https://tel.meet/wcx-fjbt-hhz?pin=1940846765126

Best regards,

Leonard, Matthias, Martijn and Qingsheng

[1] https://lists.apache.org/thread/ypt6m3qm6rh5jvhg53jvm1qpyy58mpxp
[2] https://cwiki.apache.org/confluence/display/FLINK/1.17+Release
[3] https://cwiki.apache.org/confluence/display/FLINK/Flink+Jira+Process

[jira] [Created] (FLINK-29835) BlobServer fails to provide artifacts

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-29835:
-

 Summary: BlobServer fails to provide artifacts
 Key: FLINK-29835
 URL: https://issues.apache.org/jira/browse/FLINK-29835
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.16.0, 1.17.0
Reporter: Matthias Pohl
 Attachments: 
PulsarSourceUnorderedE2ECase.testSavepoint.FileNotFoundException.log

[This 
build|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42680&view=logs&j=87489130-75dc-54e4-1f45-80c30aa367a3&t=73da6d75-f30d-5d5a-acbe-487a9dcff678&l=16001]
 failed in \{{PulsarSourceUnorderedE2ECase.testSavepoint}} due to some job 
timeout. The issue seems to be related to the BlobServer failing to provide 
some artifacts (\{{java.io.FileNotFoundException}}) that consequently causes 
classes not being found:
{code:java}
java.lang.NoClassDefFoundError: 
org/apache/pulsar/shade/org/jvnet/hk2/internal/ServiceLocatorImpl$7
      at 
org.apache.pulsar.shade.org.jvnet.hk2.internal.ServiceLocatorImpl.shutdown(ServiceLocatorImpl.java:894)
 ~[?:?]
      at 
org.apache.pulsar.shade.org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.shutdown(AbstractHk2InjectionManager.java:183)
 ~[?:?]
      at 
org.apache.pulsar.shade.org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.shutdown(ImmediateHk2InjectionManager.java:30)
 ~[?:?]
      at 
org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.close(ClientRuntime.java:371)
 ~[?:?]
      at 
org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.onShutdown(ClientRuntime.java:353)
 ~[?:?]
      at 
org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyClient.release(JerseyClient.java:190)
 ~[?:?]
      at 
org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyClient.close(JerseyClient.java:180)
 ~[?:?]
      at 
org.apache.pulsar.client.admin.internal.PulsarAdminImpl.close(PulsarAdminImpl.java:488)
 ~[?:?]
      at 
org.apache.flink.connector.pulsar.source.enumerator.PulsarSourceEnumerator.close(PulsarSourceEnumerator.java:164)
 ~[?:?]
      at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:255) 
~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
      at 
org.apache.flink.runtime.source.coordinator.SourceCoordinator.close(SourceCoordinator.java:265)
 ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
      at 
org.apache.flink.runtime.operators.coordination.ComponentClosingUtils.lambda$closeAsyncWithTimeout$0(ComponentClosingUtils.java:76)
 ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
      at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
Caused by: java.lang.ClassNotFoundException: 
org.apache.pulsar.shade.org.jvnet.hk2.internal.ServiceLocatorImpl$7
      at java.net.URLClassLoader.findClass(URLClassLoader.java:387) 
~[?:1.8.0_342]
      at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_342]
      at 
org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67)
 ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
      at 
org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
 ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
      at 
org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51)
 ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
      at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_342]
      ... 13 more{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29836) PulsarUnorderedSourceITCase failed due to an expected job restart not happening

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-29836:
-

 Summary: PulsarUnorderedSourceITCase failed due to an expected job 
restart not happening
 Key: FLINK-29836
 URL: https://issues.apache.org/jira/browse/FLINK-29836
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Pulsar
Affects Versions: 1.16.0, 1.17.0
Reporter: Matthias Pohl


[This build 
failed|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42681&view=logs&j=a5ef94ef-68c2-57fd-3794-dc108ed1c495&t=2c68b137-b01d-55c9-e603-3ff3f320364b&l=27377]
 due to a missing job restart:
{code:java}
 Nov 01 02:04:20 Expecting
Nov 01 02:04:20   
Nov 01 02:04:20 to be completed within 2M.
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

[Yuxin Tan]

Congrats, Danny!

Best,
Yuxin


Guowei Ma  于2022年11月1日周二 16:36写道：

> Congratulations Danny!
> Best,
> Guowei
>
>
> On Tue, Nov 1, 2022 at 2:20 PM weijie guo 
> wrote:
>
> > Congratulations Danny!
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Maximilian Michels  于2022年10月13日周四 21:41写道：
> >
> > > Congratulations Danny! Well deserved :)
> > >
> > > -Max
> > >
> > > On Thu, Oct 13, 2022 at 2:40 PM Yang Wang 
> wrote:
> > >
> > > > Congratulations Danny!
> > > >
> > > > Best,
> > > > Yang
> > > >
> > > > Hang Ruan  于2022年10月13日周四 10:58写道：
> > > >
> > > > > Congratulations Danny!
> > > > >
> > > > > Best,
> > > > > Hang
> > > > >
> > > > > Yun Gao  于2022年10月13日周四 10:56写道：
> > > > >
> > > > > > Congratulations Danny!
> > > > > > Best,
> > > > > > Yun Gao
> > > > > >
> --
> > > > > > From:yuxia 
> > > > > > Send Time:2022 Oct. 12 (Wed.) 09:49
> > > > > > To:dev 
> > > > > > Subject:Re: [ANNOUNCE] New Apache Flink PMC Member - Danny
> Cranmer
> > > > > > Congratulations Danny!
> > > > > > Best regards,
> > > > > > Yuxia
> > > > > > - 原始邮件 -
> > > > > > 发件人: "Xingbo Huang" 
> > > > > > 收件人: "dev" 
> > > > > > 发送时间: 星期三, 2022年 10 月 12日 上午 9:44:22
> > > > > > 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer
> > > > > > Congratulations Danny!
> > > > > > Best,
> > > > > > Xingbo
> > > > > > Sergey Nuyanzin  于2022年10月12日周三 01:26写道：
> > > > > > > Congratulations, Danny
> > > > > > >
> > > > > > > On Tue, Oct 11, 2022, 15:18 Lincoln Lee <
> lincoln.8...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Congratulations Danny!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Lincoln Lee
> > > > > > > >
> > > > > > > >
> > > > > > > > Congxian Qiu  于2022年10月11日周二
> 19:42写道：
> > > > > > > >
> > > > > > > > > Congratulations Danny!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Congxian
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Leonard Xu  于2022年10月11日周二 18:03写道：
> > > > > > > > >
> > > > > > > > > > Congratulations Danny!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Delete useless branches from Flink repository

[Leonard Xu]

Thanks Yun Tang for joining the discussion.

I’ve deleted these branches with same name tag except  release-1.14.6-rc2 which 
has no same name tag.

 release-1.12.3-rc1
  release-1.13.2-rc2
  release-1.14.3-rc1
  release-1.15.0-rc1
  release-1.14.6-rc2

Best,
Leonard



> 
> Hi Leonard,
> 
> Thanks for driving this topic, I think the release candidate branch after 
> flink-1.10 could be deleted.
> 
> I believe those branches are created by mistake as the release manager should 
> use tags instead of branch to hold the release candidate.
> Those branches are:
>release-1.12.3-rc1
>release-1.13.2-rc2
>release-1.14.3-rc1
>release-1.14.6-rc2
>release-1.15.0-rc1
> 
> 
> Best
> Yun Tang
> 
> From: Leonard Xu 
> Sent: Monday, October 31, 2022 20:07
> To: dev@flink.apache.org 
> Subject: Re: [DISCUSS] Delete useless branches from Flink repository
> 
> 
>> Actually I double checked it and the default branch on jenkins is `master`.
> 
> Great, I’ve deleted benchmark-request branch. Now, our repo looks better.
> 
> Best,
> Leonard

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

[Hamdy, Ahmed]

Congratulations, Danny!
Best,
Ahmed

On 01/11/2022, 09:51, "Yuxin Tan"  wrote:

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Congrats, Danny!

Best,
Yuxin


Guowei Ma  于2022年11月1日周二 16:36写道：

> Congratulations Danny!
> Best,
> Guowei
>
>
> On Tue, Nov 1, 2022 at 2:20 PM weijie guo 
> wrote:
>
> > Congratulations Danny!
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Maximilian Michels  于2022年10月13日周四 21:41写道：
> >
> > > Congratulations Danny! Well deserved :)
> > >
> > > -Max
> > >
> > > On Thu, Oct 13, 2022 at 2:40 PM Yang Wang 
> wrote:
> > >
> > > > Congratulations Danny!
> > > >
> > > > Best,
> > > > Yang
> > > >
> > > > Hang Ruan  于2022年10月13日周四 10:58写道：
> > > >
> > > > > Congratulations Danny!
> > > > >
> > > > > Best,
> > > > > Hang
> > > > >
> > > > > Yun Gao  于2022年10月13日周四 10:56写道：
> > > > >
> > > > > > Congratulations Danny!
> > > > > > Best,
> > > > > > Yun Gao
> > > > > >
> --
> > > > > > From:yuxia 
> > > > > > Send Time:2022 Oct. 12 (Wed.) 09:49
> > > > > > To:dev 
> > > > > > Subject:Re: [ANNOUNCE] New Apache Flink PMC Member - Danny
> Cranmer
> > > > > > Congratulations Danny!
> > > > > > Best regards,
> > > > > > Yuxia
> > > > > > - 原始邮件 -
> > > > > > 发件人: "Xingbo Huang" 
> > > > > > 收件人: "dev" 
> > > > > > 发送时间: 星期三, 2022年 10 月 12日 上午 9:44:22
> > > > > > 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer
> > > > > > Congratulations Danny!
> > > > > > Best,
> > > > > > Xingbo
> > > > > > Sergey Nuyanzin  于2022年10月12日周三 01:26写道：
> > > > > > > Congratulations, Danny
> > > > > > >
> > > > > > > On Tue, Oct 11, 2022, 15:18 Lincoln Lee <
> lincoln.8...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Congratulations Danny!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Lincoln Lee
> > > > > > > >
> > > > > > > >
> > > > > > > > Congxian Qiu  于2022年10月11日周二
> 19:42写道：
> > > > > > > >
> > > > > > > > > Congratulations Danny!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Congxian
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Leonard Xu  于2022年10月11日周二 18:03写道：
> > > > > > > > >
> > > > > > > > > > Congratulations Danny!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-264 Extract BaseCoordinatorContext

[Maximilian Michels]

Thanks Steven! My confusion stemmed from the lack of context in the FLIP.
The first version did not lay out how the refactoring would be used down
the line, e.g. by the ShuffleCoordinator. The OperatorCoordinator API is a
non-public API and before reading the code, I wasn't even aware how exactly
it worked and whether it would be available to regular operators (it was
originally intended for sources only).

I might seem pedantic here but I believe the purpose of a FLIP should be to
describe the *why* behind the changes, not only the changes itself. A FLIP
is not a formality but is a tool to communicate and discuss changes. I
think we still haven't laid out the exact reasons why we are factoring out
the base. As far as I understand now, we need the base class to deal with
concurrent updates in the custom Coordinator from the runtime (sub)tasks.
Effectively, we are enforcing an actor model for the processing of the
incoming messages such that the OperatorCoordinator can cleanly update its
state. However, if there are no actual implementations that make use of the
refactoring in Flink itself, I wonder if it would make sense to copy this
code to the downstream implementation, e.g. the ShuffleCoordinator. As soon
as it is part of Flink, we could of course try to consolidate this code.

Considering the *how* of this, there appear to be both methods from
SourceCoordinator (e.g. runInEventLoop) as well as SourceCoordinatorContext
listed in the FLIP, as well as methods which do not appear anywhere in
Flink code, e.g. subTaskReady / subTaskNotReady / sendEventToOperator. It
appears that some of this has been extracted from a downstream
implementation. It would be great to adjust this, such that it reflects the
status quo in Flink.

-Max

On Fri, Oct 28, 2022 at 5:53 AM Steven Wu  wrote:

> Max,
>
> Thanks a lot for the comments. We should clarify that the shuffle
> operator/coordinator is not really part of the Flink sink
> function/operator. shuffle operator is a custom operator that can be
> inserted right before the Iceberg writer operator. Shuffle operator
> calculates the traffic statistics and performs a custom partition/shuffle
> (DataStream#partitionCustom) to cluster the data right before they get to
> the Iceberg writer operator.
>
> We are not proposing to introduce a sink coordinator for the sink
> interface. Shuffle operator needs the CoordinatorContextBase to
> facilitate the communication btw shuffle subtasks and coordinator for
> traffic statistics aggregation. The communication part is already
> implemented by SourceCoordinatorContext.
>
> Here are some details about the communication needs.
> - subtasks periodically calculate local statistics and send to the
> coordinator for global aggregation
> - the coordinator sends the globally aggregated statistics to the subtasks
> - subtasks use the globally aggregated statistics to guide the
> partition/shuffle decision
>
> Regards,
> Steven
>
> On Thu, Oct 27, 2022 at 5:38 PM Maximilian Michels  wrote:
>
> > Hi Gang,
> >
> > Looks much better! I've actually gone through the OperatorCoordinator
> code.
> > It turns out, any operator already has an OperatorCoordinator assigned.
> > Also, any operator can add custom coordinator code. So it looks like you
> > won't have to implement any additional runtime logic to add a
> > ShuffleCoordinator. However, I'm wondering, why do you specifically need
> to
> > refactor the SourceCoordinatorContext? You could simply add your own
> > coordinator code. I'm not sure the sink requirements map to the source
> > interface so closely that you can reuse the same logic.
> >
> > If you can refactor SourceCoordinatorContext in a way that makes it fit
> > your use case, I have nothing to object here. By the way, another example
> > of an existing OperatorCoordinator is CollectSinkOperatorCoordinator
> which
> > is quite trivial but it might be worth evaluating whether you need the
> full
> > power of SourceCoordinatorContext which is why I wanted to get more
> > context.
> >
> > -Max
> >
> > On Thu, Oct 27, 2022 at 4:15 PM gang ye  wrote:
> >
> > > Hi Max,
> > > I got your concern. Since shuffling support for Flink Iceberg sink is
> not
> > > the main body of the proposal, I add another appendix part just now
> with
> > > more details about how to use CoordinatorContextBase and how to define
> > > ShufflingCoordinator.
> > >
> > > Let me know if that cannot solve your concern.
> > >
> > > Thanks
> > > Gang
> > >
> > > On Thu, Oct 27, 2022 at 1:31 PM Maximilian Michels 
> > wrote:
> > >
> > >> Hey Gang,
> > >>
> > >> What I'm looking for here is a complete picture of why the change is
> > >> necessary and what the next steps are. Ultimately, refactoring any
> code
> > >> serves a purpose. Here, we want to refactor the Coordinator code such
> > that
> > >> we can add a SinkCoordinator, similar to the SourceCoordinator. The
> FLIP
> > >> should address the next steps, i.e. how you plan to add the
> > >> SinkCoordinator, its interf

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

[Jing Ge]

Congrats, Danny!

Best regards,
Jing

On Tue, Nov 1, 2022 at 11:42 AM Hamdy, Ahmed 
wrote:

> Congratulations, Danny!
> Best,
> Ahmed
>
> On 01/11/2022, 09:51, "Yuxin Tan"  wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> Congrats, Danny!
>
> Best,
> Yuxin
>
>
> Guowei Ma  于2022年11月1日周二 16:36写道：
>
> > Congratulations Danny!
> > Best,
> > Guowei
> >
> >
> > On Tue, Nov 1, 2022 at 2:20 PM weijie guo  >
> > wrote:
> >
> > > Congratulations Danny!
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > Maximilian Michels  于2022年10月13日周四 21:41写道：
> > >
> > > > Congratulations Danny! Well deserved :)
> > > >
> > > > -Max
> > > >
> > > > On Thu, Oct 13, 2022 at 2:40 PM Yang Wang  >
> > wrote:
> > > >
> > > > > Congratulations Danny!
> > > > >
> > > > > Best,
> > > > > Yang
> > > > >
> > > > > Hang Ruan  于2022年10月13日周四 10:58写道：
> > > > >
> > > > > > Congratulations Danny!
> > > > > >
> > > > > > Best,
> > > > > > Hang
> > > > > >
> > > > > > Yun Gao  于2022年10月13日周四
> 10:56写道：
> > > > > >
> > > > > > > Congratulations Danny!
> > > > > > > Best,
> > > > > > > Yun Gao
> > > > > > >
> > --
> > > > > > > From:yuxia 
> > > > > > > Send Time:2022 Oct. 12 (Wed.) 09:49
> > > > > > > To:dev 
> > > > > > > Subject:Re: [ANNOUNCE] New Apache Flink PMC Member - Danny
> > Cranmer
> > > > > > > Congratulations Danny!
> > > > > > > Best regards,
> > > > > > > Yuxia
> > > > > > > - 原始邮件 -
> > > > > > > 发件人: "Xingbo Huang" 
> > > > > > > 收件人: "dev" 
> > > > > > > 发送时间: 星期三, 2022年 10 月 12日 上午 9:44:22
> > > > > > > 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Danny
> Cranmer
> > > > > > > Congratulations Danny!
> > > > > > > Best,
> > > > > > > Xingbo
> > > > > > > Sergey Nuyanzin  于2022年10月12日周三
> 01:26写道：
> > > > > > > > Congratulations, Danny
> > > > > > > >
> > > > > > > > On Tue, Oct 11, 2022, 15:18 Lincoln Lee <
> > lincoln.8...@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations Danny!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Lincoln Lee
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Congxian Qiu  于2022年10月11日周二
> > 19:42写道：
> > > > > > > > >
> > > > > > > > > > Congratulations Danny!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Congxian
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Leonard Xu  于2022年10月11日周二
> 18:03写道：
> > > > > > > > > >
> > > > > > > > > > > Congratulations Danny!
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Leonard
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>

Re: [DISCUSS] Remove FlinkKafkaConsumer and FlinkKafkaProducer in the master for 1.17 release

[Jing Ge]

Hi all,

I am totally fine to do step 2 with 1.17. It would give us more room to do
some breaking changes if we would do it with 1.18.

@Danny

Thanks for the hint. Commonly, all related interfaces/classes should be
graduated too. Classes like SingleThreadMultiplexSourceReaderBase and
RecordEmitter used within SourceReaderBase are a little bit different.
Because  SingleThreadMultiplexSourceReaderBase is extended by
KafkaSourceReader which is marked as @Internal. It is not necessary to
graduate them just because the KafkaSource needs to be graduated. However,
given that most of those classes have been marked as @PublicEvolving more
than one years, it is a good opportunity to graduate those low level APIs
along with KafkaSource. And this will make the change a little bit bigger
than the original expectation of removing FlinkKafkaConsumer,
FlinkKafkaProducer and only graduating the related KafkaSource, KafkaSink.
I am struggling if It makes sense to have a FLIP for it, because the API
itself will not be modified except upgrading the annotation for graduation.

Best regards,
Jing


On Tue, Nov 1, 2022 at 9:52 AM Piotr Nowojski  wrote:

> Thanks Martijn and David,
>
> And what about the FlinkKafkaProducer? In order to migrate it, the user has
> to make sure that the transactions stored on the state are committed, and
> that there are no lingering/pending transactions (lingering transactions
> are blocking further reads to be readable until they timeout). The first
> one can be achieved by stop-with-savepoint. Regular savepoint or checkpoint
> is not good enough, because transactions are committed on
> notfiyCheckpointCompleted RPC, not on the checkpoint itself
> (stop-with-savepoint is considered completed only once notifications are
> received).  But what about the latter? What about the pending/lingering
> transactions? Are they also closed in stop-with-savepoint?
>
> Best,
> Piotrek
>
> wt., 1 lis 2022 o 08:51 Martijn Visser 
> napisał(a):
>
> > Hi all,
> >
> > @Piotr
> > When FlinkKafkaConsumer was marked as deprecated, the migration steps
> were
> > included in the release notes:
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/release-notes/flink-1.14/#flink-24055httpsissuesapacheorgjirabrowseflink-24055
> > Scrolling up a bit shows how move from FlinkKafkaProducer to KafkaSink
> >
> > Given the importance of the Kafka connector, I think it's smart to write
> a
> > dedicated blog on how to migrate.
> > That could also include the differences between the two implementations,
> as
> > David just outlined.
> >
> > @Jing
> > SinkV2 was released with Flink 1.15, so with 1.16 released there will be
> 2
> > release cycles with that API at PublicEvolving. I don't think there were
> > breaking changes when the bug fixes were applied, because those bug fixes
> > were also applied in a patch version (1.15.3) which can't be done unless
> > they weren't breaking the API. Unless there will be work planned for 1.17
> > on the Sink, I think it can be moved to Public too.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Tue, Nov 1, 2022 at 7:21 AM David Anderson 
> > wrote:
> >
> > > >
> > > > [H]ow one can migrate from the
> > > > FlinkKafkaConsumer/FlinkKafkaProducer to KafkaSource/KafkaSink, while
> > > > preserving exactly-once guarantees etc?
> > >
> > >
> > > The responses from Fabian Paul in [1] and [2] address the question of
> how
> > > to handle the migration in terms of managing the state (where the short
> > > answer is "arrange for Kafka to be the source of truth").
> > >
> > > Those threads don't get into the differences in behavior between the
> two
> > > implementations. Here I'm thinking about (1) the loss of
> > > DeserializationSchema#isEndOfStream, and the fact that you can no
> longer
> > > dynamically determine when the input stream has finished, and (2) the
> > > change to how empty partitions are handled on startup (they used to be
> > > marked idle automatically, whereas now you must use withIdleness in the
> > > WatermarkStrategy).
> > >
> > > [1] https://www.mail-archive.com/user@flink.apache.org/msg44618.html
> > > [2] https://www.mail-archive.com/user@flink.apache.org/msg45864.html
> > >
> > > On Mon, Oct 31, 2022 at 7:32 PM Piotr Nowojski 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Maybe a stupid question, but how one can migrate from the
> > > > FlinkKafkaConsumer/FlinkKafkaProducer to KafkaSource/KafkaSink, while
> > > > preserving exactly-once guarantees etc? Is it possible? I've tried a
> > > quick
> > > > search and couldn't find it, but maybe I was looking in wrong places.
> > > >
> > > > Best,
> > > > Piotrek
> > > >
> > > > pon., 31 paź 2022 o 16:40 Jing Ge  napisał(a):
> > > >
> > > > > Thanks Martijn. What you said makes a lot of sense. I figure we
> > should
> > > do
> > > > > it in 2 steps.
> > > > >
> > > > >  Step 1 (with 1.17):
> > > > > - Remove FlinkKafkaConsumer.
> > > > > - Graduate Kafka Source from @PublicEvolving to @Public.
> > > > > - Update doc and l

Re: Stateful Functions with Flink 1.15 and onwards

[Tzu-Li (Gordon) Tai]

PR for upgrading to Flink 1.15.2 has been merged. Thanks for the efforts,
Galen and Filip!

We should be ready to kick off a new release for StateFun with the Flink
version upgrade.
I'll cut off a release branch now on apache/flink-statefun for
release-3.3.x to move things forward.
@Galen, @Filip if you want to, after the release branch is cut, you could
probably upgrade the master branch to Flink 1.16.x as well.

Afterwards we should decide who is available to drive the actual release
process for 3.3.0.
There's quite a few steps that would require committer write access.
Unless someone else is up for this earlier, I'll have some availability
towards the end of next week to help drive this.

Thanks,
Gordon

On Mon, Oct 31, 2022 at 12:17 PM Galen Warren 
wrote:

> Yes, that makes sense.
>
> PR is here: [FLINK-29814][statefun] Change supported Flink version to
> 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun
> (github.com) .
>
> On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann 
> wrote:
>
> > I think there might still be value in supporting 1.15 since not everyone
> > upgrades Flink very fast. Hopefully, for Statefun the diff between Flink
> > 1.15 and 1.16 boils down to changing the Flink dependencies.
> >
> > Cheers,
> > Till
> >
> > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren 
> > wrote:
> >
> >> Sure thing. One question -- Flink 1.16 was just released a few days ago.
> >> Should I support 1.15, or just go straight to 1.16?
> >>
> >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann 
> >> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> if you can open a PR for supporting Flink 1.15 Galen, then this would
> be
> >>> awesome. I've assigned you to this ticket. The next thing after merging
> >>> this PR would be creating a new StateFun release. Once we have merged
> the
> >>> PR, let's check who can help with it the fastest.
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren 
> >>> wrote:
> >>>
>  Yes, I could do that.
> 
>  On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <
>  filip.karni...@gmail.com> wrote:
> 
> > Hi All
> >
> > So what's the play here?
> >
> > Galen, what do you think about taking this on? Perhaps ++Till would
> > assign this jira to you (with your permission) given he's helped me
> out
> > with statefun work before
> > https://issues.apache.org/jira/browse/FLINK-29814
> >
> > I can try to move to move statefun to flink 1.16 when it's out
> >
> >
> > Kind regards
> > Fil
> >
> > On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <
> filip.karni...@gmail.com>
> > wrote:
> >
> >> Hi All
> >>
> >> Our use case is that we need to process elements for the same key
> >> sequentially, and this processing involves async operations.
> >>
> >> If any part of the processing fails, we store the offending and all
> >> subsequent incoming messages for that key in the state and not
> process any
> >> further messages for that key, until a retry succeeds or a human
> sends a
> >> 'skip' command message.
> >>
> >> diagram:
> >>
> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
> >> mermaid (in case mermaid.live goes down in the future):
> >> graph LR
> >> incoming[incoming events] --> job(Flink statefun job)
> >> commands[commands] -->|skip| job
> >> job --> |sequentially per key| remote(remote function)
> >> remote --> |on failure, delayed message to retry| remote
> >> remote --> |async puts/gets with side effects| other(other
> >> systems)
> >>
> >> Having the processing happen outside of Flink is nice-to-have from
> an
> >> independent scalability point of view, but is not strictly required.
> >>
> >> So long story short - no cyclic messaging, but also no way I can
> >> think of to use existing native Flink operators like async i/o
> (which when
> >> I last checked a few years back didn't have access to keyed state)
> >>
> >>
> >> P.S. Please note that there is already a pull request that has
> >> something to do wtih Flink 1.15, albeit without a description or a
> jira:
> >> https://github.com/apache/flink-statefun/pull/314
> >>
> >>
> >> On Wed, 26 Oct 2022 at 19:54, Galen Warren  >
> >> wrote:
> >>
> >>> Hi Gordon (and others),
> >>>
> >>> I'm also using this project for stateful messaging, including
> >>> messaging
> >>> among functions.
> >>>
> >>> I've contributed a small amount of code in the past and have also

Re: Stateful Functions with Flink 1.15 and onwards

[Galen Warren]

Thanks Gordon and Filip, I appreciate your help on this one.

On Tue, Nov 1, 2022 at 1:07 PM Tzu-Li (Gordon) Tai 
wrote:

> PR for upgrading to Flink 1.15.2 has been merged. Thanks for the efforts,
> Galen and Filip!
>
> We should be ready to kick off a new release for StateFun with the Flink
> version upgrade.
> I'll cut off a release branch now on apache/flink-statefun for
> release-3.3.x to move things forward.
> @Galen, @Filip if you want to, after the release branch is cut, you could
> probably upgrade the master branch to Flink 1.16.x as well.
>
> Afterwards we should decide who is available to drive the actual release
> process for 3.3.0.
> There's quite a few steps that would require committer write access.
> Unless someone else is up for this earlier, I'll have some availability
> towards the end of next week to help drive this.
>
> Thanks,
> Gordon
>
> On Mon, Oct 31, 2022 at 12:17 PM Galen Warren 
> wrote:
>
> > Yes, that makes sense.
> >
> > PR is here: [FLINK-29814][statefun] Change supported Flink version to
> > 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun
> > (github.com) .
> >
> > On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann 
> > wrote:
> >
> > > I think there might still be value in supporting 1.15 since not
> everyone
> > > upgrades Flink very fast. Hopefully, for Statefun the diff between
> Flink
> > > 1.15 and 1.16 boils down to changing the Flink dependencies.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren 
> > > wrote:
> > >
> > >> Sure thing. One question -- Flink 1.16 was just released a few days
> ago.
> > >> Should I support 1.15, or just go straight to 1.16?
> > >>
> > >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann 
> > >> wrote:
> > >>
> > >>> Hi folks,
> > >>>
> > >>> if you can open a PR for supporting Flink 1.15 Galen, then this would
> > be
> > >>> awesome. I've assigned you to this ticket. The next thing after
> merging
> > >>> this PR would be creating a new StateFun release. Once we have merged
> > the
> > >>> PR, let's check who can help with it the fastest.
> > >>>
> > >>> Cheers,
> > >>> Till
> > >>>
> > >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <
> ga...@cvillewarrens.com>
> > >>> wrote:
> > >>>
> >  Yes, I could do that.
> > 
> >  On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <
> >  filip.karni...@gmail.com> wrote:
> > 
> > > Hi All
> > >
> > > So what's the play here?
> > >
> > > Galen, what do you think about taking this on? Perhaps ++Till would
> > > assign this jira to you (with your permission) given he's helped me
> > out
> > > with statefun work before
> > > https://issues.apache.org/jira/browse/FLINK-29814
> > >
> > > I can try to move to move statefun to flink 1.16 when it's out
> > >
> > >
> > > Kind regards
> > > Fil
> > >
> > > On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <
> > filip.karni...@gmail.com>
> > > wrote:
> > >
> > >> Hi All
> > >>
> > >> Our use case is that we need to process elements for the same key
> > >> sequentially, and this processing involves async operations.
> > >>
> > >> If any part of the processing fails, we store the offending and
> all
> > >> subsequent incoming messages for that key in the state and not
> > process any
> > >> further messages for that key, until a retry succeeds or a human
> > sends a
> > >> 'skip' command message.
> > >>
> > >> diagram:
> > >>
> >
> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
> > >> mermaid (in case mermaid.live goes down in the future):
> > >> graph LR
> > >> incoming[incoming events] --> job(Flink statefun job)
> > >> commands[commands] -->|skip| job
> > >> job --> |sequentially per key| remote(remote function)
> > >> remote --> |on failure, delayed message to retry| remote
> > >> remote --> |async puts/gets with side effects| other(other
> > >> systems)
> > >>
> > >> Having the processing happen outside of Flink is nice-to-have from
> > an
> > >> independent scalability point of view, but is not strictly
> required.
> > >>
> > >> So long story short - no cyclic messaging, but also no way I can
> > >> think of to use existing native Flink operators like async i/o
> > (which when
> > >> I last checked a few years back didn't have access to keyed state)
> > >>
> > >>
> > >> P.S. Please note that there is already a pull request that has
> > >> something to do wtih Flink 1.15, albeit without a description or a
> > jira

Re: Stateful Functions with Flink 1.15 and onwards

[Tzu-Li (Gordon) Tai]

Btw, I'll assume that we're using this thread to gather consensus for
code-freezing for 3.3.x series of StateFun. I know there hasn't been much
activity on the repo, so this is just a formality really :)

>From the commit history, it looks like we're mainly including the below
major changes and bug fixes for 3.3.x:
- Flink upgrade to 1.15.2
- https://issues.apache.org/jira/browse/FLINK-26340
- https://issues.apache.org/jira/browse/FLINK-25866
- https://issues.apache.org/jira/browse/FLINK-25936
- https://issues.apache.org/jira/browse/FLINK-25933

I'll wait for 24 hours before cutting the release branch for 3.3.x, unless
anyone raises any objections before that.

Thanks,
Gordon

On Tue, Nov 1, 2022 at 10:09 AM Galen Warren 
wrote:

> Thanks Gordon and Filip, I appreciate your help on this one.
>
> On Tue, Nov 1, 2022 at 1:07 PM Tzu-Li (Gordon) Tai 
> wrote:
>
>> PR for upgrading to Flink 1.15.2 has been merged. Thanks for the efforts,
>> Galen and Filip!
>>
>> We should be ready to kick off a new release for StateFun with the Flink
>> version upgrade.
>> I'll cut off a release branch now on apache/flink-statefun for
>> release-3.3.x to move things forward.
>> @Galen, @Filip if you want to, after the release branch is cut, you could
>> probably upgrade the master branch to Flink 1.16.x as well.
>>
>> Afterwards we should decide who is available to drive the actual release
>> process for 3.3.0.
>> There's quite a few steps that would require committer write access.
>> Unless someone else is up for this earlier, I'll have some availability
>> towards the end of next week to help drive this.
>>
>> Thanks,
>> Gordon
>>
>> On Mon, Oct 31, 2022 at 12:17 PM Galen Warren 
>> wrote:
>>
>> > Yes, that makes sense.
>> >
>> > PR is here: [FLINK-29814][statefun] Change supported Flink version to
>> > 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun
>> > (github.com) .
>> >
>> > On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann 
>> > wrote:
>> >
>> > > I think there might still be value in supporting 1.15 since not
>> everyone
>> > > upgrades Flink very fast. Hopefully, for Statefun the diff between
>> Flink
>> > > 1.15 and 1.16 boils down to changing the Flink dependencies.
>> > >
>> > > Cheers,
>> > > Till
>> > >
>> > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren > >
>> > > wrote:
>> > >
>> > >> Sure thing. One question -- Flink 1.16 was just released a few days
>> ago.
>> > >> Should I support 1.15, or just go straight to 1.16?
>> > >>
>> > >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann 
>> > >> wrote:
>> > >>
>> > >>> Hi folks,
>> > >>>
>> > >>> if you can open a PR for supporting Flink 1.15 Galen, then this
>> would
>> > be
>> > >>> awesome. I've assigned you to this ticket. The next thing after
>> merging
>> > >>> this PR would be creating a new StateFun release. Once we have
>> merged
>> > the
>> > >>> PR, let's check who can help with it the fastest.
>> > >>>
>> > >>> Cheers,
>> > >>> Till
>> > >>>
>> > >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <
>> ga...@cvillewarrens.com>
>> > >>> wrote:
>> > >>>
>> >  Yes, I could do that.
>> > 
>> >  On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <
>> >  filip.karni...@gmail.com> wrote:
>> > 
>> > > Hi All
>> > >
>> > > So what's the play here?
>> > >
>> > > Galen, what do you think about taking this on? Perhaps ++Till
>> would
>> > > assign this jira to you (with your permission) given he's helped
>> me
>> > out
>> > > with statefun work before
>> > > https://issues.apache.org/jira/browse/FLINK-29814
>> > >
>> > > I can try to move to move statefun to flink 1.16 when it's out
>> > >
>> > >
>> > > Kind regards
>> > > Fil
>> > >
>> > > On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <
>> > filip.karni...@gmail.com>
>> > > wrote:
>> > >
>> > >> Hi All
>> > >>
>> > >> Our use case is that we need to process elements for the same key
>> > >> sequentially, and this processing involves async operations.
>> > >>
>> > >> If any part of the processing fails, we store the offending and
>> all
>> > >> subsequent incoming messages for that key in the state and not
>> > process any
>> > >> further messages for that key, until a retry succeeds or a human
>> > sends a
>> > >> 'skip' command message.
>> > >>
>> > >> diagram:
>> > >>
>> >
>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
>> > >> mermaid (in case mermaid.live goes down in the future):
>> > >> graph LR
>> > >> incoming[incoming events] --> job(Flink statefun job)
>> > >> commands

[GitHub] [flink-connector-shared-utils] MartijnVisser commented on a diff in pull request #1: [FLINK-29472] Add first version of release scripts

[GitBox]

MartijnVisser commented on code in PR #1:
URL: 
https://github.com/apache/flink-connector-shared-utils/pull/1#discussion_r1010743355


##
README.md:
##
@@ -1 +1,79 @@
-This repository contains utilities for [Apache 
Flink](https://flink.apache.org/) connectors.
\ No newline at end of file
+This is a collection of release utils for [Apache 
Flink](https://flink.apache.org/) connectors.
+
+# Integration
+
+The scripts assume that they are integrated into a connector repo as a 
submodule into the connector repo
+under `tools/releasing/`.
+
+# Usage
+
+Some scripts rely on environment variables to be set.  
+These are checked at the start of each script.  
+Any instance of `${some_variable}` in this document refers to an environment 
variable that is used by the respective
+script.
+
+## check_environment.sh
+
+Runs some pre-release checks for the current environment, for example that all 
required programs are available.  
+This should be run once at the start of the release process.
+
+## publish_snapshot_branch.sh
+
+Creates (and pushes!) a new snapshot branch for the current commit.  
+The branch name is automatically determined from the version in the pom.  
+This script should be called when work on a new major/minor version has 
started.

Review Comment:
   ```suggestion
   This script should be called when work on a new major/minor version of the 
connector has started.
   ```



##
_init.sh:
##
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# all scripts should contain this line + source ${SCRIPT_DIR}/_init.sh
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+export SHELLOPTS
+
+###
+
+MVN=${MVN:-mvn}
+
+if [ "$(uname)" == "Darwin" ]; then

Review Comment:
   While I have Darwin, I can also both run `sha512sum` and `shasum -a 512`. 
Don't think that's an issue for this script's purpose though



##
check_environment.sh:
##
@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
+
+source "${SCRIPT_DIR}/_init.sh"
+
+EXIT_CODE=0
+
+function check_program_available {
+  if program=$(command -v ${1}); then
+printf "\t%-10s%s\n" "${1}" "using ${program}"
+  else
+printf "\t%-10s%s\n" "${1}" "is not available."
+EXIT_CODE=1
+  fi
+}
+
+echo "Checking program availability:"
+check_program_available git
+check_program_available tar
+check_program_available rsync
+check_program_available gpg
+check_program_available perl
+check_program_available sed
+check_program_available svn
+check_program_available ${MVN}

Review Comment:
   Just to double check, any version of Maven for connectors should suffice, 
right? Doesn't need to be 3.2.5 like for Flink itself



##
README.md:
##
@@ -1 +1,79 @@
-This repository contains utilities for [Apache 
Flink](https://flink.apache.org/) connectors.
\ No newline at end of file
+This is a collection of release utils for [Apache 
Flink](https://flink.apache.org/) connectors.
+
+# Integration
+
+The scripts assume that they are integrated into a connector repo as a 
submodule into the connector repo
+under `tools/releasing/`.
+
+# Usage
+
+Some scripts rely on environment variables to be set.  
+These are checked at the start of each script.  
+Any instance of `${some_va

[jira] [Created] (FLINK-29837) SQL API does not expose the RowKind of the Row for processing Changelogs

[Eric Xiao (Jira)]

Eric Xiao created FLINK-29837:
-

 Summary: SQL API does not expose the RowKind of the Row for 
processing Changelogs
 Key: FLINK-29837
 URL: https://issues.apache.org/jira/browse/FLINK-29837
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.16.0
Reporter: Eric Xiao


When working with `{{{}ChangeLog{}}}` data in the SQL API it was a bit 
misleading to see that the `{{{}op{}}}` column appears{^}[1]{^}  the type of  
in the table schema of print results but it is not available to be used in a 
the SQL API:
{code:java}
val tableEnv = StreamTableEnvironment.create(env)

val dataStream = env.fromElements(
  Row.ofKind(RowKind.INSERT, "Alice", Int.box(12)),
  Row.ofKind(RowKind.INSERT, "Bob", Int.box(5)),
  Row.ofKind(RowKind.UPDATE_AFTER, "Alice", Int.box(100))
)(Types.ROW(Types.STRING, Types.INT))

// interpret the DataStream as a Table
val table =
  tableEnv.fromChangelogStream(dataStream, 
Schema.newBuilder().primaryKey("f0").build(), ChangelogMode.upsert())

// register the table under a name and perform an aggregation
tableEnv.createTemporaryView("InputTable", table)
tableEnv
  .sqlQuery("SELECT * FROM InputTable where op = '+I'")
  .execute()
  .print() {code}
The error logs.

 

 
{code:java}
Exception in thread "main" org.apache.flink.table.api.ValidationException: SQL 
validation failed. From line 1, column 32 to line 1, column 33: Column 'op' not 
found in any table
    at 
org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:184)
    at 
org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:109)
    at 
org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:237)
    at 
org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:105)
    at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:675)
    at 
com.shopify.trickle.pipelines.IteratorPipeline$.delayedEndpoint$com$shopify$trickle$pipelines$IteratorPipeline$1(IteratorPipeline.scala:32)
    at 
com.shopify.trickle.pipelines.IteratorPipeline$delayedInit$body.apply(IteratorPipeline.scala:11)
    at scala.Function0.apply$mcV$sp(Function0.scala:39)
    at scala.Function0.apply$mcV$sp$(Function0.scala:39)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
    at scala.App.$anonfun$main$1$adapted(App.scala:80)
    at scala.collection.immutable.List.foreach(List.scala:431)
    at scala.App.main(App.scala:80)
    at scala.App.main$(App.scala:78)
    at 
com.shopify.trickle.pipelines.IteratorPipeline$.main(IteratorPipeline.scala:11)
    at 
com.shopify.trickle.pipelines.IteratorPipeline.main(IteratorPipeline.scala) 
{code}
It would be nice to expose the `op` column to be usable in the Flink SQL APIs 
as it is in the DataStream APIs.

[1] 
[https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/data_stream_api/#examples-for-fromchangelogstream]
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29838) Hive streaming sink for partitioned table should contain metastore by default

[luoyuxia (Jira)]

luoyuxia created FLINK-29838:


 Summary: Hive streaming sink for partitioned table should contain 
metastore by default
 Key: FLINK-29838
 URL: https://issues.apache.org/jira/browse/FLINK-29838
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Hive
Reporter: luoyuxia






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29839) SQL Gateway with hiveserver2 throw exception

[chenqizhu (Jira)]

chenqizhu created FLINK-29839:
-

 Summary: SQL Gateway with hiveserver2 throw exception
 Key: FLINK-29839
 URL: https://issues.apache.org/jira/browse/FLINK-29839
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Gateway
Affects Versions: 1.16.0
 Environment: Flink version: 1.16.0
Hive version: 3.1.2
Reporter: chenqizhu


 I had starting the SQL Gateway with the HiveServer2 Endpoint, and then I 
submit SQL with Hive Beeline, but I get the following exception:
{code:java}
java.lang.UnsupportedOperationException: Unrecognized TGetInfoType value: 
CLI_ODBC_KEYWORDS.
at 
org.apache.flink.table.endpoint.hive.HiveServer2Endpoint.GetInfo(HiveServer2Endpoint.java:371)
 [flink-sql-connector-hive-3.1.2_2.12-1.16.0.jar:1.16.0]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetInfo.getResult(TCLIService.java:1537)
 [flink-sql-connector-hive-3.1.2_2.12-1.16.0.jar:1.16.0]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetInfo.getResult(TCLIService.java:1522)
 [flink-sql-connector-hive-3.1.2_2.12-1.16.0.jar:1.16.0]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) 
[flink-sql-connector-hive-3.1.2_2.12-1.16.0.jar:1.16.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
[flink-sql-connector-hive-3.1.2_2.12-1.16.0.jar:1.16.0]
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
 [flink-sql-connector-hive-3.1.2_2.12-1.16.0.jar:1.16.0]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
[?:?]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
2022-11-01 13:55:33,885 ERROR org.apache.thrift.server.TThreadPoolServer
   [] - Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Required field 'infoValue' is 
unset! Struct:TGetInfoResp(status:TStatus(statusCode:ERROR_STATUS, 
infoMessages:[*java.lang.UnsupportedOperationException:Unrecognized 
TGetInfoType value: CLI_ODBC_KEYWORDS.:9:8, 
org.apache.flink.table.endpoint.hive.HiveServer2Endpoint:GetInfo:HiveServer2Endpoint.java:371,
 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetInfo:getResult:TCLIService.java:1537,
 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetInfo:getResult:TCLIService.java:1522,
 org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39, 
org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39, 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:286,
 
java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1128, 
java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:628, 
java.lang.Thread:run:Thread.java:834], errorMessage:Unrecognized TGetInfoType 
value: CLI_ODBC_KEYWORDS.), infoValue:null)
at 
org.apache.hive.service.rpc.thrift.TGetInfoResp.validate(TGetInfoResp.java:379) 
~[flink-sql-connector-hive-3.1.2_2.12-1.16.0.jar:1.16.0]
at 
org.apache.hive.service.rpc.thrift.TCLIService$GetInfo_result.validate(TCLIService.java:5228)
 ~[flink-sql-connector-hive-3.1.2_2.12-1.16.0.jar:1.16.0]
at 
org.apache.hive.service.rpc.thrift.TCLIService$GetInfo_result$GetInfo_resultStandardScheme.write(TCLIService.java:5285)
 ~[flink-sql-connector-hive-3.1.2_2.12-1.16.0.jar:1.16.0]
at 
org.apache.hive.service.rpc.thrift.TCLIService$GetInfo_result$GetInfo_resultStandardScheme.write(TCLIService.java:5254)
 ~[flink-sql-connector-hive-3.1.2_2.12-1.16.0.jar:1.16.0]
at 
org.apache.hive.service.rpc.thrift.TCLIService$GetInfo_result.write(TCLIService.java:5205)
 ~[flink-sql-connector-hive-3.1.2_2.12-1.16.0.jar:1.16.0]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:53) 
~[flink-sql-connector-hive-3.1.2_2.12-1.16.0.jar:1.16.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
~[flink-sql-connector-hive-3.1.2_2.12-1.16.0.jar:1.16.0]
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
 [flink-sql-connector-hive-3.1.2_2.12-1.16.0.jar:1.16.0]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
[?:?]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
2022-11-01 13:55:33,886 WARN  org.apache.thrift.transport.TIOStreamTransport
   [] - Error closing output stream.
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) ~[?:?]
at java.net.SocketOutputStream.write(SocketOutputStream.java:150) ~[?:?]
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?]
at java.io.FilterOutputStream.close(FilterOutput

[jira] [Created] (FLINK-29840) Old record may overwrite new record in Table Store when snapshot committing is slow

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-29840:
---

 Summary: Old record may overwrite new record in Table Store when 
snapshot committing is slow
 Key: FLINK-29840
 URL: https://issues.apache.org/jira/browse/FLINK-29840
 Project: Flink
  Issue Type: Bug
  Components: Table Store
Affects Versions: table-store-0.3.0, table-store-0.2.2
Reporter: Caizhi Weng
Assignee: Caizhi Weng
 Fix For: table-store-0.3.0, table-store-0.2.2


Consider the following scenario when snapshot committing is slow:
* A writer produces some records at checkpoint T.
* It produces no record at checkpoint T+1 and is closed.
* It produces some records at checkpoint T+2. It will be reopened and read the 
latest sequence number from disk. However snapshot at checkpoint T may not be 
committed so the sequence number it reads might be too small.

In this scenario, records from checkpoint T may overwrite records from 
checkpoint T+2 because they have larger sequence numbers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29841) docs/content/docs/deployment/config.md has error

[hcj (Jira)]

hcj created FLINK-29841:
---

 Summary: docs/content/docs/deployment/config.md has error
 Key: FLINK-29841
 URL: https://issues.apache.org/jira/browse/FLINK-29841
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.15.2, 1.16.0
Reporter: hcj


line 64 in docs/content/docs/deployment/config.md has error.

line 64 is :

These value are configured as memory sizes, for example *1536m* or *2g*.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29842) Change commitIdentifier in Table Store snapshot to long value

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-29842:
---

 Summary: Change commitIdentifier in Table Store snapshot to long 
value
 Key: FLINK-29842
 URL: https://issues.apache.org/jira/browse/FLINK-29842
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Affects Versions: table-store-0.3.0, table-store-0.2.2
Reporter: Caizhi Weng
Assignee: Caizhi Weng
 Fix For: table-store-0.3.0, table-store-0.2.2


Currently {{commitIdentifier}} in {{Snapshot}} is a {{String}} value. However 
there are many scenarios where we need to compare two identifiers to find out 
which one is newer. For example
* In FLINK-29840, we need to store the latest modified commit for each writer. 
Only when the latest snapshot is newer than this commit can we safely close the 
writer.
* In FLINK-29805, we can read the commit identifier of the latest snapshot. All 
identifiers older than that should be filtered out.
* In FLINK-29752, we need to trigger full compaction once in a few commits. We 
can read the latest commit identifier and compare it with the full compaction 
identifier to check if full compaction is successfully committed.
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[RESULT][VOTE] FLIP-263: Improve resolving schema compatibility

[Hangxiang Yu]

Hi everyone,

Happy to announce that FLIP-263 [1] has been accepted.


>From vote thread [2], There are 9 approving votes, 7 of which are binding:

- Yanfei Lei (non-binding)
- Yuan Mei (binding)

- Zakelly Lan (non-binding)
- Yun Gao (binding)
- Godfrey He (binding)

- Dawid Wysakowicz (binding)

- Yu Li (binding)

- Tzu-Li (Gordon) Tai (binding)

- Yun Tang (binding)

There are no disapproving votes.

Thanks everyone for joining the discussion, giving feedback and voting !

Best regards,
Hangxiang.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility
[2] https://lists.apache.org/thread/0bh530j5ob11lzj48vpm883sqwgmstp8

[jira] [Created] (FLINK-29843) Euclidean Distance Measure generates NAN distance values

[Yunfeng Zhou (Jira)]

Yunfeng Zhou created FLINK-29843:


 Summary: Euclidean Distance Measure generates NAN distance values
 Key: FLINK-29843
 URL: https://issues.apache.org/jira/browse/FLINK-29843
 Project: Flink
  Issue Type: Bug
  Components: Library / Machine Learning
Affects Versions: ml-2.1.0
Reporter: Yunfeng Zhou


Currently Flink ML's `EuclideanDistanceMeasure.distance(...)` method might 
return a negative value as the distance between two vectors given the 
calculation accuracy of java doubles. This bug should be fixed to guarantee 
that the distance is a non-negative value.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Remove FlinkKafkaConsumer and FlinkKafkaProducer in the master for 1.17 release

[Qingsheng Ren]

Thanks Jing for starting the discussion.

+1 for removing FlinkKafkaConsumer, as KafkaSource has evolved for many
release cycles and should be stable enough. I have some concerns about the
new Kafka sink based on sink v2, as sink v2 still has some ongoing work in
1.17 (maybe Yun Gao could provide some inputs). Also we found some issues
of KafkaSink related to the internal mechanism of sink v2, like
FLINK-29492.

@David
About the ability of DeserializationSchema#isEndOfStream, FLIP-208 is
trying to complete this piece of the puzzle, and Hang Ruan (
ruanhang1...@gmail.com) plans to work on it in 1.17. For the partition
idleness problem could you elaborate more about it? I assume both
FlinkKafkaConsumer and KafkaSource need a WatermarkStrategy to decide
whether to mark the partition as idle.

Best,
Qingsheng
Ververica (Alibaba)

On Thu, Oct 27, 2022 at 8:06 PM Jing Ge  wrote:

> Hi Dev,
>
> I'd like to start a discussion about removing FlinkKafkaConsumer and
> FlinkKafkaProducer in 1.17.
>
> Back in the past, it was originally announced to remove it with Flink 1.15
> after Flink 1.14 had been released[1]. And then postponed to the next 1.15
> release which meant to remove it with Flink 1.16 but forgot to change the
> doc[2]. I have created a PRs to fix it. Since the 1.16 release branch has
> code freeze, it makes sense to, first of all, update the doc to say that
> FlinkKafkaConsumer will be removed with Flink 1.17 [3][4] and second start
> the discussion about removing them with the current master branch i.e. for
> the coming 1.17 release. I'm all ears and looking forward to your feedback.
> Thanks!
>
> Best regards,
> Jing
>
>
>
>
>
>
>
>
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/kafka/#kafka-sourcefunction
> [2]
>
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kafka/#kafka-sourcefunction
> [3] https://github.com/apache/flink/pull/21172
> [4] https://github.com/apache/flink/pull/21171
>

[jira] [Created] (FLINK-29844) FLIP-263: Improve resolving schema compatibility

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-29844:


 Summary: FLIP-263: Improve resolving schema compatibility
 Key: FLINK-29844
 URL: https://issues.apache.org/jira/browse/FLINK-29844
 Project: Flink
  Issue Type: Improvement
  Components: API / Type Serialization System
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)