[jira] [Created] (FLINK-34544) The release check bug in tiered memory manager of hybrid shuffle

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-34544:
-

 Summary: The release check bug in tiered memory manager of hybrid 
shuffle
 Key: FLINK-34544
 URL: https://issues.apache.org/jira/browse/FLINK-34544
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.19.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


The `TieredStorageMemoryManagerImpl` has a `numRequestedBuffers` check when 
releasing resources. However, this check is performed in the task thread, while 
the buffer recycle may occur in the Netty thread. As a result, it may 
incorrectly throw an exception when the release is too quick for the vertex, 
which has almost no data.
We should fix it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] FLIP-423 ～FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

[Yuan Mei]

Hi Devs,

This is a joint work of Yuan Mei, Zakelly Lan, Jinzhong Li, Hangxiang Yu,
Yanfei Lei and Feng Wang. We'd like to start a discussion about introducing
Disaggregated State Storage and Management in Flink 2.0.

The past decade has witnessed a dramatic shift in Flink's deployment mode,
workload patterns, and hardware improvements. We've moved from the
map-reduce era where workers are computation-storage tightly coupled nodes
to a cloud-native world where containerized deployments on Kubernetes
become standard. To enable Flink's Cloud-Native future, we introduce
Disaggregated State Storage and Management that uses DFS as primary storage
in Flink 2.0, as promised in the Flink 2.0 Roadmap.

Design Details can be found in FLIP-423[1].

This new architecture is aimed to solve the following challenges brought in
the cloud-native era for Flink.
1. Local Disk Constraints in containerization
2. Spiky Resource Usage caused by compaction in the current state model
3. Fast Rescaling for jobs with large states (hundreds of Terabytes)
4. Light and Fast Checkpoint in a native way

More specifically, we want to reach a consensus on the following issues in
this discussion:

1. Overall design
2. Proposed Changes
3. Design details to achieve Milestone1

In M1, we aim to achieve an end-to-end baseline version using DFS as
primary storage and complete core functionalities, including:

- Asynchronous State APIs (FLIP-424)[2]: Introduce new APIs for
asynchronous state access.
- Asynchronous Execution Model (FLIP-425)[3]: Implement a non-blocking
execution model leveraging the asynchronous APIs introduced in FLIP-424.
- Grouping Remote State Access (FLIP-426)[4]: Enable retrieval of remote
state data in batches to avoid unnecessary round-trip costs for remote
access
- Disaggregated State Store (FLIP-427)[5]: Introduce the initial version of
the ForSt disaggregated state store.
- Fault Tolerance/Rescale Integration (FLIP-428)[6]: Integrate
checkpointing mechanisms with the disaggregated state store for fault
tolerance and fast rescaling.

We will vote on each FLIP in separate threads to make sure each FLIP
reaches a consensus. But we want to keep the discussion within a focused
thread (this thread) for easier tracking of contexts to avoid duplicated
questions/discussions and also to think of the problem/solution in a full
picture.

Looking forward to your feedback

Best,
Yuan, Zakelly, Jinzhong, Hangxiang, Yanfei and Feng

[1] https://cwiki.apache.org/confluence/x/R4p3EQ
[2] https://cwiki.apache.org/confluence/x/SYp3EQ
[3] https://cwiki.apache.org/confluence/x/S4p3EQ
[4] https://cwiki.apache.org/confluence/x/TYp3EQ
[5] https://cwiki.apache.org/confluence/x/T4p3EQ
[6] https://cwiki.apache.org/confluence/x/UYp3EQ

Re: [ANNOUNCE] New Apache Flink Committer - Jiabao Sun

[Jiadong Lu]

Congratulations Jiabao!

Best regards,
Jiadong Lu

On 2024/2/29 11:23, Shawn Huang wrote:

Congratulations Jiabao!

Re: [ANNOUNCE] New Apache Flink Committer - Jiabao Sun

[Shawn Huang]

Congratulations Jiabao!

Best,
Shawn Huang


yh z  于2024年2月26日周一 16:17写道：

> Congratulations Jiabao!
>
> Best,
> Yunhong Zheng (SwufeHong)
>
> yu'an huang  于2024年2月26日周一 10:28写道：
>
> > Congratulations, Jiabao!
> >
> > Best,
> > Yuan
> >
> >
> > On Mon, 26 Feb 2024 at 9:38 AM, Ron liu  wrote:
> >
> > > Congratulations, Jiabao!
> > >
> > > Best,
> > > Ron
> > >
> > > Yun Tang  于2024年2月23日周五 19:59写道：
> > >
> > > > Congratulations, Jiabao!
> > > >
> > > > Best
> > > > Yun Tang
> > > > 
> > > > From: Weihua Hu 
> > > > Sent: Thursday, February 22, 2024 17:29
> > > > To: dev@flink.apache.org 
> > > > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Jiabao Sun
> > > >
> > > > Congratulations, Jiabao!
> > > >
> > > > Best,
> > > > Weihua
> > > >
> > > >
> > > > On Thu, Feb 22, 2024 at 10:34 AM Jingsong Li  >
> > > > wrote:
> > > >
> > > > > Congratulations! Well deserved!
> > > > >
> > > > > On Wed, Feb 21, 2024 at 4:36 PM Yuepeng Pan  >
> > > > wrote:
> > > > > >
> > > > > > Congratulations~ :)
> > > > > >
> > > > > > Best,
> > > > > > Yuepeng Pan
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > 在 2024-02-21 09:52:17，"Hongshun Wang" 
> > 写道：
> > > > > > >Congratulations, Jiabao :)
> > > > > > >Congratulations Jiabao!
> > > > > > >
> > > > > > >Best,
> > > > > > >Hongshun
> > > > > > >Best regards,
> > > > > > >
> > > > > > >Weijie
> > > > > > >
> > > > > > >On Tue, Feb 20, 2024 at 2:19 PM Runkang He 
> > > wrote:
> > > > > > >
> > > > > > >> Congratulations Jiabao!
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Runkang He
> > > > > > >>
> > > > > > >> Jane Chan  于2024年2月20日周二 14:18写道：
> > > > > > >>
> > > > > > >> > Congrats, Jiabao!
> > > > > > >> >
> > > > > > >> > Best,
> > > > > > >> > Jane
> > > > > > >> >
> > > > > > >> > On Tue, Feb 20, 2024 at 10:32 AM Paul Lam <
> > > paullin3...@gmail.com>
> > > > > wrote:
> > > > > > >> >
> > > > > > >> > > Congrats, Jiabao!
> > > > > > >> > >
> > > > > > >> > > Best,
> > > > > > >> > > Paul Lam
> > > > > > >> > >
> > > > > > >> > > > 2024年2月20日 10:29，Zakelly Lan 
> 写道：
> > > > > > >> > > >
> > > > > > >> > > >> Congrats! Jiabao!
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener

[Hang Ruan]

+1 (non-binding)

Best,
Hang

weijie guo  于2024年2月29日周四 09:55写道：

> +1 (binding)
>
> Best regards,
>
> Weijie
>
>
> Feng Jin  于2024年2月29日周四 09:37写道：
>
> > +1 (non-binding)
> >
> > Best,
> > Feng Jin
> >
> > On Thu, Feb 29, 2024 at 4:41 AM Márton Balassi  >
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Marton
> > >
> > > On Wed, Feb 28, 2024 at 5:14 PM Gyula Fóra 
> wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Gyula
> > > >
> > > > On Wed, Feb 28, 2024 at 11:10 AM Maciej Obuchowski <
> > > mobuchow...@apache.org
> > > > >
> > > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Best,
> > > > > Maciej Obuchowski
> > > > >
> > > > > śr., 28 lut 2024 o 10:29 Zhanghao Chen 
> > > > > napisał(a):
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Best,
> > > > > > Zhanghao Chen
> > > > > > 
> > > > > > From: Yong Fang 
> > > > > > Sent: Wednesday, February 28, 2024 10:12
> > > > > > To: dev 
> > > > > > Subject: [VOTE] FLIP-314: Support Customized Job Lineage Listener
> > > > > >
> > > > > > Hi devs,
> > > > > >
> > > > > > I would like to restart a vote about FLIP-314: Support Customized
> > Job
> > > > > > Lineage Listener[1].
> > > > > >
> > > > > > Previously, we added lineage related interfaces in FLIP-314.
> Before
> > > the
> > > > > > interfaces were developed and merged into the master, @Maciej and
> > > > > > @Zhenqiu provided valuable suggestions for the interface from the
> > > > > > perspective of the lineage system. So we updated the interfaces
> of
> > > > > FLIP-314
> > > > > > and discussed them again in the discussion thread [2].
> > > > > >
> > > > > > So I am here to initiate a new vote on FLIP-314, the vote will be
> > > open
> > > > > for
> > > > > > at least 72 hours unless there is an objection or insufficient
> > votes
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> > > > > > [2]
> > https://lists.apache.org/thread/wopprvp3ww243mtw23nj59p57cghh7mc
> > > > > >
> > > > > > Best,
> > > > > > Fang Yong
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Flink kinesis connector v4.2.0-1.18 has issue when consuming from EFO consumer

[Xiaolong Wang]

Sorry, I just attached a wrong file. Let me paste the error log:

java.lang.RuntimeException: Maximum retries exceeded for SubscribeToShard.
Failed 10 times.
at
org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisher.runWithBackoff(FanOutRecordPublisher.java:
211) ~[?:?]
at
org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisher.run(FanOutRecordPublisher.java:
130) ~[?:?]
at
org.apache.flink.streaming.connectors.kinesis.internals.ShardConsumer.run(ShardConsumer.java:
114) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?
]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.lang.IllegalStateException: Connection pool shut down
at
org.apache.flink.kinesis.shaded.org.apache.http.util.Asserts.check(Asserts.java:
34) ~[?:?]
at
org.apache.flink.kinesis.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:
269) ~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory
$
DelegatingHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:
75) ~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory
$
InstrumentedHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:
57) ~[?:?]
at
org.apache.flink.kinesis.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:
176) ~[?:?]
at
org.apache.flink.kinesis.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:
186) ~[?:?]
at
org.apache.flink.kinesis.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:
185) ~[?:?]
at
org.apache.flink.kinesis.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:
83) ~[?:?]
at
org.apache.flink.kinesis.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:
56) ~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:
72) ~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:
254) ~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient.access
$500(ApacheHttpClient.java:104) ~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient
$1.call(ApacheHttpClient.java:231) ~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient
$1.call(ApacheHttpClient.java:228) ~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:
67) ~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:
77) ~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:
56) ~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:
39) ~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder
$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:
?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder
$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:
?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder
$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:
?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder
$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[?:
?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:
72) ~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:
42) ~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:
78) ~[?:?]
at

[jira] [Created] (FLINK-34543) Support Full Partition Processing On Non-keyed DataStream

[Wencong Liu (Jira)]

Wencong Liu created FLINK-34543:
---

 Summary: Support Full Partition Processing On Non-keyed DataStream
 Key: FLINK-34543
 URL: https://issues.apache.org/jira/browse/FLINK-34543
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Affects Versions: 1.20.0
Reporter: Wencong Liu
 Fix For: 1.20.0


1. Introduce MapParititon, SortPartition, Aggregate, Reduce API in DataStream.
2. Introduce SortPartition API in KeyedStream.

The related FLIP can be found in 
[FLIP-380|https://cwiki.apache.org/confluence/display/FLINK/FLIP-380%3A+Support+Full+Partition+Processing+On+Non-keyed+DataStream].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34542) Improve Gradle Quick Start build.gradle with Better Gradle API Alternatives

[Lennon Yu (Jira)]

Lennon Yu created FLINK-34542:
-

 Summary: Improve Gradle Quick Start build.gradle with Better 
Gradle API Alternatives
 Key: FLINK-34542
 URL: https://issues.apache.org/jira/browse/FLINK-34542
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Lennon Yu


This is a ticket of misc. improvements on the build.gradle script provided at 
{{Application Development}} >> {{Project Configuration}} >> {{Overview}} >> 
{{Getting Started:}}
 * {{{}Add {{mergeServiceFiles(){} call to the {{shadowJar}} configuration 
block
 ** Absence of this will cause class-not-found errors in SPI related class 
loading if the user has multiple connectors/formats in their implementation.
 * Move the top level {{mainClassName}} project property setting into 
application \{ mainClass = 'foo.Bar' }
 ** This is because the top-level mainClassName property will be deprecated in 
Gradle 9.0+
 * Replace the use of {{sourceCompatibility}} and {{targetCompatibility}} 
properties with java \{ toolChain { languageVersion = 
JavaLanguageVersion.of(17) } }
 ** This is the recommended way by Gradle to streamline langauge version 
configuration.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener

[weijie guo]

+1 (binding)

Best regards,

Weijie


Feng Jin  于2024年2月29日周四 09:37写道：

> +1 (non-binding)
>
> Best,
> Feng Jin
>
> On Thu, Feb 29, 2024 at 4:41 AM Márton Balassi 
> wrote:
>
> > +1 (binding)
> >
> > Marton
> >
> > On Wed, Feb 28, 2024 at 5:14 PM Gyula Fóra  wrote:
> >
> > > +1 (binding)
> > >
> > > Gyula
> > >
> > > On Wed, Feb 28, 2024 at 11:10 AM Maciej Obuchowski <
> > mobuchow...@apache.org
> > > >
> > > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Best,
> > > > Maciej Obuchowski
> > > >
> > > > śr., 28 lut 2024 o 10:29 Zhanghao Chen 
> > > > napisał(a):
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Best,
> > > > > Zhanghao Chen
> > > > > 
> > > > > From: Yong Fang 
> > > > > Sent: Wednesday, February 28, 2024 10:12
> > > > > To: dev 
> > > > > Subject: [VOTE] FLIP-314: Support Customized Job Lineage Listener
> > > > >
> > > > > Hi devs,
> > > > >
> > > > > I would like to restart a vote about FLIP-314: Support Customized
> Job
> > > > > Lineage Listener[1].
> > > > >
> > > > > Previously, we added lineage related interfaces in FLIP-314. Before
> > the
> > > > > interfaces were developed and merged into the master, @Maciej and
> > > > > @Zhenqiu provided valuable suggestions for the interface from the
> > > > > perspective of the lineage system. So we updated the interfaces of
> > > > FLIP-314
> > > > > and discussed them again in the discussion thread [2].
> > > > >
> > > > > So I am here to initiate a new vote on FLIP-314, the vote will be
> > open
> > > > for
> > > > > at least 72 hours unless there is an objection or insufficient
> votes
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> > > > > [2]
> https://lists.apache.org/thread/wopprvp3ww243mtw23nj59p57cghh7mc
> > > > >
> > > > > Best,
> > > > > Fang Yong
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener

[Feng Jin]

+1 (non-binding)

Best,
Feng Jin

On Thu, Feb 29, 2024 at 4:41 AM Márton Balassi 
wrote:

> +1 (binding)
>
> Marton
>
> On Wed, Feb 28, 2024 at 5:14 PM Gyula Fóra  wrote:
>
> > +1 (binding)
> >
> > Gyula
> >
> > On Wed, Feb 28, 2024 at 11:10 AM Maciej Obuchowski <
> mobuchow...@apache.org
> > >
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Maciej Obuchowski
> > >
> > > śr., 28 lut 2024 o 10:29 Zhanghao Chen 
> > > napisał(a):
> > >
> > > > +1 (non-binding)
> > > >
> > > > Best,
> > > > Zhanghao Chen
> > > > 
> > > > From: Yong Fang 
> > > > Sent: Wednesday, February 28, 2024 10:12
> > > > To: dev 
> > > > Subject: [VOTE] FLIP-314: Support Customized Job Lineage Listener
> > > >
> > > > Hi devs,
> > > >
> > > > I would like to restart a vote about FLIP-314: Support Customized Job
> > > > Lineage Listener[1].
> > > >
> > > > Previously, we added lineage related interfaces in FLIP-314. Before
> the
> > > > interfaces were developed and merged into the master, @Maciej and
> > > > @Zhenqiu provided valuable suggestions for the interface from the
> > > > perspective of the lineage system. So we updated the interfaces of
> > > FLIP-314
> > > > and discussed them again in the discussion thread [2].
> > > >
> > > > So I am here to initiate a new vote on FLIP-314, the vote will be
> open
> > > for
> > > > at least 72 hours unless there is an objection or insufficient votes
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> > > > [2] https://lists.apache.org/thread/wopprvp3ww243mtw23nj59p57cghh7mc
> > > >
> > > > Best,
> > > > Fang Yong
> > > >
> > >
> >
>

Question around Flink's AdaptiveBatchScheduler

[Venkatakrishnan Sowrirajan]

Hi Flink devs,

With Flink's AdaptiveBatchScheduler

(Note:
this is different from AdaptiveScheduler
),
the scheduler automatically determines the correct number of downstream
tasks required to process the shuffle generated by the upstream vertex.

I have a question regarding the current behavior. There are 2 configs which
are in interplay here.
1. jobmanager.adaptive-batch-scheduler.default-source-parallelism

 - The default parallelism of data source.
2. jobmanager.adaptive-batch-scheduler.max-parallelism

-
Upper bound of allowed parallelism to set adaptively.

Currently, if "
jobmanager.adaptive-batch-scheduler.default-source-parallelism
"
is greater than "jobmanager.adaptive-batch-scheduler.max-parallelism
",
Flink application fails with the below message:

"Vertex's parallelism should be smaller than or equal to vertex's max
parallelism."

This is the corresponding code in Flink's DefaultVertexParallelismInfo
.
My question is, "default-source-parallelism" config should be independent
from the "max-parallelism" flag. The former controls the default source
parallelism while the latter controls the max number of partitions to write
the intermediate shuffle.

If this is true, then the above check should be fixed. Otherwise, wanted to
understand why the "default-source-parallelism` should be less than the
"max-parallelism"

Thanks
Venkat

[jira] [Created] (FLINK-34541) Flink uses insecure http confluent endpoint in its build

[PJ Fanning (Jira)]

PJ Fanning created FLINK-34541:
--

 Summary: Flink uses insecure http confluent endpoint in its build
 Key: FLINK-34541
 URL: https://issues.apache.org/jira/browse/FLINK-34541
 Project: Flink
  Issue Type: Improvement
  Components: Build System
Reporter: PJ Fanning


See 
https://github.com/apache/flink/blob/641f4f4d0d0156b84bdb9ba528b1dd96f7ae9d9c/flink-end-to-end-tests/test-scripts/kafka-common.sh#L55

Please use https instead.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener

[Márton Balassi]

+1 (binding)

Marton

On Wed, Feb 28, 2024 at 5:14 PM Gyula Fóra  wrote:

> +1 (binding)
>
> Gyula
>
> On Wed, Feb 28, 2024 at 11:10 AM Maciej Obuchowski  >
> wrote:
>
> > +1 (non-binding)
> >
> > Best,
> > Maciej Obuchowski
> >
> > śr., 28 lut 2024 o 10:29 Zhanghao Chen 
> > napisał(a):
> >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Zhanghao Chen
> > > 
> > > From: Yong Fang 
> > > Sent: Wednesday, February 28, 2024 10:12
> > > To: dev 
> > > Subject: [VOTE] FLIP-314: Support Customized Job Lineage Listener
> > >
> > > Hi devs,
> > >
> > > I would like to restart a vote about FLIP-314: Support Customized Job
> > > Lineage Listener[1].
> > >
> > > Previously, we added lineage related interfaces in FLIP-314. Before the
> > > interfaces were developed and merged into the master, @Maciej and
> > > @Zhenqiu provided valuable suggestions for the interface from the
> > > perspective of the lineage system. So we updated the interfaces of
> > FLIP-314
> > > and discussed them again in the discussion thread [2].
> > >
> > > So I am here to initiate a new vote on FLIP-314, the vote will be open
> > for
> > > at least 72 hours unless there is an objection or insufficient votes
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> > > [2] https://lists.apache.org/thread/wopprvp3ww243mtw23nj59p57cghh7mc
> > >
> > > Best,
> > > Fang Yong
> > >
> >
>

Re: [DISCUSS] Apache Bahir retired

[Martijn Visser]

Hi all,

+1 to have a connector FLIP to propose a Kudu connector. I'm +0 overall
because I don't see a lot of activity happening in newly proposed
connectors, but if there's demand for it and people want to volunteer with
contributions, there's no reason to block it.

Best regards,

Martijn

On Wed, Feb 28, 2024 at 4:31 PM Márton Balassi 
wrote:

> Hi team,
>
> Thanks for bringing this up, Feri. I am +1 for maintaining the Kudu
> connector as an external Flink connector.
>
> As per the legal/trademark questions this is actually fair game because one
> does not donate code to a specific Apache project, technically it is
> donated to the Apache Software foundation. Consequently moving between ASF
> projects is fine, I would add a line to the NOTICE file stating that this
> code originally lived in Bahir once we forked it.
>
> Although I did not find an easy to link precedent this is also implied in
> the Attic Bahir site [1] ("notify us if you fork outside Apache") and in
> this [2] Apache community dev list chat. We should notify the Attic team in
> any case. :-)
>
> [1] https://attic.apache.org/projects/bahir.html
> [2] https://lists.apache.org/thread/p31mz4x4dcvd43f026d5p05rpglzfyrt
>
> On Tue, Feb 27, 2024 at 10:09 AM Ferenc Csaky 
> wrote:
>
> > Thank you Leonard for sharing your thoughts on this topic.
> >
> > I agree that complying with the Flink community connector
> > development process would be a must, if there are no legal or
> > copyright issues, I would be happy to take that task for this
> > particular case.
> >
> > I am no legal/copyright expert myslef, but Bahir uses the Apache
> > 2.0 license as well, so I believe it should be possible without too many
> > complications, but I try to look for help on that front.
> >
> > FYI we are using and supporting a downstream fork of the Kudu connector
> on
> > top of Flink 1.18 without any major modifications, so it is pretty up to
> > date upstream as well.
> >
> > Regards,
> > Ferenc
> >
> >
> >
> >
> > On Monday, February 26th, 2024 at 10:29, Leonard Xu 
> > wrote:
> >
> > >
> > >
> > > Hey, Ferenc
> > >
> > > Thanks for initiating this discussion. Apache Bahir is a great project
> > that provided significant assistance to many Apache Flink/Spark users.
> It's
> > pity news that it has been retired.
> > >
> > > I believe that connectivity is crucial for building the ecosystem of
> the
> > Flink such a computing engine. The community, or at least I, would
> actively
> > support the introduction and maintenance of new connectors. Therefore,
> > adding a Kudu connector or other connectors from Bahir makes sense to me,
> > as long as we adhere to the development process for connectors in the
> Flink
> > community[1].
> > > I recently visited the Bahir Flink repository. Although the last
> release
> > of Bahir Flink was in August ’22[2] which is compatible with Flink 1.14,
> > its latest code is compatible with Flink 1.17[3]. So, based on the
> existing
> > codebase, developing an official Apache Flink connector for Kudu or other
> > connectors should be manageable. One point to consider is that if we're
> not
> > developing a connector entirely from scratch but based on an existing
> > repository, we must ensure that there are no copyright issues. Here, "no
> > issues" means satisfying both Apache Bahir's and Apache Flink's copyright
> > requirements. Honestly, I'm not an expert in copyright or legal matters.
> If
> > you're interested in contributing to the Kudu connector, it might be
> > necessary to attract other experienced community members to participate
> in
> > this aspect.
> > >
> > > Best,
> > > Leonard
> > >
> > > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+Connector+Template
> > > [2] https://github.com/apache/bahir-flink/releases/tag/v1.1.0
> > > [3] https://github.com/apache/bahir-flink/blob/master/pom.xml#L116
> > >
> > >
> > >
> > > > 2024年2月22日 下午6:37，Ferenc Csaky ferenc.cs...@pm.me.INVALID 写道：
> > > >
> > > > Hello devs,
> > > >
> > > > Just saw that the Bahir project is retired [1]. Any plans on what's
> > happening with the Flink connectors that were part of this project? We
> > specifically use the Kudu connector and integrate it to our platform at
> > Cloudera, so we would be okay to maintain it. Would it be possible to
> carry
> > it over as separate connector repu under the Apache umbrella similarly as
> > it happened with the external connectors previously?
> > > >
> > > > Thanks,
> > > > Ferenc
> >
>

[jira] [Created] (FLINK-34540) Tune number of task slots

[Maximilian Michels (Jira)]

Maximilian Michels created FLINK-34540:
--

 Summary: Tune number of task slots
 Key: FLINK-34540
 URL: https://issues.apache.org/jira/browse/FLINK-34540
 Project: Flink
  Issue Type: Sub-task
  Components: Autoscaler, Kubernetes Operator
Reporter: Maximilian Michels
Assignee: Maximilian Michels
 Fix For: kubernetes-operator-1.8.0


Adjustments similar to FLINK-34152, but simpler because we only need to adjust 
heap memory and metaspace for the JobManager.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34539) Tune JobManager memory of autoscaled jobs

[Maximilian Michels (Jira)]

Maximilian Michels created FLINK-34539:
--

 Summary: Tune JobManager memory of autoscaled jobs
 Key: FLINK-34539
 URL: https://issues.apache.org/jira/browse/FLINK-34539
 Project: Flink
  Issue Type: Sub-task
  Components: Autoscaler, Kubernetes Operator
Reporter: Maximilian Michels
Assignee: Maximilian Michels
 Fix For: kubernetes-operator-1.8.0


The current autoscaling algorithm adjusts the parallelism of the job task 
vertices according to the processing needs. By adjusting the parallelism, we 
systematically scale the amount of CPU for a task. At the same time, we also 
indirectly change the amount of memory tasks have at their dispense. However, 
there are some problems with this.
 # Memory is overprovisioned: On scale up we may add more memory than we 
actually need. Even on scale down, the memory / cpu ratio can still be off and 
too much memory is used.
 # Memory is underprovisioned: For stateful jobs, we risk running into 
OutOfMemoryErrors on scale down. Even before running out of memory, too little 
memory can have a negative impact on the effectiveness of the scaling.

We lack the capability to tune memory proportionally to the processing needs. 
In the same way that we measure CPU usage and size the tasks accordingly, we 
need to evaluate memory usage and adjust the heap memory size.

https://docs.google.com/document/d/19GXHGL_FvN6WBgFvLeXpDABog2H_qqkw1_wrpamkFSc/edit

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34538) Tune memory of autoscaled jobs

[Maximilian Michels (Jira)]

Maximilian Michels created FLINK-34538:
--

 Summary: Tune memory of autoscaled jobs
 Key: FLINK-34538
 URL: https://issues.apache.org/jira/browse/FLINK-34538
 Project: Flink
  Issue Type: New Feature
  Components: Autoscaler, Kubernetes Operator
Reporter: Maximilian Michels
Assignee: Maximilian Michels
 Fix For: kubernetes-operator-1.8.0


The current autoscaling algorithm adjusts the parallelism of the job task 
vertices according to the processing needs. By adjusting the parallelism, we 
systematically scale the amount of CPU for a task. At the same time, we also 
indirectly change the amount of memory tasks have at their dispense. However, 
there are some problems with this.
 # Memory is overprovisioned: On scale up we may add more memory than we 
actually need. Even on scale down, the memory / cpu ratio can still be off and 
too much memory is used.
 # Memory is underprovisioned: For stateful jobs, we risk running into 
OutOfMemoryErrors on scale down. Even before running out of memory, too little 
memory can have a negative impact on the effectiveness of the scaling.

We lack the capability to tune memory proportionally to the processing needs. 
In the same way that we measure CPU usage and size the tasks accordingly, we 
need to evaluate memory usage and adjust the heap memory size.

https://docs.google.com/document/d/19GXHGL_FvN6WBgFvLeXpDABog2H_qqkw1_wrpamkFSc/edit

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener

[Gyula Fóra]

+1 (binding)

Gyula

On Wed, Feb 28, 2024 at 11:10 AM Maciej Obuchowski 
wrote:

> +1 (non-binding)
>
> Best,
> Maciej Obuchowski
>
> śr., 28 lut 2024 o 10:29 Zhanghao Chen 
> napisał(a):
>
> > +1 (non-binding)
> >
> > Best,
> > Zhanghao Chen
> > 
> > From: Yong Fang 
> > Sent: Wednesday, February 28, 2024 10:12
> > To: dev 
> > Subject: [VOTE] FLIP-314: Support Customized Job Lineage Listener
> >
> > Hi devs,
> >
> > I would like to restart a vote about FLIP-314: Support Customized Job
> > Lineage Listener[1].
> >
> > Previously, we added lineage related interfaces in FLIP-314. Before the
> > interfaces were developed and merged into the master, @Maciej and
> > @Zhenqiu provided valuable suggestions for the interface from the
> > perspective of the lineage system. So we updated the interfaces of
> FLIP-314
> > and discussed them again in the discussion thread [2].
> >
> > So I am here to initiate a new vote on FLIP-314, the vote will be open
> for
> > at least 72 hours unless there is an objection or insufficient votes
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> > [2] https://lists.apache.org/thread/wopprvp3ww243mtw23nj59p57cghh7mc
> >
> > Best,
> > Fang Yong
> >
>

Re: [DISCUSS] Apache Bahir retired

[Márton Balassi]

Hi team,

Thanks for bringing this up, Feri. I am +1 for maintaining the Kudu
connector as an external Flink connector.

As per the legal/trademark questions this is actually fair game because one
does not donate code to a specific Apache project, technically it is
donated to the Apache Software foundation. Consequently moving between ASF
projects is fine, I would add a line to the NOTICE file stating that this
code originally lived in Bahir once we forked it.

Although I did not find an easy to link precedent this is also implied in
the Attic Bahir site [1] ("notify us if you fork outside Apache") and in
this [2] Apache community dev list chat. We should notify the Attic team in
any case. :-)

[1] https://attic.apache.org/projects/bahir.html
[2] https://lists.apache.org/thread/p31mz4x4dcvd43f026d5p05rpglzfyrt

On Tue, Feb 27, 2024 at 10:09 AM Ferenc Csaky 
wrote:

> Thank you Leonard for sharing your thoughts on this topic.
>
> I agree that complying with the Flink community connector
> development process would be a must, if there are no legal or
> copyright issues, I would be happy to take that task for this
> particular case.
>
> I am no legal/copyright expert myslef, but Bahir uses the Apache
> 2.0 license as well, so I believe it should be possible without too many
> complications, but I try to look for help on that front.
>
> FYI we are using and supporting a downstream fork of the Kudu connector on
> top of Flink 1.18 without any major modifications, so it is pretty up to
> date upstream as well.
>
> Regards,
> Ferenc
>
>
>
>
> On Monday, February 26th, 2024 at 10:29, Leonard Xu 
> wrote:
>
> >
> >
> > Hey, Ferenc
> >
> > Thanks for initiating this discussion. Apache Bahir is a great project
> that provided significant assistance to many Apache Flink/Spark users. It's
> pity news that it has been retired.
> >
> > I believe that connectivity is crucial for building the ecosystem of the
> Flink such a computing engine. The community, or at least I, would actively
> support the introduction and maintenance of new connectors. Therefore,
> adding a Kudu connector or other connectors from Bahir makes sense to me,
> as long as we adhere to the development process for connectors in the Flink
> community[1].
> > I recently visited the Bahir Flink repository. Although the last release
> of Bahir Flink was in August ’22[2] which is compatible with Flink 1.14,
> its latest code is compatible with Flink 1.17[3]. So, based on the existing
> codebase, developing an official Apache Flink connector for Kudu or other
> connectors should be manageable. One point to consider is that if we're not
> developing a connector entirely from scratch but based on an existing
> repository, we must ensure that there are no copyright issues. Here, "no
> issues" means satisfying both Apache Bahir's and Apache Flink's copyright
> requirements. Honestly, I'm not an expert in copyright or legal matters. If
> you're interested in contributing to the Kudu connector, it might be
> necessary to attract other experienced community members to participate in
> this aspect.
> >
> > Best,
> > Leonard
> >
> > [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+Connector+Template
> > [2] https://github.com/apache/bahir-flink/releases/tag/v1.1.0
> > [3] https://github.com/apache/bahir-flink/blob/master/pom.xml#L116
> >
> >
> >
> > > 2024年2月22日 下午6:37，Ferenc Csaky ferenc.cs...@pm.me.INVALID 写道：
> > >
> > > Hello devs,
> > >
> > > Just saw that the Bahir project is retired [1]. Any plans on what's
> happening with the Flink connectors that were part of this project? We
> specifically use the Kudu connector and integrate it to our platform at
> Cloudera, so we would be okay to maintain it. Would it be possible to carry
> it over as separate connector repu under the Apache umbrella similarly as
> it happened with the external connectors previously?
> > >
> > > Thanks,
> > > Ferenc
>

Re: [DISCUSS] Move CheckpointingMode to flink-core

[Xintong Song]

Personally, I'd be in favor of option 2. And based on the fact that
migrating from the deprecated CheckpointingMode to the new one takes barely
any effort (simply re-import the class), I'd be fine with removing the
deprecated class in 2.0.

But I'd also be fine with the other options.

Either way, agree that we should not block 1.19 on this.

Best,

Xintong



On Wed, Feb 28, 2024 at 6:16 PM Junrui Lee  wrote:

> Hi Zakelly,
>
> +1 for option 1. I prefer to minimize unnecessary additional development
> and discussions due to internal code relocations and to avoid imposing
> migration costs on users.
>
> Best regards,
> Junrui
>
> Zakelly Lan  于2024年2月28日周三 14:46写道：
>
> > Hi Lincoln,
> >
> > Given that we have finished the testing for 1.19, I agree it is better
> not
> > merge this into 1.19. Thanks for RMs' attention!
> >
> > Hi Chesney and Junrui,
> >
> > Thanks for your advice. My original intention is to move the class as
> well
> > as change the package to make it clean. But it involves much more effort.
> > Here are several options we have:
> >
> >1. Move CheckpointingMode to flink-core and keep the same package. No
> >more deprecation and API changes. But it will leave a
> >'org.apache.flink.streaming.api' package in flink-core.
> >2. Introduce new CheckpointingMode in package
> >'org.apache.flink.core.execution' and deprecate the old one. Deprecate
> > the
> >corresponding getter/setter of 'CheckpointConfig' and introduce new
> ones
> >with a similar but different name (e.g. set/getCheckpointMode). We
> will
> >discuss the removal of those deprecation later in 2.x.
> >3. Based on 1, move CheckpointingMode to package
> >'org.apache.flink.core.execution' in 2.0. This is a breaking change
> that
> >needs more discussion.
> >
> > Both ways work. I'm slightly inclined to option 1, or option 3 if we all
> > agree, since the new getter/setter may also bring in confusions thus we
> > cannot make the API purely clean. WDYT?
> >
> >
> > Best,
> > Zakelly
> >
> > On Wed, Feb 28, 2024 at 10:14 AM Junrui Lee  wrote:
> >
> > > Hi Zakelly,
> > >
> > > I agree with Chesnay's response. I would suggest that during the
> process
> > of
> > > moving CheckpointingMode from the flink-streaming-java module to the
> > > flink-core module, we should keep the package name unchanged. This
> > approach
> > > would be completely transparent to users. In fact, this practice should
> > be
> > > applicable to many of our desired moves from flink-streaming-java to
> > > higher-level modules, such as flink-runtime and flink-core.
> > >
> > > Best,
> > > Junrui
> > >
> > > Chesnay Schepler  于2024年2月28日周三 05:18写道：
> > >
> > > > Moving classes (== keep the same package) to a module higher up in
> the
> > > > dependency tree should not be a breaking change and can imo be done
> > > > anytime without any risk to users.
> > > >
> > > > On 27/02/2024 17:01, Lincoln Lee wrote:
> > > > > Hi Zakelly,
> > > > >
> > > > > Thanks for letting us 1.19 RMs know about this!
> > > > >
> > > > > This change has been discussed during today's release sync meeting,
> > we
> > > > > suggest not merge it into 1.19.
> > > > > We can continue discussing the removal in 2.x separately.
> > > > >
> > > > > Best,
> > > > > Lincoln Lee
> > > > >
> > > > >
> > > > > Hangxiang Yu  于2024年2月27日周二 11:28写道：
> > > > >
> > > > >> Hi, Zakelly.
> > > > >> Thanks for driving this.
> > > > >> Moving this class to flink-core makes sense to me which could make
> > the
> > > > code
> > > > >> path and configs clearer.
> > > > >> It's marked as @Public from 1.0 and 1.20 should be the next
> > long-term
> > > > >> version, so 1.19 should have been a suitable version to do it.
> > > > >> And also look forward to thoughts of other developers/RMs since
> 1.19
> > > is
> > > > >> currently under a feature freeze status.
> > > > >>
> > > > >> On Mon, Feb 26, 2024 at 6:42 PM Zakelly Lan <
> zakelly@gmail.com>
> > > > wrote:
> > > > >>
> > > > >>> Hi devs,
> > > > >>>
> > > > >>> When working on the FLIP-406[1], I realized that moving all
> options
> > > of
> > > > >>> ExecutionCheckpointingOptions(flink-streaming-java) to
> > > > >>> CheckpointingOptions(flink-core) depends on relocating the
> > > > >>> enum CheckpointingMode(flink-streaming-java) to flink-core
> module.
> > > > >> However,
> > > > >>> the CheckpointingMode is annotated as @Public and used by
> > datastream
> > > > api
> > > > >>> like 'CheckpointConfig#setCheckpointingMode'. So I'd like to
> start
> > a
> > > > >>> discussion on moving the CheckpointingMode to flink-core. It is
> in
> > a
> > > > >> little
> > > > >>> bit of a hurry if we want the old enum to be entirely removed in
> > > Flink
> > > > >> 2.x
> > > > >>> series, since the deprecation should be shipped in the upcoming
> > Flink
> > > > >> 1.19.
> > > > >>> I suggest not creating a dedicated FLIP and treating this as a
> > > sub-task
> > > > >> of
> > > > >>> FLIP-406.
> > > > >>>
> > > > >>> I prepared a 

Re: Flink kinesis connector v4.2.0-1.18 has issue when consuming from EFO consumer

[Aleksandr Pilipenko]

Hi,

Could you please provide more information on the error you are observing?
Attached file does not have anything related to Kinesis or any errors.

Best,
Aleksandr

On Wed, 28 Feb 2024 at 02:28, Xiaolong Wang
 wrote:

> Hi,
>
> I used the flink-connector-kinesis (4.0.2-1.18) to consume from Kinesis.
> The job can start but will fail within 1 hour. Detailed error log
> is attached.
>
> When I changed the version of the flink-connector-kinesis to `1.15.2` ,
> everything settled.
>
> Any idea to fix it ?
>
>

[jira] [Created] (FLINK-34537) Autoscaler JDBC

[ConradJam (Jira)]

ConradJam created FLINK-34537:
-

 Summary: Autoscaler JDBC
 Key: FLINK-34537
 URL: https://issues.apache.org/jira/browse/FLINK-34537
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Affects Versions: 1.8.0
Reporter: ConradJam
 Fix For: 1.8.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Move CheckpointingMode to flink-core

[Junrui Lee]

Hi Zakelly,

+1 for option 1. I prefer to minimize unnecessary additional development
and discussions due to internal code relocations and to avoid imposing
migration costs on users.

Best regards,
Junrui

Zakelly Lan  于2024年2月28日周三 14:46写道：

> Hi Lincoln,
>
> Given that we have finished the testing for 1.19, I agree it is better not
> merge this into 1.19. Thanks for RMs' attention!
>
> Hi Chesney and Junrui,
>
> Thanks for your advice. My original intention is to move the class as well
> as change the package to make it clean. But it involves much more effort.
> Here are several options we have:
>
>1. Move CheckpointingMode to flink-core and keep the same package. No
>more deprecation and API changes. But it will leave a
>'org.apache.flink.streaming.api' package in flink-core.
>2. Introduce new CheckpointingMode in package
>'org.apache.flink.core.execution' and deprecate the old one. Deprecate
> the
>corresponding getter/setter of 'CheckpointConfig' and introduce new ones
>with a similar but different name (e.g. set/getCheckpointMode). We will
>discuss the removal of those deprecation later in 2.x.
>3. Based on 1, move CheckpointingMode to package
>'org.apache.flink.core.execution' in 2.0. This is a breaking change that
>needs more discussion.
>
> Both ways work. I'm slightly inclined to option 1, or option 3 if we all
> agree, since the new getter/setter may also bring in confusions thus we
> cannot make the API purely clean. WDYT?
>
>
> Best,
> Zakelly
>
> On Wed, Feb 28, 2024 at 10:14 AM Junrui Lee  wrote:
>
> > Hi Zakelly,
> >
> > I agree with Chesnay's response. I would suggest that during the process
> of
> > moving CheckpointingMode from the flink-streaming-java module to the
> > flink-core module, we should keep the package name unchanged. This
> approach
> > would be completely transparent to users. In fact, this practice should
> be
> > applicable to many of our desired moves from flink-streaming-java to
> > higher-level modules, such as flink-runtime and flink-core.
> >
> > Best,
> > Junrui
> >
> > Chesnay Schepler  于2024年2月28日周三 05:18写道：
> >
> > > Moving classes (== keep the same package) to a module higher up in the
> > > dependency tree should not be a breaking change and can imo be done
> > > anytime without any risk to users.
> > >
> > > On 27/02/2024 17:01, Lincoln Lee wrote:
> > > > Hi Zakelly,
> > > >
> > > > Thanks for letting us 1.19 RMs know about this!
> > > >
> > > > This change has been discussed during today's release sync meeting,
> we
> > > > suggest not merge it into 1.19.
> > > > We can continue discussing the removal in 2.x separately.
> > > >
> > > > Best,
> > > > Lincoln Lee
> > > >
> > > >
> > > > Hangxiang Yu  于2024年2月27日周二 11:28写道：
> > > >
> > > >> Hi, Zakelly.
> > > >> Thanks for driving this.
> > > >> Moving this class to flink-core makes sense to me which could make
> the
> > > code
> > > >> path and configs clearer.
> > > >> It's marked as @Public from 1.0 and 1.20 should be the next
> long-term
> > > >> version, so 1.19 should have been a suitable version to do it.
> > > >> And also look forward to thoughts of other developers/RMs since 1.19
> > is
> > > >> currently under a feature freeze status.
> > > >>
> > > >> On Mon, Feb 26, 2024 at 6:42 PM Zakelly Lan 
> > > wrote:
> > > >>
> > > >>> Hi devs,
> > > >>>
> > > >>> When working on the FLIP-406[1], I realized that moving all options
> > of
> > > >>> ExecutionCheckpointingOptions(flink-streaming-java) to
> > > >>> CheckpointingOptions(flink-core) depends on relocating the
> > > >>> enum CheckpointingMode(flink-streaming-java) to flink-core module.
> > > >> However,
> > > >>> the CheckpointingMode is annotated as @Public and used by
> datastream
> > > api
> > > >>> like 'CheckpointConfig#setCheckpointingMode'. So I'd like to start
> a
> > > >>> discussion on moving the CheckpointingMode to flink-core. It is in
> a
> > > >> little
> > > >>> bit of a hurry if we want the old enum to be entirely removed in
> > Flink
> > > >> 2.x
> > > >>> series, since the deprecation should be shipped in the upcoming
> Flink
> > > >> 1.19.
> > > >>> I suggest not creating a dedicated FLIP and treating this as a
> > sub-task
> > > >> of
> > > >>> FLIP-406.
> > > >>>
> > > >>> I prepared a minimal change of providing new APIs and deprecating
> the
> > > old
> > > >>> ones[2], which could be merged to 1.19 if we agree to do so.
> > > >>>
> > > >>> Looking forward to your thoughts! Also cc RMs of 1.19 about this.
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
> > > >>> [2]
> > > >>>
> > > >>>
> > > >>
> > >
> >
> https://github.com/apache/flink/commit/9bdd237d0322df8853f1b9e6ae658f77b9175237
> > > >>> Best,
> > > >>> Zakelly
> > > >>>
> > > >>
> > > >> --
> > > >> Best,
> > > >> Hangxiang.
> > > >>
> > >
> > >
> >
>

Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener

[Maciej Obuchowski]

+1 (non-binding)

Best,
Maciej Obuchowski

śr., 28 lut 2024 o 10:29 Zhanghao Chen 
napisał(a):

> +1 (non-binding)
>
> Best,
> Zhanghao Chen
> 
> From: Yong Fang 
> Sent: Wednesday, February 28, 2024 10:12
> To: dev 
> Subject: [VOTE] FLIP-314: Support Customized Job Lineage Listener
>
> Hi devs,
>
> I would like to restart a vote about FLIP-314: Support Customized Job
> Lineage Listener[1].
>
> Previously, we added lineage related interfaces in FLIP-314. Before the
> interfaces were developed and merged into the master, @Maciej and
> @Zhenqiu provided valuable suggestions for the interface from the
> perspective of the lineage system. So we updated the interfaces of FLIP-314
> and discussed them again in the discussion thread [2].
>
> So I am here to initiate a new vote on FLIP-314, the vote will be open for
> at least 72 hours unless there is an objection or insufficient votes
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> [2] https://lists.apache.org/thread/wopprvp3ww243mtw23nj59p57cghh7mc
>
> Best,
> Fang Yong
>

Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener

[Zhanghao Chen]

+1 (non-binding)

Best,
Zhanghao Chen

From: Yong Fang 
Sent: Wednesday, February 28, 2024 10:12
To: dev 
Subject: [VOTE] FLIP-314: Support Customized Job Lineage Listener

Hi devs,

I would like to restart a vote about FLIP-314: Support Customized Job
Lineage Listener[1].

Previously, we added lineage related interfaces in FLIP-314. Before the
interfaces were developed and merged into the master, @Maciej and
@Zhenqiu provided valuable suggestions for the interface from the
perspective of the lineage system. So we updated the interfaces of FLIP-314
and discussed them again in the discussion thread [2].

So I am here to initiate a new vote on FLIP-314, the vote will be open for
at least 72 hours unless there is an objection or insufficient votes

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
[2] https://lists.apache.org/thread/wopprvp3ww243mtw23nj59p57cghh7mc

Best,
Fang Yong

Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener

[Yangze Guo]

+1 (binding)

Best,
Yangze Guo

On Wed, Feb 28, 2024 at 11:54 AM Peter Huang  wrote:
>
> +1 (non-binding)
>
> Thanks for consolidating the opinions from both Flink and OpenLineage
> communities. Look forward to the collaboration.
>
>
> Best Regards
> Peter Huang
>
> On Tue, Feb 27, 2024 at 6:13 PM Yong Fang  wrote:
>
> > Hi devs,
> >
> > I would like to restart a vote about FLIP-314: Support Customized Job
> > Lineage Listener[1].
> >
> > Previously, we added lineage related interfaces in FLIP-314. Before the
> > interfaces were developed and merged into the master, @Maciej and
> > @Zhenqiu provided valuable suggestions for the interface from the
> > perspective of the lineage system. So we updated the interfaces of FLIP-314
> > and discussed them again in the discussion thread [2].
> >
> > So I am here to initiate a new vote on FLIP-314, the vote will be open for
> > at least 72 hours unless there is an objection or insufficient votes
> >
> > [1]
> >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> > [2] https://lists.apache.org/thread/wopprvp3ww243mtw23nj59p57cghh7mc
> >
> > Best,
> > Fang Yong
> >

[RESULT] [VOTE] flink-connector-parent 1.1.0, release candidate #2

[Etienne Chauchot]

Hi everyone,

I'm happy to announce that we have unanimously approved this release.

There are 8 approving votes, 3 of which are binding:

 * Qingsheng Ren (binding)
 * Rui Fan (non-binding)
 * Leonard Xu (binding)
 * Hang Ruan (non-binding)
 * Sergey Nuyanzin (non-binding)
 * Zhongqiang Gong (non-binding)
 * Yanquan Lv (non-binding)
 * Chesnay Schepler (binding)


There are no disapproving votes.

Thanks everyone!

Best

Etienne

Re: [VOTE] Release flink-connector-parent 1.1.0 release candidate #2

[Etienne Chauchot]

Hi all,

The vote on flink-connector-parent 1.1.0 RC2 is now closed. The result 
will be announced in a separate email.


Best

Etienne

Le 27/02/2024 à 12:23, Chesnay Schepler a écrit :

+1
- pom contents
- source contents
- Website PR

On 19/02/2024 18:33, Etienne Chauchot wrote:

Hi everyone,
Please review and vote on the release candidate #2 for the version 
1.1.0, as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to 
dist.apache.org [2], which are signed with the key with fingerprint 
D1A76BA19D6294DD0033F6843A019F0B8DD163EA [3],

* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag v1.1.0-rc2 [5],
* website pull request listing the new release [6].

* confluence wiki: connector parent upgrade to version 1.1.0 that 
will be validated after the artifact is released (there is no PR 
mechanism on the wiki) [7]



The vote will be open for at least 72 hours. It is adopted by 
majority approval, with at least 3 PMC affirmative votes.


Thanks,
Etienne

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353442
[2] 
https://dist.apache.org/repos/dist/dev/flink/flink-connector-parent-1.1.0-rc2

[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] 
https://repository.apache.org/content/repositories/orgapacheflink-1707
[5] 
https://github.com/apache/flink-connector-shared-utils/releases/tag/v1.1.0-rc2


[6] https://github.com/apache/flink-web/pull/717

[7] 
https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development

[jira] [Created] (FLINK-34536) Support reading long value as Timestamp column in JSON format

[yisha zhou (Jira)]

yisha zhou created FLINK-34536:
--

 Summary: Support reading long value as Timestamp column in JSON 
format
 Key: FLINK-34536
 URL: https://issues.apache.org/jira/browse/FLINK-34536
 Project: Flink
  Issue Type: Improvement
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.19.0
Reporter: yisha zhou


In many scenarios, timestamp data is stored as Long value and expected to be 
operated as Timestamp value. It's not user-friendly to use an UDF to convert 
the data before operating it.

Meanwhile, in Avro format, it seems it can receive several types of input and 
convert it into TimestampData. Hope the same ability can be introduced into 
JSON format.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)