[jira] [Created] (FLINK-34007) Flink Job stuck in suspend state after recovery from failure in HA Mode

[Zhenqiu Huang (Jira)]

Zhenqiu Huang created FLINK-34007:
-

 Summary: Flink Job stuck in suspend state after recovery from 
failure in HA Mode
 Key: FLINK-34007
 URL: https://issues.apache.org/jira/browse/FLINK-34007
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.18.1, 1.18.2
Reporter: Zhenqiu Huang


The observation is that Job manager goes to suspend state with a failed 
container not able to register itself to resource manager after timeout.

JM Log:

2024-01-04 02:58:39,210 INFO  
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner [] - 
JobMasterServiceLeadershipRunner for job 217cee964b2cfdc3115fb74cac0ec550 was 
revoked leadership with leader id eda6fee6-ce02-4076-9a99-8c43a92629f7. 
Stopping current JobMasterServiceProcess.
2024-01-04 02:58:58,347 INFO  
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - 
http://172.16.71.11:8081 lost leadership
2024-01-04 02:58:58,347 INFO  
org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl [] - 
Resource manager service is revoked leadership with session id 
eda6fee6-ce02-4076-9a99-8c43a92629f7.
2024-01-04 02:58:58,348 INFO  
org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunner [] - 
DefaultDispatcherRunner was revoked the leadership with leader id 
eda6fee6-ce02-4076-9a99-8c43a92629f7. Stopping the DispatcherLeaderProcess.
2024-01-04 02:58:58,348 INFO  
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - 
Stopping SessionDispatcherLeaderProcess.
2024-01-04 02:58:58,349 INFO  
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Stopping 
dispatcher pekko.tcp://flink@172.16.71.11:6123/user/rpc/dispatcher_1.
2024-01-04 02:58:58,349 INFO  org.apache.flink.runtime.jobmaster.JobMaster  
   [] - Stopping the JobMaster for job 
'amp-ade-fitness-clickstream-projection-uat' (217cee964b2cfdc3115fb74cac0ec550).
2024-01-04 02:58:58,349 INFO  
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Stopping all 
currently running jobs of dispatcher 
pekko.tcp://flink@172.16.71.11:6123/user/rpc/dispatcher_1.
2024-01-04 02:58:58,351 INFO  
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job 
217cee964b2cfdc3115fb74cac0ec550 reached terminal state SUSPENDED.
2024-01-04 02:58:58,352 INFO  
org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - 
Stopping credential renewal
2024-01-04 02:58:58,352 INFO  
org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - 
Stopped credential renewal
2024-01-04 02:58:58,352 INFO  
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] 
- Closing the slot manager.
2024-01-04 02:58:58,351 INFO  
org.apache.flink.runtime.executiongraph.ExecutionGraph   [] - Job 
amp-ade-fitness-clickstream-projection-uat (217cee964b2cfdc3115fb74cac0ec550) 
switched from state RUNNING to SUSPENDED.
org.apache.flink.util.FlinkException: AdaptiveScheduler is being stopped.
at 
org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.closeAsync(AdaptiveScheduler.java:474)
 ~[flink-dist-1.18.1.6-ase.jar:1.18.1.6-ase]
at 
org.apache.flink.runtime.jobmaster.JobMaster.stopScheduling(JobMaster.java:1093)
 ~[flink-dist-1.18.1.6-ase.jar:1.18.1.6-ase]
at 
org.apache.flink.runtime.jobmaster.JobMaster.stopJobExecution(JobMaster.java:1056)
 ~[flink-dist-1.18.1.6-ase.jar:1.18.1.6-ase]
at 
org.apache.flink.runtime.jobmaster.JobMaster.onStop(JobMaster.java:454) 
~[flink-dist-1.18.1.6-ase.jar:1.18.1.6-ase]
at 
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:239)
 ~[flink-dist-1.18.1.6-ase.jar:1.18.1.6-ase]
at 
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor$StartedState.lambda$terminate$0(PekkoRpcActor.java:574)
 ~[flink-rpc-akkadb952114-fa83-4aba-b20a-b7e5771ce59c.jar:1.18.1.6-ase]
at 
org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
 ~[flink-dist-1.18.1.6-ase.jar:1.18.1.6-ase]
at 
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor$StartedState.terminate(PekkoRpcActor.java:573)
 ~[flink-rpc-akkadb952114-fa83-4aba-b20a-b7e5771ce59c.jar:1.18.1.6-ase]
at 
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleControlMessage(PekkoRpcActor.java:196)
 ~[flink-rpc-akkadb952114-fa83-4aba-b20a-b7e5771ce59c.jar:1.18.1.6-ase]
at 
org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33) 
[flink-rpc-akkadb952114-fa83-4aba-b20a-b7e5771ce59c.jar:1.18.1.6-ase]
at 
org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29) 
[flink-rpc-akkadb952114-fa83-4aba-b20a-b7e5771ce59c.jar:1.18.1.6-ase]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) 
[flink-rpc-akkadb952114-fa83-4aba-b20a-b7e5771ce59c.jar:1.18.1.6-ase]

[jira] [Created] (FLINK-34006) Flink terminates the execution of an application when there is a network problem between TaskManagers

[Sophia (Jira)]

Sophia created FLINK-34006:
--

 Summary: Flink terminates the execution of an application when 
there is a network problem between TaskManagers
 Key: FLINK-34006
 URL: https://issues.apache.org/jira/browse/FLINK-34006
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing, Runtime / Task
Affects Versions: 1.17.1
Reporter: Sophia


Flink terminates an application when two TaskManager are disconnected although 
there are enough resources in the cluster to run the application and we use 
checkpoint restart.

 

We deploy Flink v(1.17.1) on a cluster of six nodes with Ubuntu 18.04, the 
cluster consists of a JobManager and five TaskManagers. We use Flink's 
Standalone resource manager. We set the number of slots per TaskManager to one, 
and submit a WordCount application with a level of parallelism equal to three. 
We enable Flink checkpointing and restart failover strategy to attempt a 
restart in case of failure three times before termination and the time between 
attempts to 10 seconds.

The application starts running on the first 3 TaskManager.

If the communication is broken between two of the TaskManager that run the 
application, the job fails, and the JobManager tries to restart the job again. 
When the job fails the resources on the TaskManager are free. When the 
JobManager restarts the job, it selects the same three TaskManager it choose in 
the first attempt, and the job fails again. After three trials, Flink 
terminates the job with an exception: Connecting to remote task manager has 
failed.

 

These are the JobManager Configurations:
 * taskmanager.numberOfTaskSlots: 1
 * Enable checkpointing: --checkpointing
 * execution.checkpointing.interval: 3min
 * Enabling restart failover strategy
 * restart-strategy.type: fixed-delay
 * restart-strategy.fixed-delay.attempts: 3
 * restart-strategy.fixed-delay.delay: 10 s

 

command: ./bin/flink run -p 3 examples/streaming/WordCount.jar --checkpointing 
--input ~/flink/alice.txt



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34005) Implement restore tests for MiniBatchAssigner node

[Bonnie Varghese (Jira)]

Bonnie Varghese created FLINK-34005:
---

 Summary: Implement restore tests for MiniBatchAssigner node
 Key: FLINK-34005
 URL: https://issues.apache.org/jira/browse/FLINK-34005
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Bonnie Varghese
Assignee: Bonnie Varghese






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-398: Improve Serialization Configuration And Usage In Flink

[Ken Krugler]

Hi Fang Yong,

Thanks for the response, and I understand the desire to limit the impact of 
this FLIP.

I guess I should spend the time to start a new FLIP on switching to Fury, which 
could include cleaning up method names.

In the context of “facilitate user understanding”, one aspect of this cleanup 
is the current ExecutionConfig.enable/disable/hasGenericTypes() methods.

These are inconsistent with the current xxxKryo() methods, and cause confusion 
whenever I’m teaching a Flink course :)

Regards,

— Ken




> On Jan 4, 2024, at 6:40 PM, Yong Fang  wrote:
> 
> Hi Ken,
> 
> Sorry for the late reply. After discussing with @Xintong, we think it is 
> better to keep the method names in the FLIP mainly for the following reasons:
> 
> 1. This FLIP is mainly to support the configurable serializer while keeping 
> consistent with Flink at the semantic layer. Keeping the existing naming 
> rules can facilitate user understanding. 
> 
> 2. In the future, if Flink can choose Fury as the generic serializer, we can 
> update the corresponding methods in that FLIP after the discussion of Fury is 
> completed. This will be a minor modification, and we can avoid over-design in 
> the current FLIP.
> 
> Thanks for your feedback!
> 
> Best,
> Fang Yong
> 
> On Fri, Dec 29, 2023 at 12:38 PM Ken Krugler  > wrote:
>> Hi Xintong,
>> 
>> I agree that decoupling from Kryo is a bigger topic, well beyond the scope 
>> of this FLIP.
>> 
>> The reason I’d brought up Fury is that this increases my confidence that 
>> Flink will want to decouple from Kryo sooner rather than later.
>> 
>> So I feel it would be worth investing in a (minor) name change now, to 
>> improve that migration path in the future. Thus my suggestion for avoiding 
>> the explicit use of Kryo in method names.
>> 
>> Regards,
>> 
>> — Ken
>> 
>> 
>> 
>> 
>> > On Dec 17, 2023, at 7:16 PM, Xintong Song > > > wrote:
>> > 
>> > Hi Ken,
>> > 
>> > I think the main purpose of this FLIP is to change how users interact with
>> > the knobs for customizing the serialization behaviors, from requiring code
>> > changes to working with pure configurations. Redesigning the knobs (i.e.,
>> > names, semantics, etc.), on the other hand, is not the purpose of this
>> > FLIP. Preserving the existing names and semantics should also help minimize
>> > the migration cost for existing users. Therefore, I'm in favor of not
>> > changing them.
>> > 
>> > Concerning decoupling from Kryo, and introducing other serialization
>> > frameworks like Fury, I think that's a bigger topic that is worth further
>> > discussion. At the moment, I'm not aware of any community consensus on
>> > doing so. And even if in the future we decide to do so, the changes needed
>> > should be the same w/ or w/o this FLIP. So I'd suggest not to block this
>> > FLIP on these issues.
>> > 
>> > WDYT?
>> > 
>> > Best,
>> > 
>> > Xintong
>> > 
>> > 
>> > 
>> > On Fri, Dec 15, 2023 at 1:40 AM Ken Krugler > > >
>> > wrote:
>> > 
>> >> Hi Yong,
>> >> 
>> >> Looks good, thanks for creating this.
>> >> 
>> >> One comment - related to my recent email about Fury, I would love to see
>> >> the v2 serialization decoupled from Kryo.
>> >> 
>> >> As part of that, instead of using xxxKryo in methods, call them 
>> >> xxxGeneric.
>> >> 
>> >> A more extreme change would be to totally rely on Fury (so no more POJO
>> >> serializer). Fury is faster than the POJO serializer in my tests, but this
>> >> would be a much bigger change.
>> >> 
>> >> Though it could dramatically simplify the Flink serialization support.
>> >> 
>> >> — Ken
>> >> 
>> >> PS - a separate issue is how to migrate state from Kryo to something like
>> >> Fury, which supports schema evolution. I think this might be possible, by
>> >> having a smarter deserializer that identifies state as being created by
>> >> Kryo, and using (shaded) Kryo to deserialize, while still writing as Fury.
>> >> 
>> >>> On Dec 6, 2023, at 6:35 PM, Yong Fang > >>> > wrote:
>> >>> 
>> >>> Hi devs,
>> >>> 
>> >>> I'd like to start a discussion about FLIP-398: Improve Serialization
>> >>> Configuration And Usage In Flink [1].
>> >>> 
>> >>> Currently, users can register custom data types and serializers in Flink
>> >>> jobs through various methods, including registration in code,
>> >>> configuration, and annotations. These lead to difficulties in upgrading
>> >>> Flink jobs and priority issues.
>> >>> 
>> >>> In flink-2.0 we would like to manage job data types and serializers
>> >> through
>> >>> configurations. This FLIP will introduce a unified option for data type
>> >> and
>> >>> serializer and users can configure all custom data types and
>> >>> pojo/kryo/custom serializers. In addition, this FLIP will add more
>> >> built-in
>> >>> serializers for complex data types such as List and Map, and optimize the
>> >>> management of Avro Serializers.
>> >>> 
>> 

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

[Lu Niu]

Thank you Dong and Xuannan!

Yes. We can take on this task. Any help during bootstrapping would be
greatly appreciated! I realize there is already a voting thread "[VOTE]
FLIP-329: Add operator attribute to specify support for object-reuse". What
else do we need?

Best
Lu

On Fri, Jan 5, 2024 at 12:46 AM Xuannan Su  wrote:

> Hi Lu,
>
> I believe this feature is very useful. However, I currently lack the
> capacity to work on it in the near future. I think it would be great
> if you could take on the task. I am willing to offer assistance if
> there are any questions about the FLIP, or to review the PR if needed.
>
> Please let me know if you are interested in taking over this task. And
> also think that we should start the voting thread if no future
> comments on this FLIP.
>
> Best,
> Xuannan
>
>
>
> On Fri, Jan 5, 2024 at 2:23 PM Dong Lin  wrote:
> >
> > Hi Lu,
> >
> > I am not actively working on Flink and this JIRA recently. If Xuannan
> does not plan to work on this anytime soon, I personally think it will be
> great if you can help work on this FLIP. Maybe we can start the voting
> thread if there is no further comment on this FLIP.
> >
> > Xuannan, what do you think?
> >
> > Thanks,
> > Dong
> >
> >
> > On Fri, Jan 5, 2024 at 2:03 AM Lu Niu  wrote:
> >>
> >> Hi,
> >>
> >> Is this still under active development? I notice
> https://issues.apache.org/jira/browse/FLINK-32476 is labeled as
> deprioritized. If this is the case, would it be acceptable for us to take
> on the task?
> >>
> >> Best
> >> Lu
> >>
> >>
> >>
> >> On Thu, Oct 19, 2023 at 4:26 PM Ken Krugler <
> kkrugler_li...@transpac.com> wrote:
> >>>
> >>> Hi Dong,
> >>>
> >>> Sorry for not seeing this initially. I did have one question about the
> description of the issue in the FLIP:
> >>>
> >>> > However, in cases where the upstream and downstream operators do not
> store or access references to the input or output records, this deep-copy
> overhead becomes unnecessary
> >>>
> >>> I was interested in getting clarification as to what you meant by “or
> access references…”, to see if it covered this situation:
> >>>
> >>> StreamX —forward--> operator1
> >>> StreamX —forward--> operator2
> >>>
> >>> If operator1 modifies the record, and object re-use is enabled, then
> operator2 will see the modified version, right?
> >>>
> >>> Thanks,
> >>>
> >>> — Ken
> >>>
> >>> > On Jul 2, 2023, at 7:24 PM, Xuannan Su 
> wrote:
> >>> >
> >>> > Hi all,
> >>> >
> >>> > Dong(cc'ed) and I are opening this thread to discuss our proposal to
> >>> > add operator attribute to allow operator to specify support for
> >>> > object-reuse [1].
> >>> >
> >>> > Currently, the default configuration for pipeline.object-reuse is set
> >>> > to false to avoid data corruption, which can result in suboptimal
> >>> > performance. We propose adding APIs that operators can utilize to
> >>> > inform the Flink runtime whether it is safe to reuse the emitted
> >>> > records. This enhancement would enable Flink to maximize its
> >>> > performance using the default configuration.
> >>> >
> >>> > Please refer to the FLIP document for more details about the proposed
> >>> > design and implementation. We welcome any feedback and opinions on
> >>> > this proposal.
> >>> >
> >>> > Best regards,
> >>> >
> >>> > Dong and Xuannan
> >>> >
> >>> > [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255073749
> >>>
> >>> --
> >>> Ken Krugler
> >>> http://www.scaleunlimited.com
> >>> Custom big data solutions
> >>> Flink & Pinot
> >>>
> >>>
> >>>
>

Re: [VOTE] FLIP-387: Support named parameters for functions and call procedures

[Alexey Leonov-Vendrovskiy]

Thanks for starting the vote!
Do you mind adding a link from the FLIP to this thread?

Thanks,
Alexey

On Thu, Jan 4, 2024 at 6:48 PM Feng Jin  wrote:

> Hi everyone
>
> Thanks for all the feedback about the FLIP-387: Support named parameters
> for functions and call procedures [1] [2] .
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours(excluding weekends,until Jan 10, 12:00AM GMT) unless there is an
> objection or an insufficient number of votes.
>
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-387%3A+Support+named+parameters+for+functions+and+call+procedures
> [2] https://lists.apache.org/thread/bto7mpjvcx7d7k86owb00dwrm65jx8cn
>
>
> Best,
> Feng Jin
>

[jira] [Created] (FLINK-34004) TestingCheckpointIDCounter can easily lead to NPEs

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-34004:


 Summary: TestingCheckpointIDCounter can easily lead to NPEs
 Key: FLINK-34004
 URL: https://issues.apache.org/jira/browse/FLINK-34004
 Project: Flink
  Issue Type: Technical Debt
  Components: Tests
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.19.0


The TestingCheckpointIDCounter builder doesn't define safe defaults for all 
builder parameters. Using it can easily lead to surprising null pointer 
exceptions in tests when code is being modified to call more methods.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] Release flink-connector-hbase, release candidate #1

[Martijn Visser]

Hi everyone,
Please review and vote on the release candidate #1 for the version
3.0.1, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

This version is compatible with Flink 1.16.x, 1.17.x and 1.18.x

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org
[2], which are signed with the key with fingerprint
A5F3BCE4CBE993573EC5966A65321B8382B219AF [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag v3.0.1-rc1 [5],
* website pull request listing the new release [6].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Release Manager

[1] https://issues.apache.org/jira/projects/FLINK/versions/12353603
[2] https://dist.apache.org/repos/dist/dev/flink/flink-connector-hbase-3.0.1-rc1
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1692/
[5] https://github.com/apache/flink-connector-hbase/releases/tag/v3.0.1-rc1
[6] https://github.com/apache/flink-web/pull/708

[jira] [Created] (FLINK-34003) Bump CI flink version on flink-connector-hbase

[Martijn Visser (Jira)]

Martijn Visser created FLINK-34003:
--

 Summary: Bump CI flink version on flink-connector-hbase
 Key: FLINK-34003
 URL: https://issues.apache.org/jira/browse/FLINK-34003
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / HBase
Reporter: Martijn Visser
Assignee: Martijn Visser






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34002) Bump CI flink version on flink-connector-elasticsearch

[Martijn Visser (Jira)]

Martijn Visser created FLINK-34002:
--

 Summary: Bump CI flink version on flink-connector-elasticsearch
 Key: FLINK-34002
 URL: https://issues.apache.org/jira/browse/FLINK-34002
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / ElasticSearch
Reporter: Martijn Visser
Assignee: Martijn Visser






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-connector-jdbc, release candidate #1

[Martijn Visser]

Hi,

Hmmm, it would have been good to mark the Jira ticket as a Blocker
then for the JDBC connector. Since it's marked as Critical, it doesn't
appear. It has also been open for multiple months, so it doesn't
really feel like a Blocker. I'm +0 with including this fix, but then
we should either get that in quickly or revert FLINK-16024, especially
since this bug ticket has been open for multiple months. Right now, it
means that we don't have a working JDBC connector for Flink 1.17 and
Flink 1.18. That shouldn't be OK.

Thanks,

Martijn

On Fri, Jan 5, 2024 at 2:31 PM Sergey Nuyanzin  wrote:
>
> Thanks for driving this
>
> the thing which makes me thinking about -1 (not sure yet and that's why
> asking here) is that there is FLINK-33365 [1]
> mentioned as a blocker for JDBC connector release at [2]
> Since the reason for that is FLINK-16024 [3] as also was explained in
> comments for [1].
>
> So should we wait for a fix of [1] or revert [3] for 3.1.x and continue
> releasing 3.1.2?
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-33365
> [2] https://lists.apache.org/thread/sdkm5qshqozow9sljz6c0qjft6kg9cwc
>
> [3] https://issues.apache.org/jira/browse/FLINK-16024
>
> On Fri, Jan 5, 2024 at 2:19 PM Martijn Visser 
> wrote:
>
> > Hi everyone,
> > Please review and vote on the release candidate #1 for the version
> > 3.1.2, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > This version is compatible with Flink 1.16.x, 1.17.x and 1.18.x.
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> > [2], which are signed with the key with fingerprint
> > A5F3BCE4CBE993573EC5966A65321B8382B219AF [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag v3.1.2-rc1 [5],
> > * website pull request listing the new release [6].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1]
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354088
> > [2]
> > https://dist.apache.org/repos/dist/dev/flink/flink-connector-jdbc-3.1.2-rc1
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1691/
> > [5] https://github.com/apache/flink-connector-jdbc/releases/tag/v3.1.2-rc1
> > [6] https://github.com/apache/flink-web/pull/707
> >
>
>
> --
> Best regards,
> Sergey

[DISCUSS] FLIP-413: Enable unaligned checkpoints by default

[Piotr Nowojski]

Ops, fixing the topic.

Hi!
>
> I would like to propose by default to enable unaligned checkpoints and
> also simultaneously increase the aligned checkpoints timeout from 0ms to
> 5s. I think this change is the right one to do for the majority of Flink
> users.
>
> For more rationale please take a look into the short FLIP-413 [1].
>
> What do you all think?
>
> Best,
> Piotrek
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-413%3A+Enable+unaligned+checkpoints+by+default
>

FLIP-413: Enable unaligned checkpoints by default

[Piotr Nowojski]

Hi!

I would like to propose by default to enable unaligned checkpoints and also
simultaneously increase the aligned checkpoints timeout from 0ms to 5s. I
think this change is the right one to do for the majority of Flink users.

For more rationale please take a look into the short FLIP-413 [1].

What do you all think?

Best,
Piotrek

https://cwiki.apache.org/confluence/display/FLINK/FLIP-413%3A+Enable+unaligned+checkpoints+by+default

Re: [VOTE] Release flink-connector-jdbc, release candidate #1

[Sergey Nuyanzin]

Thanks for driving this

the thing which makes me thinking about -1 (not sure yet and that's why
asking here) is that there is FLINK-33365 [1]
mentioned as a blocker for JDBC connector release at [2]
Since the reason for that is FLINK-16024 [3] as also was explained in
comments for [1].

So should we wait for a fix of [1] or revert [3] for 3.1.x and continue
releasing 3.1.2?


[1] https://issues.apache.org/jira/browse/FLINK-33365
[2] https://lists.apache.org/thread/sdkm5qshqozow9sljz6c0qjft6kg9cwc

[3] https://issues.apache.org/jira/browse/FLINK-16024

On Fri, Jan 5, 2024 at 2:19 PM Martijn Visser 
wrote:

> Hi everyone,
> Please review and vote on the release candidate #1 for the version
> 3.1.2, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> This version is compatible with Flink 1.16.x, 1.17.x and 1.18.x.
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2], which are signed with the key with fingerprint
> A5F3BCE4CBE993573EC5966A65321B8382B219AF [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag v3.1.2-rc1 [5],
> * website pull request listing the new release [6].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354088
> [2]
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-jdbc-3.1.2-rc1
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1691/
> [5] https://github.com/apache/flink-connector-jdbc/releases/tag/v3.1.2-rc1
> [6] https://github.com/apache/flink-web/pull/707
>


-- 
Best regards,
Sergey

[VOTE] Release flink-connector-jdbc, release candidate #1

[Martijn Visser]

Hi everyone,
Please review and vote on the release candidate #1 for the version
3.1.2, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

This version is compatible with Flink 1.16.x, 1.17.x and 1.18.x.

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org
[2], which are signed with the key with fingerprint
A5F3BCE4CBE993573EC5966A65321B8382B219AF [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag v3.1.2-rc1 [5],
* website pull request listing the new release [6].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Release Manager

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354088
[2] https://dist.apache.org/repos/dist/dev/flink/flink-connector-jdbc-3.1.2-rc1
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1691/
[5] https://github.com/apache/flink-connector-jdbc/releases/tag/v3.1.2-rc1
[6] https://github.com/apache/flink-web/pull/707

Show data skew score on Flink Dashboard?

[Kartoglu, Emre]

Hello,

Is there a reason why a type of data skew score (probably just a percentage) is 
not shown on the Flink Dashboard / UI?

Currently users have to click on each operator and check how much data each 
subtask is processing to tell if there is skew. This is not efficient, 
especially cumbersome and also error prone for big job graphs.

It would be useful to have this shown on the operator. Possibly also a warning 
message at the top or somewhere “more meta” if a significant amount of skew is 
detected (so that users don’t have to zoom in on each and every operator to 
check the skew score).

I’d be happy to create a ticket for it, if there are no objections?

Kind regards,
Emre

Re: [RESULT][VOTE] Release flink-connector-hbase v3.0.0, release candidate #2

[Martijn Visser]

Hi Ferenc,

I honestly don't know what happened, but I'm currently pushing the
last changes in. Thanks for the reminder!

Best regards,

Martijn

On Thu, Nov 2, 2023 at 4:49 PM Ferenc Csaky  wrote:
>
> Hi Martijn!
>
> Is this work in progress?
>
> Thanks,
> Ferenc
>
>
>
>
> --- Original Message ---
> On Tuesday, September 12th, 2023 at 10:47, Martijn Visser 
>  wrote:
>
>
> >
> >
> > I'm happy to announce that we have unanimously approved this release.
> >
> > There are 7 approving votes, 3 of which are binding:
> > * Ahmed (non-binding)
> > * Sergey (non-binding)
> > * Samrat (non-binding)
> > * Ferenc (non-binding)
> > * Danny (binding)
> > * Leonard (binding)
> > * Dong (binding)
> >
> > There are no disapproving votes.
> >
> > I'll work on completing the release. Thanks all!
> >
> > Best regards,
> >
> > Martijn

Re: [NOTICE] Hive connector externalization

[Martijn Visser]

Sounds good to me! Thanks

On Fri, Jan 5, 2024 at 12:57 AM Sergey Nuyanzin  wrote:
>
> Hi Martijn
> thanks for reminding
> yep, I think you are right we need a release for that.
>
> IIRC so far there is no volunteers for that, so I would volunteer
>
> On Thu, Dec 28, 2023 at 1:27 PM Martijn Visser 
> wrote:
>
> > Hi Sergey,
> >
> > Is the next step that we need to generate a release of the
> > externalized code? Did someone already volunteer for that?
> >
> > Best regards,
> >
> > Martijn
> >
> > On Mon, Dec 11, 2023 at 3:00 AM yuxia  wrote:
> > >
> > > Thanks Sergey for the work. Happy to see we can externalize Hive
> > connector finally.
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > - 原始邮件 -
> > > 发件人: "snuyanzin" 
> > > 收件人: "dev" 
> > > 发送时间: 星期六, 2023年 12 月 09日 上午 6:24:35
> > > 主题: [NOTICE] Hive connector externalization
> > >
> > > Hi everyone
> > >
> > > We are getting close to the externalization of Hive connector[1].
> > > Since currently externalized version is already passing tests against
> > > release-1.18 and release-1.19 then I'm going to remove Hive connector
> > code
> > > from Flink main repo[2]. For that reason I would kindly ask to avoid
> > > merging of Hive connector related changes to Flink main repo (master
> > > branch) in order to make this smoother. Instead it would be better to
> > > create/merge  prs to connector's repo[3]
> > >
> > > Also huge shoutout to Yuxia Luo, Martijn Visser, Ryan Skraba for the
> > review
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-30064
> > > [2] https://issues.apache.org/jira/browse/FLINK-33786
> > > [3] https://github.com/apache/flink-connector-hive
> > >
> > > --
> > > Best regards,
> > > Sergey
> >
>
>
> --
> Best regards,
> Sergey

Re: [DISCUSS] FLIP-316: Introduce SQL Driver

[Márton Balassi]

Thanks, Paul.

Ferenc and I have been looking into unblocking the Kubernetes path via an
updated implementation for FLINK-28915 to ship the jars conveniently there.
You can expect an updated PR there next week. Looking forward to your
findings in the YARN POC.

On Mon, Dec 11, 2023 at 4:01 AM Paul Lam  wrote:

> Hi Ferenc,
>
> Sorry for my late reply.
>
> > Is any active work happening on this FLIP? As far as I see there
> > are blockers that needs to happen first to implement regarding
> > artifact distribution.
>
> You’re right. There’s a block in K8s application mode, but none in
> YARN application. I’m doing a POC on YARN application mode
> before starting a vote thread.
>
> I’ve been busy lately, but the FLIP is active for sure. The situation
> will change in a couple of weeks.
>
> Thank you for reaching out! I’ll let you know when the POC is completed.
>
> Best,
> Paul Lam
>
> > 2023年11月21日 06:01，Ferenc Csaky  写道：
> >
> > Hello devs,
> >
> > Is any active work happening on this FLIP? As far as I see there
> > are blockers that needs to happen first to implement regarding
> > artifact distribution.
> >
> > Is this work in halt completetly or some efforts are going into
> > resolve the blockers first or something?
> >
> > Our platform would benefit this feature a lot, we have a kind of
> > working custom implementation at the moment, but it is uniquely
> > adapted to our app and platform.
> >
> > I could help out to move this forward.
> >
> > Best,
> > Ferenc
> >
> >
> >
> > On Friday, June 30th, 2023 at 04:53, Paul Lam  > wrote:
> >
> >
> >>
> >>
> >> Hi Jing,
> >>
> >> Thanks for your input!
> >>
> >>> Would you like to add
> >>> one section to describe(better with script/code example) how to use it
> in
> >>> these two scenarios from users' perspective?
> >>
> >>
> >> OK. I’ll update the FLIP with the code snippet after I get the POC
> branch done.
> >>
> >>> NIT: the pictures have transparent background when readers click on
> it. It
> >>> would be great if you can replace them with pictures with white
> background.
> >>
> >>
> >> Fixed. Thanks for pointing that out :)
> >>
> >> Best,
> >> Paul Lam
> >>
> >>> 2023年6月27日 06:51，Jing Ge j...@ververica.com.INVALID 写道：
> >>>
> >>> Hi Paul,
> >>>
> >>> Thanks for driving it and thank you all for the informative
> discussion! The
> >>> FLIP is in good shape now. As described in the FLIP, SQL Driver will be
> >>> mainly used to run Flink SQLs in two scenarios: 1. SQL client/gateway
> in
> >>> application mode and 2. external system integration. Would you like to
> add
> >>> one section to describe(better with script/code example) how to use it
> in
> >>> these two scenarios from users' perspective?
> >>>
> >>> NIT: the pictures have transparent background when readers click on
> it. It
> >>> would be great if you can replace them with pictures with white
> background.
> >>>
> >>> Best regards,
> >>> Jing
> >>>
> >>> On Mon, Jun 26, 2023 at 1:31 PM Paul Lam   mailto:paullin3...@gmail.com> wrote:
> >>>
>  Hi Shengkai,
> 
> > * How can we ship the json plan to the JobManager?
> 
>  The Flink K8s module should be responsible for file distribution. We
> could
>  introduce
>  an option like `kubernetes.storage.dir`. For each flink cluster, there
>  would be a
>  dedicated subdirectory, with the pattern like
>  `${kubernetes.storage.dir}/${cluster-id}`.
> 
>  All resources-related options (e.g. pipeline jars, json plans) that
> are
>  configured with
>  scheme `file://`   >
> would be uploaded to the resource directory
>  and downloaded to the
>  jobmanager, before SQL Driver accesses the files with the original
>  filenames.
> 
> > * Classloading strategy
> 
>  We could directly specify the SQL Gateway jar as the jar file in
>  PackagedProgram.
>  It would be treated like a normal user jar and the SQL Driver is
> loaded
>  into the user
>  classloader. WDYT?
> 
> > * Option `$internal.sql-gateway.driver.sql-config` is string type
> > I think it's better to use Map type here
> 
>  By Map type configuration, do you mean a nested map that contains all
>  configurations?
> 
>  I hope I've explained myself well, it’s a file that contains the
> extra SQL
>  configurations, which would be shipped to the jobmanager.
> 
> > * PoC branch
> 
>  Sure. I’ll let you know once I get the job done.
> 
>  Best,
>  Paul Lam
> 
> > 2023年6月26日 14:27，Shengkai Fang  fskm...@gmail.com> mailto:fskm...@gmail.com> 写道：
> >
> > Hi, Paul.
> >
> > Thanks for your update. I have a few questions about the new design:
> >
> > * How can we ship the json plan to the JobManager?
> >
> > The current design only exposes an option about the URL of the json
> > plan. It seems the gateway is responsible to upload to an external
> stroage.

[jira] [Created] (FLINK-34001) doc diff from code

[yong yang (Jira)]

yong yang created FLINK-34001:
-

 Summary: doc diff from code
 Key: FLINK-34001
 URL: https://issues.apache.org/jira/browse/FLINK-34001
 Project: Flink
  Issue Type: Bug
  Components: API / State Processor
Affects Versions: 1.18.0
Reporter: yong yang


doc:

https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/concepts/overview/#idle-state-retention-time

The current TTL value for both left and right side is {{{}"0 ms"{}}}, which 
means the state retention is not enabled. 

 

but i test find :

The current TTL value for both left and right side is {{{}"0 ms"{}}}, which 
means the state is permanence keep!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

[Xuannan Su]

Hi Lu,

I believe this feature is very useful. However, I currently lack the
capacity to work on it in the near future. I think it would be great
if you could take on the task. I am willing to offer assistance if
there are any questions about the FLIP, or to review the PR if needed.

Please let me know if you are interested in taking over this task. And
also think that we should start the voting thread if no future
comments on this FLIP.

Best,
Xuannan



On Fri, Jan 5, 2024 at 2:23 PM Dong Lin  wrote:
>
> Hi Lu,
>
> I am not actively working on Flink and this JIRA recently. If Xuannan does 
> not plan to work on this anytime soon, I personally think it will be great if 
> you can help work on this FLIP. Maybe we can start the voting thread if there 
> is no further comment on this FLIP.
>
> Xuannan, what do you think?
>
> Thanks,
> Dong
>
>
> On Fri, Jan 5, 2024 at 2:03 AM Lu Niu  wrote:
>>
>> Hi,
>>
>> Is this still under active development? I notice 
>> https://issues.apache.org/jira/browse/FLINK-32476 is labeled as 
>> deprioritized. If this is the case, would it be acceptable for us to take on 
>> the task?
>>
>> Best
>> Lu
>>
>>
>>
>> On Thu, Oct 19, 2023 at 4:26 PM Ken Krugler  
>> wrote:
>>>
>>> Hi Dong,
>>>
>>> Sorry for not seeing this initially. I did have one question about the 
>>> description of the issue in the FLIP:
>>>
>>> > However, in cases where the upstream and downstream operators do not 
>>> > store or access references to the input or output records, this deep-copy 
>>> > overhead becomes unnecessary
>>>
>>> I was interested in getting clarification as to what you meant by “or 
>>> access references…”, to see if it covered this situation:
>>>
>>> StreamX —forward--> operator1
>>> StreamX —forward--> operator2
>>>
>>> If operator1 modifies the record, and object re-use is enabled, then 
>>> operator2 will see the modified version, right?
>>>
>>> Thanks,
>>>
>>> — Ken
>>>
>>> > On Jul 2, 2023, at 7:24 PM, Xuannan Su  wrote:
>>> >
>>> > Hi all,
>>> >
>>> > Dong(cc'ed) and I are opening this thread to discuss our proposal to
>>> > add operator attribute to allow operator to specify support for
>>> > object-reuse [1].
>>> >
>>> > Currently, the default configuration for pipeline.object-reuse is set
>>> > to false to avoid data corruption, which can result in suboptimal
>>> > performance. We propose adding APIs that operators can utilize to
>>> > inform the Flink runtime whether it is safe to reuse the emitted
>>> > records. This enhancement would enable Flink to maximize its
>>> > performance using the default configuration.
>>> >
>>> > Please refer to the FLIP document for more details about the proposed
>>> > design and implementation. We welcome any feedback and opinions on
>>> > this proposal.
>>> >
>>> > Best regards,
>>> >
>>> > Dong and Xuannan
>>> >
>>> > [1] 
>>> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255073749
>>>
>>> --
>>> Ken Krugler
>>> http://www.scaleunlimited.com
>>> Custom big data solutions
>>> Flink & Pinot
>>>
>>>
>>>

Re: [VOTE] FLIP-387: Support named parameters for functions and call procedures

[Timo Walther]

Thanks for the last minute change.

+1 (binding)

Cheers,
Timo


On 05.01.24 08:59, Feng Jin wrote:

Hi Timo,

Thank you for the suggestion. Previously, I thought most parameters were
optional, so the default value was set to true.

Your concern is reasonable. We should declare it as false by default and
developers should explicitly state if a parameter is optional instead of
using our default value.

Regarding this part, I have already made modifications in the document.


Best,
Feng


On Fri, Jan 5, 2024 at 3:38 PM Timo Walther  wrote:


Thanks, for starting the VOTE thread and thanks for considering my
feedback. One last comment before I'm also happy to give my +1 to this:

Why is ArgumentHint's default isOptinal=true? Shouldn't it be false by
default? Many function implementers will forget to set this to false and
suddenly get NULLs passed to their functions. Marking an argument as
optional should be an explicit decision of an implementer.

Regards,
Timo


On 05.01.24 05:06, Lincoln Lee wrote:

+1 (binding)

Best,
Lincoln Lee


Benchao Li  于2024年1月5日周五 11:46写道：


+1 (binding)

Feng Jin  于2024年1月5日周五 10:49写道：


Hi everyone

Thanks for all the feedback about the FLIP-387: Support named

parameters

for functions and call procedures [1] [2] .

I'd like to start a vote for it. The vote will be open for at least 72
hours(excluding weekends,until Jan 10, 12:00AM GMT) unless there is an
objection or an insufficient number of votes.



[1]




https://cwiki.apache.org/confluence/display/FLINK/FLIP-387%3A+Support+named+parameters+for+functions+and+call+procedures

[2] https://lists.apache.org/thread/bto7mpjvcx7d7k86owb00dwrm65jx8cn


Best,
Feng Jin




--

Best,
Benchao Li

Re: [VOTE] FLIP-387: Support named parameters for functions and call procedures

[Feng Jin]

Hi Timo,

Thank you for the suggestion. Previously, I thought most parameters were
optional, so the default value was set to true.

Your concern is reasonable. We should declare it as false by default and
developers should explicitly state if a parameter is optional instead of
using our default value.

Regarding this part, I have already made modifications in the document.


Best,
Feng


On Fri, Jan 5, 2024 at 3:38 PM Timo Walther  wrote:

> Thanks, for starting the VOTE thread and thanks for considering my
> feedback. One last comment before I'm also happy to give my +1 to this:
>
> Why is ArgumentHint's default isOptinal=true? Shouldn't it be false by
> default? Many function implementers will forget to set this to false and
> suddenly get NULLs passed to their functions. Marking an argument as
> optional should be an explicit decision of an implementer.
>
> Regards,
> Timo
>
>
> On 05.01.24 05:06, Lincoln Lee wrote:
> > +1 (binding)
> >
> > Best,
> > Lincoln Lee
> >
> >
> > Benchao Li  于2024年1月5日周五 11:46写道：
> >
> >> +1 (binding)
> >>
> >> Feng Jin  于2024年1月5日周五 10:49写道：
> >>>
> >>> Hi everyone
> >>>
> >>> Thanks for all the feedback about the FLIP-387: Support named
> parameters
> >>> for functions and call procedures [1] [2] .
> >>>
> >>> I'd like to start a vote for it. The vote will be open for at least 72
> >>> hours(excluding weekends,until Jan 10, 12:00AM GMT) unless there is an
> >>> objection or an insufficient number of votes.
> >>>
> >>>
> >>>
> >>> [1]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-387%3A+Support+named+parameters+for+functions+and+call+procedures
> >>> [2] https://lists.apache.org/thread/bto7mpjvcx7d7k86owb00dwrm65jx8cn
> >>>
> >>>
> >>> Best,
> >>> Feng Jin
> >>
> >>
> >>
> >> --
> >>
> >> Best,
> >> Benchao Li
> >>
> >
>
>