[jira] [Created] (FLINK-31903) Caching records fails in BroadcastUtils#withBroadcastStream

[Zhipeng Zhang (Jira)]

Zhipeng Zhang created FLINK-31903:
-

 Summary: Caching records fails in 
BroadcastUtils#withBroadcastStream
 Key: FLINK-31903
 URL: https://issues.apache.org/jira/browse/FLINK-31903
 Project: Flink
  Issue Type: Bug
  Components: Library / Machine Learning
Affects Versions: ml-2.3.0
Reporter: Zhipeng Zhang


When caching more than 1,000,000 records using BroadcastUtils#withBroadcast, it 
leads to exception as follows:
{code:java}
Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint 
tolerable failure threshold.
    at 
org.apache.flink.runtime.checkpoint.CheckpointFailureManager.checkFailureAgainstCounter(CheckpointFailureManager.java:206)
    at 
org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:191)
    at 
org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleCheckpointException(CheckpointFailureManager.java:124)
    at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2078)
    at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:1038)
    at 
org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$declineCheckpoint$2(ExecutionGraphHandler.java:103)
    at 
org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119)
    at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 {code}
It seems that the bug comes from caching too many records when calling 

AbstractBroadcastWrapperOperator#snapshot. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31904) fix current serveral flink nullable type handle

[jackylau (Jira)]

jackylau created FLINK-31904:


 Summary: fix current serveral flink nullable type handle
 Key: FLINK-31904
 URL: https://issues.apache.org/jira/browse/FLINK-31904
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Affects Versions: 1.18.0
Reporter: jackylau






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31905) Exception thrown when accessing nested field of the result of Python UDF with complex type

[Dian Fu (Jira)]

Dian Fu created FLINK-31905:
---

 Summary: Exception thrown when accessing nested field of the 
result of Python UDF with complex type
 Key: FLINK-31905
 URL: https://issues.apache.org/jira/browse/FLINK-31905
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Reporter: Dian Fu


For the following job:
{code}
import logging, sys

from pyflink.common import Row
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.table import Schema, DataTypes, TableDescriptor, 
StreamTableEnvironment
from pyflink.table.expressions import col, row
from pyflink.table.udf import ACC, T, udaf, AggregateFunction, udf

logging.basicConfig(stream=sys.stdout, level=logging.ERROR, 
format="%(message)s")


class EmitLastState(AggregateFunction):
"""
Aggregator that emits the latest state for the purpose of
enabling parallelism on CDC tables.
"""

def create_accumulator(self) -> ACC:
return Row(None, None)

def accumulate(self, accumulator: ACC, *args):
key, obj = args
if (accumulator[0] is None) or (key > accumulator[0]):
accumulator[0] = key
accumulator[1] = obj

def retract(self, accumulator: ACC, *args):
pass

def get_value(self, accumulator: ACC) -> T:
return accumulator[1]


some_complex_inner_type = DataTypes.ROW(
[
DataTypes.FIELD("f0", DataTypes.STRING()),
DataTypes.FIELD("f1", DataTypes.STRING())
]
)

some_complex_type = DataTypes.ROW(
[
DataTypes.FIELD(k, DataTypes.ARRAY(some_complex_inner_type))
for k in ("f0", "f1", "f2")
]
+ [
DataTypes.FIELD("f3", DataTypes.DATE()),
DataTypes.FIELD("f4", DataTypes.VARCHAR(32)),
DataTypes.FIELD("f5", DataTypes.VARCHAR(2)),
]
)

@udf(input_types=DataTypes.STRING(), result_type=some_complex_type)
def complex_udf(s):
return Row(f0=None, f1=None, f2=None, f3=None, f4=None, f5=None)


if __name__ == "__main__":
env = StreamExecutionEnvironment.get_execution_environment()
table_env = StreamTableEnvironment.create(env)
table_env.get_config().set('pipeline.classpaths', 
'file:///Users/dianfu/code/src/workspace/pyflink-examples/flink-sql-connector-postgres-cdc-2.1.1.jar')

# Create schema
_schema = {
"p_key": DataTypes.INT(False),
"modified_id": DataTypes.INT(False),
"content": DataTypes.STRING()
}
schema = Schema.new_builder().from_fields(
*zip(*[(k, v) for k, v in _schema.items()])
).\
primary_key("p_key").\
build()

# Create table descriptor
descriptor = TableDescriptor.for_connector("postgres-cdc").\
option("hostname", "host.docker.internal").\
option("port", "5432").\
option("database-name", "flink_issue").\
option("username", "root").\
option("password", "root").\
option("debezium.plugin.name", "pgoutput").\
option("schema-name", "flink_schema").\
option("table-name", "flink_table").\
option("slot.name", "flink_slot").\
schema(schema).\
build()

table_env.create_temporary_table("flink_table", descriptor)

# Create changelog stream
stream = table_env.from_path("flink_table")\

# Define UDAF
accumulator_type = DataTypes.ROW(
[
DataTypes.FIELD("f0", DataTypes.INT(False)),
DataTypes.FIELD("f1", DataTypes.ROW([DataTypes.FIELD(k, v) for k, v 
in _schema.items()])),
]
)
result_type = DataTypes.ROW([DataTypes.FIELD(k, v) for k, v in 
_schema.items()])
emit_last = udaf(EmitLastState(), accumulator_type=accumulator_type, 
result_type=result_type)

# Emit last state based on modified_id to enable parallel processing
stream = stream.\
group_by(col("p_key")).\
select(
col("p_key"),
emit_last(col("modified_id"),row(*(col(k) for k in 
_schema.keys())).cast(result_type)).alias("tmp_obj")
)

# Select the elements of the objects
stream = stream.select(*(col("tmp_obj").get(k).alias(k) for k in 
_schema.keys()))

# We apply a UDF which parses the xml and returns a complex nested structure
stream = stream.select(col("p_key"), 
complex_udf(col("content")).alias("nested_obj"))

# We select an element from the nested structure in order to flatten it
# The next line is the line causing issues, commenting the next line will 
make the pipeline work
stream = stream.select(col("p_key"), col("nested_obj").get("f0"))

# Interestingly, the below part does work...
# stream = stream.select(col("nested_obj").get("f0"))

table_env.to_changelog_stream(stream).print()

# Execute
env.execute_async()
{code}

{code}
py4j.protocol.Py4JJavaError: An error occurred while calling 
o8.toChangelogStream.
: java.lang.IndexOutOfBoundsException: Index 1 out of bounds for length 1
at 

[jira] [Created] (FLINK-31906) typeof should only return type exclude nullable

[jackylau (Jira)]

jackylau created FLINK-31906:


 Summary: typeof should only return type exclude nullable 
 Key: FLINK-31906
 URL: https://issues.apache.org/jira/browse/FLINK-31906
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.18.0
Reporter: jackylau






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Member - Leonard Xu

[Jing Ge]

Congrats! Leonard!



Best regards,

Jing

On Mon, Apr 24, 2023 at 5:53 AM Matthias Pohl
 wrote:

> Congrats, Leonard :)
>
> On Mon, Apr 24, 2023, 05:17 Yangze Guo  wrote:
>
> > Congratulations, Leonard!
> >
> > Best,
> > Yangze Guo
> >
> > On Mon, Apr 24, 2023 at 10:05 AM Shuo Cheng  wrote:
> > >
> > > Congratulations, Leonard.
> > >
> > > Best,
> > > Shuo
> > >
> > > On Sun, Apr 23, 2023 at 7:43 PM Sergey Nuyanzin 
> > wrote:
> > >
> > > > Congratulations, Leonard!
> > > >
> > > > On Sun, Apr 23, 2023 at 1:38 PM Zhipeng Zhang <
> zhangzhipe...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Congratulations, Leonard.
> > > > >
> > > > > Hang Ruan  于2023年4月23日周日 19:03写道：
> > > > > >
> > > > > > Congratulations, Leonard.
> > > > > >
> > > > > > Best,
> > > > > > Hang
> > > > > >
> > > > > > Yanfei Lei  于2023年4月23日周日 18:34写道：
> > > > > >
> > > > > > > Congratulations, Leonard!
> > > > > > >
> > > > > > > Best,
> > > > > > > Yanfei
> > > > > > >
> > > > > > > liu ron  于2023年4月23日周日 17:45写道：
> > > > > > > >
> > > > > > > > Congratulations, Leonard.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Ron
> > > > > > > >
> > > > > > > > Zhanghao Chen  于2023年4月23日周日
> > 17:33写道：
> > > > > > > >
> > > > > > > > > Congratulations, Leonard!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Zhanghao Chen
> > > > > > > > > 
> > > > > > > > > From: Shammon FY 
> > > > > > > > > Sent: Sunday, April 23, 2023 17:22
> > > > > > > > > To: dev@flink.apache.org 
> > > > > > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member -
> > Leonard Xu
> > > > > > > > >
> > > > > > > > > Congratulations, Leonard!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Shammon FY
> > > > > > > > >
> > > > > > > > > On Sun, Apr 23, 2023 at 5:07 PM Xianxun Ye <
> > > > > yesorno828...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congratulations, Leonard!
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > >
> > > > > > > > > > Xianxun
> > > > > > > > > >
> > > > > > > > > > > 2023年4月23日 09:10，Lincoln Lee 
> > 写道：
> > > > > > > > > > >
> > > > > > > > > > > Congratulations, Leonard!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > best,
> > > > > Zhipeng
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Sergey
> > > >
> >
>

[jira] [Created] (FLINK-31907) Remove unused fields inside of ExecutionSlotSharingGroupBuilder

[Rui Fan (Jira)]

Rui Fan created FLINK-31907:
---

 Summary: Remove unused fields inside of 
ExecutionSlotSharingGroupBuilder
 Key: FLINK-31907
 URL: https://issues.apache.org/jira/browse/FLINK-31907
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.18.0
Reporter: Rui Fan
Assignee: Rui Fan


FLINK-22767 introduced `availableGroupsForJobVertex` to improve the performance 
during task to slot scheduler.

After FLINK-22767, the `executionSlotSharingGroups`[2] is unused, and it can be 
removed.

 

[1] 
https://github.com/apache/flink/blob/f9b3e0b7bc0432001b4a197539a0712b16e0b33b/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/LocalInputPreferredSlotSharingStrategy.java#L153

[2] 
https://github.com/apache/flink/blob/f9b3e0b7bc0432001b4a197539a0712b16e0b33b/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/LocalInputPreferredSlotSharingStrategy.java#L136



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Member - Qingsheng Ren

[Zakelly Lan]

Congratulations, Qingsheng!

Best regards,
Zakelly

On Mon, Apr 24, 2023 at 11:52 AM Matthias Pohl
 wrote:
>
> Congratulations, Qingsheng! :)
>
> On Mon, Apr 24, 2023, 05:17 Yangze Guo  wrote:
>
> > Congratulations, Qingsheng!
> >
> > Best,
> > Yangze Guo
> >
> > On Mon, Apr 24, 2023 at 10:05 AM Shuo Cheng  wrote:
> > >
> > > Congratulations, Qingsheng!
> > >
> > > Best,
> > > Shuo
> > >
> > > On Sun, Apr 23, 2023 at 7:43 PM Sergey Nuyanzin 
> > wrote:
> > >
> > > > Congratulations, Qingsheng!
> > > >
> > > > On Sun, Apr 23, 2023 at 1:37 PM Zhipeng Zhang  > >
> > > > wrote:
> > > >
> > > > > Congratulations, Qingsheng!
> > > > >
> > > > > Hang Ruan  于2023年4月23日周日 19:03写道：
> > > > > >
> > > > > > Congratulations, Qingsheng!
> > > > > >
> > > > > > Best,
> > > > > > Hang
> > > > > >
> > > > > > Yanfei Lei  于2023年4月23日周日 18:33写道：
> > > > > >
> > > > > > > Congratulations, Qingsheng!
> > > > > > >
> > > > > > > Best,
> > > > > > > Yanfei
> > > > > > >
> > > > > > > liu ron  于2023年4月23日周日 17:47写道：
> > > > > > > >
> > > > > > > > Congratulations, Qingsheng.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Ron
> > > > > > > >
> > > > > > > > Zhanghao Chen  于2023年4月23日周日
> > 17:32写道：
> > > > > > > >
> > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Zhanghao Chen
> > > > > > > > > 
> > > > > > > > > From: Shammon FY 
> > > > > > > > > Sent: Sunday, April 23, 2023 17:22
> > > > > > > > > To: dev@flink.apache.org 
> > > > > > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member -
> > Qingsheng
> > > > Ren
> > > > > > > > >
> > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Shammon FY
> > > > > > > > >
> > > > > > > > > On Sun, Apr 23, 2023 at 4:40 PM Weihua Hu <
> > > > huweihua@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Weihua
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sun, Apr 23, 2023 at 3:53 PM Yun Tang  > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > > >
> > > > > > > > > > > Best
> > > > > > > > > > > Yun Tang
> > > > > > > > > > > 
> > > > > > > > > > > From: weijie guo 
> > > > > > > > > > > Sent: Sunday, April 23, 2023 14:50
> > > > > > > > > > > To: dev@flink.apache.org 
> > > > > > > > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member -
> > > > > Qingsheng Ren
> > > > > > > > > > >
> > > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > >
> > > > > > > > > > > Weijie
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Geng Biao  于2023年4月23日周日 14:29写道：
> > > > > > > > > > >
> > > > > > > > > > > > Congrats, Qingsheng!
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Biao Geng
> > > > > > > > > > > >
> > > > > > > > > > > > 获取 Outlook for iOS
> > > > > > > > > > > > 
> > > > > > > > > > > > 发件人: Wencong Liu 
> > > > > > > > > > > > 发送时间: Sunday, April 23, 2023 11:06:39 AM
> > > > > > > > > > > > 收件人: dev@flink.apache.org 
> > > > > > > > > > > > 主题: Re:[ANNOUNCE] New Apache Flink PMC Member -
> > Qingsheng
> > > > Ren
> > > > > > > > > > > >
> > > > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Wencong LIu
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > At 2023-04-21 19:47:52, "Jark Wu" 
> > > > wrote:
> > > > > > > > > > > > >Hi everyone,
> > > > > > > > > > > > >
> > > > > > > > > > > > >We are thrilled to announce that Leonard Xu has
> > joined the
> > > > > Flink
> > > > > > > > > PMC!
> > > > > > > > > > > > >
> > > > > > > > > > > > >Leonard has been an active member of the Apache Flink
> > > > > community
> > > > > > > for
> > > > > > > > > > many
> > > > > > > > > > > > >years and became a committer in Nov 2021. He has been
> > > > > involved
> > > > > > > in
> > > > > > > > > > > various
> > > > > > > > > > > > >areas of the project, from code contributions to
> > community
> > > > > > > building.
> > > > > > > > > > His
> > > > > > > > > > > > >contributions are mainly focused on Flink SQL and
> > > > > connectors,
> > > > > > > > > > especially
> > > > > > > > > > > > >leading the flink-cdc-connectors project to receive
> > 3.8+K
> > > > > GitHub
> > > > > > > > > > stars.
> > > > > > > > > > > He
> > > > > > > > > > > > >authored 150+ PRs, and reviewed 250+ 

Re: [ANNOUNCE] New Apache Flink PMC Member - Leonard Xu

[Zakelly Lan]

Congratulations, Leonard!


Best regards,
Zakelly

On Mon, Apr 24, 2023 at 3:25 PM Jing Ge  wrote:
>
> Congrats! Leonard!
>
>
>
> Best regards,
>
> Jing
>
> On Mon, Apr 24, 2023 at 5:53 AM Matthias Pohl
>  wrote:
>
> > Congrats, Leonard :)
> >
> > On Mon, Apr 24, 2023, 05:17 Yangze Guo  wrote:
> >
> > > Congratulations, Leonard!
> > >
> > > Best,
> > > Yangze Guo
> > >
> > > On Mon, Apr 24, 2023 at 10:05 AM Shuo Cheng  wrote:
> > > >
> > > > Congratulations, Leonard.
> > > >
> > > > Best,
> > > > Shuo
> > > >
> > > > On Sun, Apr 23, 2023 at 7:43 PM Sergey Nuyanzin 
> > > wrote:
> > > >
> > > > > Congratulations, Leonard!
> > > > >
> > > > > On Sun, Apr 23, 2023 at 1:38 PM Zhipeng Zhang <
> > zhangzhipe...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Congratulations, Leonard.
> > > > > >
> > > > > > Hang Ruan  于2023年4月23日周日 19:03写道：
> > > > > > >
> > > > > > > Congratulations, Leonard.
> > > > > > >
> > > > > > > Best,
> > > > > > > Hang
> > > > > > >
> > > > > > > Yanfei Lei  于2023年4月23日周日 18:34写道：
> > > > > > >
> > > > > > > > Congratulations, Leonard!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yanfei
> > > > > > > >
> > > > > > > > liu ron  于2023年4月23日周日 17:45写道：
> > > > > > > > >
> > > > > > > > > Congratulations, Leonard.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Ron
> > > > > > > > >
> > > > > > > > > Zhanghao Chen  于2023年4月23日周日
> > > 17:33写道：
> > > > > > > > >
> > > > > > > > > > Congratulations, Leonard!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Zhanghao Chen
> > > > > > > > > > 
> > > > > > > > > > From: Shammon FY 
> > > > > > > > > > Sent: Sunday, April 23, 2023 17:22
> > > > > > > > > > To: dev@flink.apache.org 
> > > > > > > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member -
> > > Leonard Xu
> > > > > > > > > >
> > > > > > > > > > Congratulations, Leonard!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Shammon FY
> > > > > > > > > >
> > > > > > > > > > On Sun, Apr 23, 2023 at 5:07 PM Xianxun Ye <
> > > > > > yesorno828...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Congratulations, Leonard!
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > >
> > > > > > > > > > > Xianxun
> > > > > > > > > > >
> > > > > > > > > > > > 2023年4月23日 09:10，Lincoln Lee 
> > > 写道：
> > > > > > > > > > > >
> > > > > > > > > > > > Congratulations, Leonard!
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > best,
> > > > > > Zhipeng
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Sergey
> > > > >
> > >
> >

Re: [ANNOUNCE] New Apache Flink PMC Member - Qingsheng Ren

[Jing Ge]

Congrats! Qingsheng!



Best regards,

Jing

On Mon, Apr 24, 2023 at 9:35 AM Zakelly Lan  wrote:

> Congratulations, Qingsheng!
>
> Best regards,
> Zakelly
>
> On Mon, Apr 24, 2023 at 11:52 AM Matthias Pohl
>  wrote:
> >
> > Congratulations, Qingsheng! :)
> >
> > On Mon, Apr 24, 2023, 05:17 Yangze Guo  wrote:
> >
> > > Congratulations, Qingsheng!
> > >
> > > Best,
> > > Yangze Guo
> > >
> > > On Mon, Apr 24, 2023 at 10:05 AM Shuo Cheng 
> wrote:
> > > >
> > > > Congratulations, Qingsheng!
> > > >
> > > > Best,
> > > > Shuo
> > > >
> > > > On Sun, Apr 23, 2023 at 7:43 PM Sergey Nuyanzin  >
> > > wrote:
> > > >
> > > > > Congratulations, Qingsheng!
> > > > >
> > > > > On Sun, Apr 23, 2023 at 1:37 PM Zhipeng Zhang <
> zhangzhipe...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Congratulations, Qingsheng!
> > > > > >
> > > > > > Hang Ruan  于2023年4月23日周日 19:03写道：
> > > > > > >
> > > > > > > Congratulations, Qingsheng!
> > > > > > >
> > > > > > > Best,
> > > > > > > Hang
> > > > > > >
> > > > > > > Yanfei Lei  于2023年4月23日周日 18:33写道：
> > > > > > >
> > > > > > > > Congratulations, Qingsheng!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yanfei
> > > > > > > >
> > > > > > > > liu ron  于2023年4月23日周日 17:47写道：
> > > > > > > > >
> > > > > > > > > Congratulations, Qingsheng.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Ron
> > > > > > > > >
> > > > > > > > > Zhanghao Chen  于2023年4月23日周日
> > > 17:32写道：
> > > > > > > > >
> > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Zhanghao Chen
> > > > > > > > > > 
> > > > > > > > > > From: Shammon FY 
> > > > > > > > > > Sent: Sunday, April 23, 2023 17:22
> > > > > > > > > > To: dev@flink.apache.org 
> > > > > > > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member -
> > > Qingsheng
> > > > > Ren
> > > > > > > > > >
> > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Shammon FY
> > > > > > > > > >
> > > > > > > > > > On Sun, Apr 23, 2023 at 4:40 PM Weihua Hu <
> > > > > huweihua@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Weihua
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Apr 23, 2023 at 3:53 PM Yun Tang <
> myas...@live.com
> > > >
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > > > >
> > > > > > > > > > > > Best
> > > > > > > > > > > > Yun Tang
> > > > > > > > > > > > 
> > > > > > > > > > > > From: weijie guo 
> > > > > > > > > > > > Sent: Sunday, April 23, 2023 14:50
> > > > > > > > > > > > To: dev@flink.apache.org 
> > > > > > > > > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member -
> > > > > > Qingsheng Ren
> > > > > > > > > > > >
> > > > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > >
> > > > > > > > > > > > Weijie
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Geng Biao  于2023年4月23日周日
> 14:29写道：
> > > > > > > > > > > >
> > > > > > > > > > > > > Congrats, Qingsheng!
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Biao Geng
> > > > > > > > > > > > >
> > > > > > > > > > > > > 获取 Outlook for iOS
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 发件人: Wencong Liu 
> > > > > > > > > > > > > 发送时间: Sunday, April 23, 2023 11:06:39 AM
> > > > > > > > > > > > > 收件人: dev@flink.apache.org 
> > > > > > > > > > > > > 主题: Re:[ANNOUNCE] New Apache Flink PMC Member -
> > > Qingsheng
> > > > > Ren
> > > > > > > > > > > > >
> > > > > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Wencong LIu
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > At 2023-04-21 19:47:52, "Jark Wu" <
> imj...@gmail.com>
> > > > > wrote:
> > > > > > > > > > > > > >Hi everyone,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >We are thrilled to announce that Leonard Xu has
> > > joined the
> > > > > > Flink
> > > > > > > > > > PMC!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >Leonard has been an active member of the Apache
> Flink
> > > > > > community
> > > > > > > > for
> > > > > > > > > > > many
> > > > > > > > > > > > > >years and became a committer in Nov 2021. He has
> been
> > > > > > involved
> > > > > > > > in
> > > > > > > > > 

Re: [DISCUSS FLINKSQL PARALLELISM]

[Jing Ge]

Hi Green,



Since FLIP-292 opened the door to do fine-grained tuning at operator level
for Flink SQL jobs, I would also suggest leveraging the compiled json to do
further config optimization like Yun Tang already mentioned. We should
consider making it(leveraging the compiled json plan) the stand process for
Flink SQL job fine-grained tuning.



Best regards,

Jing

On Wed, Apr 19, 2023 at 8:44 AM Yun Tang  wrote:

> I noticed that Yuxia had replied that "sink.paralleilsm" could help in
> some cases.
>
> I think a better way is to integrate it with streamGraph or extend
> CompiledPlan just as FLIP-292 setting state TTL per operator [1] does.
>
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240883951
>
> Best
> Yun Tang
> 
> From: GREEN <1286649...@qq.com.INVALID>
> Sent: Tuesday, April 18, 2023 17:21
> To: dev 
> Subject: Re: [DISCUSS FLINKSQL PARALLELISM]
>
> During the process of generating streamgraph，I can modify the edge
> partitioner by configuring parameters.
> Just need to know in advance the structure of the streamgraph,This can be
> obtained by printing log.
>
>
>
> ---Original---
> From: "liu ron" Date: Tue, Apr 18, 2023 09:37 AM
> To: "dev" Subject: Re: [DISCUSS FLINKSQL PARALLELISM]
>
>
> Hi, Green
>
> Thanks for driving this discussion, in batch mode we have the Adaptive
> Batch Scheduler which automatically derives operator parallelism based on
> data volume at runtime, so we don't need to care about the parallelism.
> However, in stream mode, currently, Flink SQL can only set the parallelism
> of an operator globally, and many users would like to set the parallelism
> of an operator individually, which seems to be a pain point at the moment,
> and it would make sense to support set parallelism at operator granularity.
> Do you have any idea about the solution for this problem?
>
> Best,
> Ron
>
>
> GREEN <1286649...@qq.com.invalid> 于2023年4月14日周五 16:03写道：
>
> > Problem： 
> >
> >
> > Currently, FlinkSQL can  set a unified parallelism in the job,it
> > cannot set parallelism for each operator.
> > This can cause resource waste  On the occasion of  high
> > parallelism and small data volume.there may also be too many small
> > file  for  writing HDFS Scene.
> >
> >
> > Solution：
> > I can modify FlinkSQL to support operator parallelism.Is it
> meaningful to
> > do this？Let's discuss.
>

Re: Re: [DISCUSS] FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement

[Jing Ge]

Hi Mang,



Thanks for clarifying it. I am trying to understand your thoughts. Do you
actually mean the boundedness[1] instead of the execution modes[2]? I.e.
the atomic CTAS will be only supported for bounded data.



Best regards,

Jing



[1] https://flink.apache.org/what-is-flink/flink-architecture/

[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution_mode/#execution-mode-batchstreaming

On Wed, Apr 19, 2023 at 9:14 AM Mang Zhang  wrote:

> hi, Jing
>
> Thank you for your reply.
>
> >1. It looks like you found another way to design the atomic CTAS with new
> >serializable TwoPhaseCatalogTable instead of making Catalog serializable as
> >described in FLIP-218. Did I understand correctly?
> Yes, when I was implementing the FLIP-218 solution, I encountered problems 
> with Catalog/CatalogTable serialization deserialization, for example, after 
> deserialization CatalogTable could not be converted to Hive Table. Also, 
> Catalog serialization is still a heavy operation, but it may not actually be 
> necessary, we just need Create Table.
> Therefore, the TwoPhaseCatalogTable program is proposed, which also 
> facilitates the implementation of the subsequent data lake, ReplaceTable and 
> other functions.
>
> >2. I am a little bit confused about the isStreamingMode parameter of
> >Catalog#twoPhaseCreateTable(...), since it is the selector argument(code
> >smell) we should commonly avoid in the public interface. According to the
> >FLIP,  isStreamingMode will be used by the Catalog to determine whether to
> >support atomic or not. With this selector argument, there will be two
> >different logics built within one method and it is hard to follow without
> >reading the code or the doc carefully(another concern is to keep the doc
> >and code alway be consistent) i.e. sometimes there will be no difference by
> >using true/false isStreamingMode, sometimes they are quite different -
> >atomic vs. non-atomic. Another question is, before we call
> >Catalog#twoPhaseCreateTable(...), we have to know the value of
> >isStreamingMode. In case only non-atomic is supported for streaming mode,
> >we could just follow FLIP-218 instead of (twistedly) calling
> >Catalog#twoPhaseCreateTable(...) with a false isStreamingMode. Did I miss
> >anything here?
>
> Here's what I think about this issue, atomic CTAS wants to be the default
> behavior and only fall back to non-atomic CTAS if it's completely
> unattainable. Atomic CTAS will bring a better experience to users.
> Flink is already a stream batch unified engine, In our company kwai, many
> users are also using flink to do batch data processing, but still running
> in Stream mode.
> The boundary between stream and batch is gradually blurred, stream mode
> jobs may also FINISH, so I added the isStreamingMode parameter, this
> provides different atomicity implementations in Batch and Stream modes.
> Not only to determine if atomicity is supported, but also to help select
> different TwoPhaseCatalogTable implementations to provide different levels
> of atomicity!
>
> Looking forward to more feedback.
>
>
>
> --
>
> Best regards,
>
> Mang Zhang
>
>
>
> At 2023-04-15 04:20:40, "Jing Ge"  wrote:
> >Hi Mang,
> >
> >This is the FLIP I was looking forward to after FLIP-218. Thanks for
> >driving it. I have two questions and would like to know your thoughts,
> >thanks:
> >
> >1. It looks like you found another way to design the atomic CTAS with new
> >serializable TwoPhaseCatalogTable instead of making Catalog serializable as
> >described in FLIP-218. Did I understand correctly?
> >2. I am a little bit confused about the isStreamingMode parameter of
> >Catalog#twoPhaseCreateTable(...), since it is the selector argument(code
> >smell) we should commonly avoid in the public interface. According to the
> >FLIP,  isStreamingMode will be used by the Catalog to determine whether to
> >support atomic or not. With this selector argument, there will be two
> >different logics built within one method and it is hard to follow without
> >reading the code or the doc carefully(another concern is to keep the doc
> >and code alway be consistent) i.e. sometimes there will be no difference by
> >using true/false isStreamingMode, sometimes they are quite different -
> >atomic vs. non-atomic. Another question is, before we call
> >Catalog#twoPhaseCreateTable(...), we have to know the value of
> >isStreamingMode. In case only non-atomic is supported for streaming mode,
> >we could just follow FLIP-218 instead of (twistedly) calling
> >Catalog#twoPhaseCreateTable(...) with a false isStreamingMode. Did I miss
> >anything here?
> >
> >Best regards,
> >Jing
> >
> >On Fri, Apr 14, 2023 at 1:55 PM yuxia  wrote:
> >
> >> Hi, Mang.
> >> +1 for completing the support for atomicity of CTAS, this is very useful
> >> in batch scenarios and integrate with the data lake which support
> >> transcation.
> >>
> >> I just have one question, IIUC, the DynamiacTableSink will need to know
> >> i

[jira] [Created] (FLINK-31909) Using BroadcastUtils#withBroadcast in iteration perround mode got stuck

[Zhipeng Zhang (Jira)]

Zhipeng Zhang created FLINK-31909:
-

 Summary: Using BroadcastUtils#withBroadcast in iteration perround 
mode got stuck
 Key: FLINK-31909
 URL: https://issues.apache.org/jira/browse/FLINK-31909
 Project: Flink
  Issue Type: Bug
  Components: Library / Machine Learning
Reporter: Zhipeng Zhang
 Fix For: ml-2.3.0


Using BroadcastUtils#withBroadcastStream in iterations in per round mode could 
possibly lead to stuck. 

 

It seems that the there is a task waiting for the mail from the mailbox.

 
{code:java}
 793    "tail-map-head-Parallel Collection Source (1/1)#0" #200 prio=5 
os_prio=31 tid=0x7faabb571800 nid=0x18c03 waiting on condition 
[0x700013aae000]
 793    java.lang.Thread.State: TIMED_WAITING (parking)
 794     at sun.misc.Unsafe.park(Native Method)
 795     - parking to wait for  <0x000747805568> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 796     at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 797     at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
 798     at 
org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:149)
 799     at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:335)
 800     at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324)
 801     at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201)
 802     at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804)
 803     at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753)
 804     at 
org.apache.flink.runtime.taskmanager.Task$$Lambda$1430/1226027100.run(Unknown 
Source)
 805     at 
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948)
 806     at 
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
 807     at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741)
 808     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
 809     at java.lang.Thread.run(Thread.java:748) {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31908) cast expr to type with not null should not change nullable of expr

[jackylau (Jira)]

jackylau created FLINK-31908:


 Summary: cast expr to type with not null  should not change 
nullable of expr
 Key: FLINK-31908
 URL: https://issues.apache.org/jira/browse/FLINK-31908
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.18.0
Reporter: jackylau


{code:java}
Stream getTestSetSpecs() {
return Stream.of(
TestSetSpec.forFunction(BuiltInFunctionDefinitions.CAST)
.onFieldsWithData(new Integer[]{1, 2}, 3)
.andDataTypes(DataTypes.ARRAY(INT()), INT())
.testSqlResult(
"CAST(f0 AS ARRAY)",
new Double[]{1.0d, 2.0d},
DataTypes.ARRAY(DOUBLE().notNull(;
} {code}
but the result type should DataTypes.ARRAY(DOUBLE())), the root cause is 
calcite bug



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-connector-aws, 4.1.0 for Flink 1.17

[Danny Cranmer]

+1 (binding)

- Verified signatures and hashes
- Content of maven repo look good
- Ran simple EFO and non-EFO Kinesis source test job locally

On Wed, Apr 19, 2023 at 7:30 PM Martijn Visser 
wrote:

> +1 (binding)
>
> - Validated hashes
> - Verified signature
> - Verified that no binaries exist in the source archive
> - Build the source with Maven
> - Verified licenses
> - Verified web PRs
>
> On Wed, Apr 12, 2023 at 12:13 PM Samrat Deb  wrote:
>
> > +1 (non binding)
> >
> >
> > On Tue, 4 Apr 2023 at 1:42 AM, Elphas Tori  wrote:
> >
> > > +1 (non-binding)
> > >
> > > + verified hashes and signatures
> > > + checked local build of website pull request and approved
> > >
> > > On 2023/04/03 16:19:00 Danny Cranmer wrote:
> > > > Hi everyone,
> > > > Please review and vote on the binaries for flink-connector-aws
> version
> > > > 4.1.0-1.17, as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > The v4.1.0 source release has already been approved [1], this vote is
> > to
> > > > distribute the binaries for Flink 1.17 support.
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * all artifacts to be deployed to the Maven Central Repository [2],
> > which
> > > > are signed with the key with fingerprint
> > > > 0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3],
> > > > * website pull request listing the new release [4].
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Danny
> > > >
> > > > [1] https://lists.apache.org/thread/7q3ysg9jz5cjwdgdmgckbnqhxh44ncmv
> > > > [2]
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1602/
> > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [4] https://github.com/apache/flink-web/pull/628
> > > >
> > >
> >
>

Re: [VOTE] Release flink-connector-aws, 4.1.0 for Flink 1.17

[Danny Cranmer]

Thanks all, this VOTE is now closed and I will follow up with the results.

On Mon, Apr 24, 2023 at 8:45 AM Danny Cranmer 
wrote:

> +1 (binding)
>
> - Verified signatures and hashes
> - Content of maven repo look good
> - Ran simple EFO and non-EFO Kinesis source test job locally
>
> On Wed, Apr 19, 2023 at 7:30 PM Martijn Visser 
> wrote:
>
>> +1 (binding)
>>
>> - Validated hashes
>> - Verified signature
>> - Verified that no binaries exist in the source archive
>> - Build the source with Maven
>> - Verified licenses
>> - Verified web PRs
>>
>> On Wed, Apr 12, 2023 at 12:13 PM Samrat Deb 
>> wrote:
>>
>> > +1 (non binding)
>> >
>> >
>> > On Tue, 4 Apr 2023 at 1:42 AM, Elphas Tori 
>> wrote:
>> >
>> > > +1 (non-binding)
>> > >
>> > > + verified hashes and signatures
>> > > + checked local build of website pull request and approved
>> > >
>> > > On 2023/04/03 16:19:00 Danny Cranmer wrote:
>> > > > Hi everyone,
>> > > > Please review and vote on the binaries for flink-connector-aws
>> version
>> > > > 4.1.0-1.17, as follows:
>> > > > [ ] +1, Approve the release
>> > > > [ ] -1, Do not approve the release (please provide specific
>> comments)
>> > > >
>> > > > The v4.1.0 source release has already been approved [1], this vote
>> is
>> > to
>> > > > distribute the binaries for Flink 1.17 support.
>> > > >
>> > > > The complete staging area is available for your review, which
>> includes:
>> > > > * all artifacts to be deployed to the Maven Central Repository [2],
>> > which
>> > > > are signed with the key with fingerprint
>> > > > 0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3],
>> > > > * website pull request listing the new release [4].
>> > > >
>> > > > The vote will be open for at least 72 hours. It is adopted by
>> majority
>> > > > approval, with at least 3 PMC affirmative votes.
>> > > >
>> > > > Thanks,
>> > > > Danny
>> > > >
>> > > > [1]
>> https://lists.apache.org/thread/7q3ysg9jz5cjwdgdmgckbnqhxh44ncmv
>> > > > [2]
>> > >
>> https://repository.apache.org/content/repositories/orgapacheflink-1602/
>> > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> > > > [4] https://github.com/apache/flink-web/pull/628
>> > > >
>> > >
>> >
>>
>

[RESULT] [VOTE] flink-connector-aws 4.1.0 for Flink 1.17

[Danny Cranmer]

I'm happy to announce that we have unanimously approved this release.

There are 5 approving votes, 3 of which are binding:
* Elphas
* Robert (binding)
* Samrat
* Martijn (binding)
* Danny (binding)

There are no disapproving votes.

Thanks everyone!
Danny

[jira] [Created] (FLINK-31910) Using BroadcastUtils#withBroadcast in iteration perround mode got stuck

[Zhipeng Zhang (Jira)]

Zhipeng Zhang created FLINK-31910:
-

 Summary: Using BroadcastUtils#withBroadcast in iteration perround 
mode got stuck
 Key: FLINK-31910
 URL: https://issues.apache.org/jira/browse/FLINK-31910
 Project: Flink
  Issue Type: Bug
  Components: Library / Machine Learning
Affects Versions: ml-2.3.0
Reporter: Zhipeng Zhang


Using BroadcastUtils#withBroadcast in iteration perround mode got stuck. From 
the thread dump, it seems that the head operator and criteria node are stuck 
and waiting for a mail.

 
{code:java}
"output-head-Parallel Collection Source -> Sink: Unnamed (4/4)#0" #228 prio=5 
os_prio=31 tid=0x7f9e1d083800 nid=0x19b07 waiting on condition 
[0x700013db6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x000747a83270> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
    at 
org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:149)
    at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:335)
    at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324)
    at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201)
    at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804)
    at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753)
    at 
org.apache.flink.runtime.taskmanager.Task$$Lambda$1383/280145505.run(Unknown 
Source)
    at 
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948)
    at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
    at java.lang.Thread.run(Thread.java:748) {code}
 

The demo for this bug could be found here: 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31911) Bad address construction in SqlClientTest

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-31911:


 Summary: Bad address construction in SqlClientTest
 Key: FLINK-31911
 URL: https://issues.apache.org/jira/browse/FLINK-31911
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Client, Tests
Affects Versions: 1.16.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.18.0


The SqlClientTest constructs a host:port pair with this:

{code}
InetSocketAddress.createUnresolved(

SQL_GATEWAY_REST_ENDPOINT_EXTENSION.getTargetAddress(),

SQL_GATEWAY_REST_ENDPOINT_EXTENSION.getTargetPort())
.toString()
{code}

This is unnecessarily complicated and fails on Java 17 because the toString 
representation is _not_ guaranteed to return something of the form host:port.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31912) Upgrade bytebuddy

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-31912:


 Summary: Upgrade bytebuddy
 Key: FLINK-31912
 URL: https://issues.apache.org/jira/browse/FLINK-31912
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.18.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31913) sql-client.sh does not respect env.java.opts.all/client

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-31913:


 Summary: sql-client.sh does not respect env.java.opts.all/client
 Key: FLINK-31913
 URL: https://issues.apache.org/jira/browse/FLINK-31913
 Project: Flink
  Issue Type: Sub-task
  Components: Deployment / Scripts, Table SQL / Client
Affects Versions: 1.17.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.18.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31914) Failing to close FlinkKafkaInternalProducer created in KafkaWriter with exactly-once semantic results in memory leak

[datariver (Jira)]

datariver created FLINK-31914:
-

 Summary: Failing to close FlinkKafkaInternalProducer created in 
KafkaWriter with exactly-once semantic results in memory leak
 Key: FLINK-31914
 URL: https://issues.apache.org/jira/browse/FLINK-31914
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.15.0
Reporter: datariver
 Attachments: image-2023-04-24-16-11-22-251.png

Hi [~arvid] , If Exactly-Once writing is enabled, Kafka's transactional writing 
will be used. KafkaWriter will create FlinkKafkaInternalProducer in the 
initialization and snapshotState methods, but there is no place to close it. As 
Checkpoints increase, Producers will continue to accumulate. Each Producer 
maintains a Buffer, which will cause memory leaks and Job OOM.
By dumping an in-memory instance of Task Manager, you can see that there are a 
lot of Producers:

!image-2023-04-24-16-11-22-251.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31915) Python API incorrectly passes env.java.opts as single argument

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-31915:


 Summary: Python API incorrectly passes env.java.opts as single 
argument
 Key: FLINK-31915
 URL: https://issues.apache.org/jira/browse/FLINK-31915
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Affects Versions: 1.16.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.18.0


The python API passes all java options as a single string argument, which 
typically means that the JVM will reject them.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31916) Python API only respects deprecated env.java.opts key

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-31916:


 Summary: Python API only respects deprecated env.java.opts key
 Key: FLINK-31916
 URL: https://issues.apache.org/jira/browse/FLINK-31916
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python, Runtime / Configuration
Reporter: Chesnay Schepler
 Fix For: 1.18.0


pyflink_gateway_server.py is only reading the deprecated env.java.opts from the 
configuration.

This key should only be used as a fallback, with env.java.opts.tm/jm/client 
being the actual keys to support.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31917) Loss of Idempotence in JsonSerDe Round Trip for AggregateCall and RexNode

[Jane Chan (Jira)]

Jane Chan created FLINK-31917:
-

 Summary: Loss of Idempotence in JsonSerDe Round Trip for 
AggregateCall and RexNode
 Key: FLINK-31917
 URL: https://issues.apache.org/jira/browse/FLINK-31917
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.18.0
Reporter: Jane Chan
 Fix For: 1.18.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31918) Pulsar Source does not failing build against Flink 1.18 on nightly CI

[Danny Cranmer (Jira)]

Danny Cranmer created FLINK-31918:
-

 Summary: Pulsar Source does not failing build against Flink 1.18 
on nightly CI
 Key: FLINK-31918
 URL: https://issues.apache.org/jira/browse/FLINK-31918
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Pulsar
Reporter: Danny Cranmer


[https://github.com/apache/flink-connector-pulsar/actions/runs/4783897408/jobs/8504710249]

 
{{Error:  Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) 
on project flink-connector-pulsar: Compilation failure }}
{{[150|https://github.com/apache/flink-connector-pulsar/actions/runs/4783897408/jobs/8504710249#step:13:151]Error:
  
/home/runner/work/flink-connector-pulsar/flink-connector-pulsar/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/reader/PulsarSourceFetcherManager.java:[52,8]
 org.apache.flink.connector.pulsar.source.reader.PulsarSourceFetcherManager is 
not abstract and does not override abstract method 
removeSplits(java.util.List)
 in org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-connector-pulsar, release candidate #1

[Danny Cranmer]

+1 (binding)

- Release notes look ok
- Verified signature and hashes of source archive
- Tag is present in Github
- Reviewed Web PR
- Contents of MAven repo look good
- No binaries in the source archive
- Built source with Maven
- Verified NOTICE files

Some observations
- Nightly CI is failing build for release tag due to 1.18 incompatibility,
I cut a ticket for that https://issues.apache.org/jira/browse/FLINK-31918
- NOTICE files need year updating to 2023 however this is not a release
blocker



On Wed, Apr 19, 2023 at 7:50 PM Martijn Visser 
wrote:

> @Yufan can you make sure that these changes are properly included in the
> closed Jira tickets, under "Release Notes" ?
>
> +1 (binding)
>
> - Validated hashes
> - Verified signature
> - Verified that no binaries exist in the source archive
> - Build the source with Maven
> - Verified licenses
> - Verified web PRs
>
> On Thu, Apr 13, 2023 at 4:32 PM Yufan Sheng  wrote:
>
> > +1 from the developer of flink-connector-pulsar
> >
> > I just want to add the comments here for the newly introduced breaking
> > changes in 4.0.0
> >
> > Breaking Changes
> >
> > Some classes and interfaces which are annotated with @PublicEvolving
> > have been changed in this release. And this may affect the end-users.
> >
> > Pulsar Sink
> >
> > 1. PulsarMessage should be created by using the
> > PulsarMessage.builder() method. The PulsarMessageBuilder can’t be
> > created directly like before.
> > 2. The route method in the TopicRouter interface should return a
> > TopicPartition instead of a topic name.
> > 3. All the static methods in PulsarSerializationSchema have been
> > removed. You should set them (Schema, SerializationSchema) by directly
> > using PulsarSinkBuilder.setSerializationSchema().
> >
> > Pulsar Source
> >
> > 1. The subscription type setting has been removed from the connector.
> > All the connector’s subscriptions will be created in Exclusive type.
> > The checkpoint for Shared and Key_Shared subscriptions couldn’t be
> > used since this release.
> > 2. The setSubscriptionType method has been removed from the
> > PulsarSourceBuilder.
> > 3. The fromMessageTime method has been removed from the StartCursor.
> > Pulsar doesn’t support seeking from message time.
> > 4. The RangeGenerator interface removed the deprecated open method and
> > the keyShareMode method. Because we don’t support the Key_Shared
> > subscription now. The RangeGenerator is only used in Exclusive
> > subscription to filter the desired keys.
> > 5. TopicPartition only exposes two constructors with the
> > @PublicEvolving annotation now.
> > 6. All the static methods in PulsarDeserializationSchema have been
> > removed. You should set them (Schema, SerializationSchema) by directly
> > using PulsarSourceBuilder.setDeserializationSchema().
> > 7. The first argument of the open method in
> > PulsarDeserializationSchema has been changed from
> > InitializationContext to a new PulsarInitializationContext interface.
> >
> > On Thu, Apr 13, 2023 at 4:58 PM Martijn Visser  >
> > wrote:
> > >
> > > Hi everyone,
> > > Please review and vote on the release candidate #1 for the version
> 4.0.0,
> > > as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > >
> > > The complete staging area is available for your review, which includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release to be deployed to dist.apache.org
> > [2],
> > > which are signed with the key with fingerprint
> > > A5F3BCE4CBE993573EC5966A65321B8382B219AF [3],
> > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > * source code tag v4.0.0-rc1 [5],
> > > * website pull request listing the new release [6].
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Release Manager
> > >
> > > [1]
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352653
> > > [2]
> > >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-pulsar-4.0.0-rc1
> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1608/
> > > [5]
> > https://github.com/apache/flink-connector-pulsar/releases/tag/v4.0.0-rc1
> > > [6] https://github.com/apache/flink-web/pull/633
> >
>

Re: [VOTE] Release flink-connector-mongodb v1.0.1, release candidate #1

[Danny Cranmer]

+1 (binding)

- Release notes look ok
- Verified signature and hashes of source archive
- Tag is present in Github
- Contents of Maven repo look good
- No binaries in the source archive
- Built source with Maven, tests pass
- Verified NOTICE files

- NOTICE file year needs updating to 2023, however this is not a release
blocker


On Thu, Apr 20, 2023 at 5:56 PM Ahmed Hamdy  wrote:

> +1 (non-bindnig)
> - Checked JIRA release notes
> - verified hashes
> - verified signatures
> - built from source code succeeded
> - checked Github release tag
> - verified pom files point to version
> - reviewed and Approved web PR
>
> Best
> Ahmed Hamdy
>
>
> On Thu, 20 Apr 2023 at 02:43, Leonard Xu  wrote:
>
> >  +1 (binding)
> >
> > - verified signatures
> > - verified hashsums
> > - checked Github release tag
> > - built from source code succeeded
> > - reviewed the web PR
> >
> > Best,
> > Leonard
> >
> >
> > > On Apr 20, 2023, at 2:41 AM, Martijn Visser 
> > wrote:
> > >
> > > +1 (binding)
> > >
> > > - Validated hashes
> > > - Verified signature
> > > - Verified that no binaries exist in the source archive
> > > - Build the source with Maven
> > > - Verified licenses
> > > - Verified web PRs
> > >
> > > On Fri, Apr 14, 2023 at 9:55 AM Samrat Deb 
> > wrote:
> > >
> > >> +1 (non binding)
> > >>
> > >> On Fri, 14 Apr 2023 at 1:23 PM, Teoh, Hong
>  > >
> > >> wrote:
> > >>
> > >>> Thanks Danny
> > >>>
> > >>> +1 (non-binding)
> > >>>
> > >>> * Hashes and Signatures look good
> > >>> * All required files on dist.apache.org
> > >>> * Tag is present in Github
> > >>> * Verified source archive does not contain any binary files
> > >>> * Source archive builds using maven
> > >>>
> > >>>
> > >>> Cheers,
> > >>> Hong
> > >>>
> > >>>
> >  On 14 Apr 2023, at 06:36, Elphas Toringepi 
> > >> wrote:
> > 
> >  CAUTION: This email originated from outside of the organization. Do
> > not
> > >>> click links or open attachments unless you can confirm the sender and
> > >> know
> > >>> the content is safe.
> > 
> > 
> > 
> >  Thanks Danny
> > 
> >  +1 (non-binding)
> > 
> >  * Checked JIRA release notes
> >  * Verified signature and checksum for Apache source
> >  * Checked source code tag
> >  * Confirmed Github and Apache source releases are identical
> >  * Source code builds (JDK 11)
> >  * Approved website pull request
> > 
> >  Kind regards,
> >  Elphas
> > 
> > 
> >  On Thu, Apr 13, 2023 at 4:20 PM Danny Cranmer <
> > dannycran...@apache.org
> > >>>
> >  wrote:
> > 
> > > Hi everyone,
> > > Please review and vote on the release candidate #${RC_NUM} for the
> > >>> version
> > > ${NEW_VERSION}, as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific
> comments)
> > >
> > > This version supports both Flink 1.16.x and Flink 1.17.x
> > >
> > > The complete staging area is available for your review, which
> > >> includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release to be deployed to
> > >> dist.apache.org
> > > [2],
> > > which are signed with the key with fingerprint
> > > 0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3],
> > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > * source code tag v1.0.1-rc1 [5],
> > > * website pull request listing the new release [6].
> > >
> > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Danny
> > >
> > > [1]
> > >
> > >
> > >>>
> > >>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353094
> > > [2]
> > >
> > >
> > >>>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-mongodb-1.0.1-rc1
> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > [4]
> > >
> > >>
> https://repository.apache.org/content/repositories/orgapacheflink-1616/
> > > [5]
> > >
> > >>>
> > >>
> >
> https://github.com/apache/flink-connector-mongodb/releases/tag/v1.0.1-rc1
> > > [6] https://github.com/apache/flink-web/pull/640
> > >
> > >>>
> > >>>
> > >>
> >
> >
>

Re: [VOTE] Release flink-connector-mongodb v1.0.1, release candidate #1

[Danny Cranmer]

Thanks all, this vote is now closed. I will announce the results in a
separate thread.

On Mon, Apr 24, 2023 at 10:21 AM Danny Cranmer 
wrote:

> +1 (binding)
>
> - Release notes look ok
> - Verified signature and hashes of source archive
> - Tag is present in Github
> - Contents of Maven repo look good
> - No binaries in the source archive
> - Built source with Maven, tests pass
> - Verified NOTICE files
>
> - NOTICE file year needs updating to 2023, however this is not a release
> blocker
>
>
> On Thu, Apr 20, 2023 at 5:56 PM Ahmed Hamdy  wrote:
>
>> +1 (non-bindnig)
>> - Checked JIRA release notes
>> - verified hashes
>> - verified signatures
>> - built from source code succeeded
>> - checked Github release tag
>> - verified pom files point to version
>> - reviewed and Approved web PR
>>
>> Best
>> Ahmed Hamdy
>>
>>
>> On Thu, 20 Apr 2023 at 02:43, Leonard Xu  wrote:
>>
>> >  +1 (binding)
>> >
>> > - verified signatures
>> > - verified hashsums
>> > - checked Github release tag
>> > - built from source code succeeded
>> > - reviewed the web PR
>> >
>> > Best,
>> > Leonard
>> >
>> >
>> > > On Apr 20, 2023, at 2:41 AM, Martijn Visser > >
>> > wrote:
>> > >
>> > > +1 (binding)
>> > >
>> > > - Validated hashes
>> > > - Verified signature
>> > > - Verified that no binaries exist in the source archive
>> > > - Build the source with Maven
>> > > - Verified licenses
>> > > - Verified web PRs
>> > >
>> > > On Fri, Apr 14, 2023 at 9:55 AM Samrat Deb 
>> > wrote:
>> > >
>> > >> +1 (non binding)
>> > >>
>> > >> On Fri, 14 Apr 2023 at 1:23 PM, Teoh, Hong
>> > > >
>> > >> wrote:
>> > >>
>> > >>> Thanks Danny
>> > >>>
>> > >>> +1 (non-binding)
>> > >>>
>> > >>> * Hashes and Signatures look good
>> > >>> * All required files on dist.apache.org
>> > >>> * Tag is present in Github
>> > >>> * Verified source archive does not contain any binary files
>> > >>> * Source archive builds using maven
>> > >>>
>> > >>>
>> > >>> Cheers,
>> > >>> Hong
>> > >>>
>> > >>>
>> >  On 14 Apr 2023, at 06:36, Elphas Toringepi 
>> > >> wrote:
>> > 
>> >  CAUTION: This email originated from outside of the organization. Do
>> > not
>> > >>> click links or open attachments unless you can confirm the sender
>> and
>> > >> know
>> > >>> the content is safe.
>> > 
>> > 
>> > 
>> >  Thanks Danny
>> > 
>> >  +1 (non-binding)
>> > 
>> >  * Checked JIRA release notes
>> >  * Verified signature and checksum for Apache source
>> >  * Checked source code tag
>> >  * Confirmed Github and Apache source releases are identical
>> >  * Source code builds (JDK 11)
>> >  * Approved website pull request
>> > 
>> >  Kind regards,
>> >  Elphas
>> > 
>> > 
>> >  On Thu, Apr 13, 2023 at 4:20 PM Danny Cranmer <
>> > dannycran...@apache.org
>> > >>>
>> >  wrote:
>> > 
>> > > Hi everyone,
>> > > Please review and vote on the release candidate #${RC_NUM} for the
>> > >>> version
>> > > ${NEW_VERSION}, as follows:
>> > > [ ] +1, Approve the release
>> > > [ ] -1, Do not approve the release (please provide specific
>> comments)
>> > >
>> > > This version supports both Flink 1.16.x and Flink 1.17.x
>> > >
>> > > The complete staging area is available for your review, which
>> > >> includes:
>> > > * JIRA release notes [1],
>> > > * the official Apache source release to be deployed to
>> > >> dist.apache.org
>> > > [2],
>> > > which are signed with the key with fingerprint
>> > > 0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3],
>> > > * all artifacts to be deployed to the Maven Central Repository
>> [4],
>> > > * source code tag v1.0.1-rc1 [5],
>> > > * website pull request listing the new release [6].
>> > >
>> > > The vote will be open for at least 72 hours. It is adopted by
>> > majority
>> > > approval, with at least 3 PMC affirmative votes.
>> > >
>> > > Thanks,
>> > > Danny
>> > >
>> > > [1]
>> > >
>> > >
>> > >>>
>> > >>
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353094
>> > > [2]
>> > >
>> > >
>> > >>>
>> > >>
>> >
>> https://dist.apache.org/repos/dist/dev/flink/flink-connector-mongodb-1.0.1-rc1
>> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> > > [4]
>> > >
>> > >>
>> https://repository.apache.org/content/repositories/orgapacheflink-1616/
>> > > [5]
>> > >
>> > >>>
>> > >>
>> >
>> https://github.com/apache/flink-connector-mongodb/releases/tag/v1.0.1-rc1
>> > > [6] https://github.com/apache/flink-web/pull/640
>> > >
>> > >>>
>> > >>>
>> > >>
>> >
>> >
>>
>

[RESULT] [VOTE] flink-connector-mongodb 1.0.1, release candidate #1

[Danny Cranmer]

I'm happy to announce that we have unanimously approved this release.

There are 8 approving votes, 3 of which are binding:
* Jiabao Sun
* Elphas Toringepi
* Hong Teoh
* Samrat Deb
* Martijn Visser (binding)
* Leonard Xu (binding)
* Ahmed Hamdy
* Danny Cranmer (binding)

There are no disapproving votes.

Thanks everyone!

[jira] [Created] (FLINK-31919) Skip ClosureCleaner if object can be serialized

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-31919:


 Summary: Skip ClosureCleaner if object can be serialized
 Key: FLINK-31919
 URL: https://issues.apache.org/jira/browse/FLINK-31919
 Project: Flink
  Issue Type: Sub-task
  Components: API / Core
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.18.0


Given an object the ClosureCleaner currently recursively digs into every 
non-static/transient field of the given object. This causes a problem on Java 
17 because these reflective accesses all need to be explicitly allowed 
beforehand.

Instead, we could limit the CC to objects that fail serialization, because if 
something can be serialized there isn't anything for the CC to do.
This should allow us to avoid a lot of unnecessary reflection accesses to 
immutable JDK classes, like Strings/BigDecimals etc etc.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)

[Chesnay Schepler]

As it turns out Kryo isn't a blocker; we ran into a JDK bug.

On 31/03/2023 08:57, Chesnay Schepler wrote:

https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide

Kroy themselves state that v5 likely can't read v2 data.

However, both versions can be on the classpath without classpath as v5 
offers a versioned artifact that includes the version in the package.


It probably wouldn't be difficult to migrate a savepoint to Kryo v5, 
purely from a read/write perspective.


The bigger question is how we expose this new Kryo version in the API. 
If we stick to the versioned jar we need to either duplicate all 
current Kryo-related APIs or find a better way to integrate other 
serialization stacks.


On 30/03/2023 17:50, Piotr Nowojski wrote:

Hey,

> 1. The Flink community agrees that we upgrade Kryo to a later 
version, which means breaking all checkpoint/savepoint compatibility 
and releasing a Flink 2.0 with Java 17 support added and Java 8 and 
Flink Scala API support dropped. This is probably the quickest way, 
but would still mean that we expose Kryo in the Flink APIs, which is 
the main reason why we haven't been able to upgrade Kryo at all.


This sounds pretty bad to me.

Has anyone looked into what it would take to provide a smooth 
migration from Kryo2 -> Kryo5?


Best,
Piotrek

czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa 
 napisał(a):


Hi Martijn,

just to be sure, if all state-related classes use a POJO
serializer, Kryo will never come into play, right? Given
FLINK-16686 [1], I wonder how many users actually have jobs with
Kryo and RocksDB, but even if there aren't many, that still
leaves those who don't use RocksDB for checkpoints/savepoints.

If Kryo were to stay in the Flink APIs in v1.X, is it impossible
to let users choose between v2/v5 jars by separating them like
log4j2 jars?

[1] https://issues.apache.org/jira/browse/FLINK-16686

Regards,
Alexis.

Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser
:

Hi all,

I also saw a thread on this topic from Clayton Wohl [1] on
this topic, which I'm including in this discussion thread to
avoid that it gets lost.

From my perspective, there's two main ways to get to Java 17:

1. The Flink community agrees that we upgrade Kryo to a later
version, which means breaking all checkpoint/savepoint
compatibility and releasing a Flink 2.0 with Java 17 support
added and Java 8 and Flink Scala API support dropped. This is
probably the quickest way, but would still mean that we
expose Kryo in the Flink APIs, which is the main reason why
we haven't been able to upgrade Kryo at all.
2. There's a contributor who makes a contribution that bumps
Kryo, but either a) automagically reads in all old
checkpoints/savepoints in using Kryo v2 and writes them to
new snapshots using Kryo v5 (like is mentioned in the Kryo
migration guide [2][3] or b) provides an offline tool that
allows users that are interested in migrating their snapshots
manually before starting from a newer version. That
potentially could prevent the need to introduce a new Flink
major version. In both scenarios, ideally the contributor
would also help with avoiding the exposure of Kryo so that we
will be in a better shape in the future.

It would be good to get the opinion of the community for
either of these two options, or potentially for another one
that I haven't mentioned. If it appears that there's an
overall agreement on the direction, I would propose that a
FLIP gets created which describes the entire process.

Looking forward to the thoughts of others, including the
Users (therefore including the User ML).

Best regards,

Martijn

[1]
https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
[2]
https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
[3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5

On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi
 wrote:

I agree, there are several options to mitigate the
migration from v2 to v5.
yet, Oracle roadmap is to end JDK 11 support in September
this year.




From: ConradJam 
Sent: Thursday, March 16, 2023 4:36 AM
To: dev@flink.apache.org 
Subject: Re: [Discussion] - Release major Flink version
to support JDK 17 (LTS)

EXTERNAL EMAIL



Thanks for your start this discuss


I have been tracking this problem for a long time, until
I saw a
conversation in ISSUSE a few days ago and learned that
the Kryo version
 

[ANNOUNCE] Apache flink-connector-mongodb 1.0.1 released

[Danny Cranmer]

The Apache Flink community is very happy to announce the release of Apache
flink-connector-mongodb 1.0.1 for Apache Flink 1.16 and 1.17.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353094

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Danny

[ANNOUNCE] Apache flink-connector-aws 4.1.0 for Flink 1.17 released

[Danny Cranmer]

The Apache Flink community is very happy to announce the release of Apache
flink-connector-aws 4.1.0 for Apache Flink 1.17.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352646

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Danny

[jira] [Created] (FLINK-31920) Flaky tests

[Maximilian Michels (Jira)]

Maximilian Michels created FLINK-31920:
--

 Summary: Flaky tests 
 Key: FLINK-31920
 URL: https://issues.apache.org/jira/browse/FLINK-31920
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Reporter: Maximilian Michels


[ERROR] Errors: 
[ERROR]   FlinkOperatorTest.testConfigurationPassedToJOSDK:63 » NullPointer
[ERROR]   FlinkOperatorTest.testLeaderElectionConfig:108 » NullPointer
[ERROR]   HealthProbeTest.testHealthProbeEndpoint:64 » NullPointer
[INFO] 
[ERROR] Tests run: 323, Failures: 0, Errors: 3, Skipped: 0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31921) Create a mini cluster based metric collection test

[Maximilian Michels (Jira)]

Maximilian Michels created FLINK-31921:
--

 Summary: Create a mini cluster based metric collection test
 Key: FLINK-31921
 URL: https://issues.apache.org/jira/browse/FLINK-31921
 Project: Flink
  Issue Type: Improvement
  Components: Autoscaler, Kubernetes Operator
Reporter: Maximilian Michels


We would benefit from an e2e test for metric collection which verifies 
assumptions we have about Flink metrics.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-287: Extend Sink#InitContext to expose ExecutionConfig and JobID

[João Boto]

Hi @Gordon,

`InitContext#createInputSerializer()` is a great option and will solve more 
than one problem, but I cant find a way to get the TypeInformation on 
InitContextImpl (I can be missing something)

On current (legacy) implementations we rely on interface ´ 
InputTypeConfigurable´ to get the TypeInformation but this will not work for 
Sink2 as is not implemented (DataStream.sinkTo vs DataStream.addSink)
As a side note, the ExecutionConfig provided by this interface could not be 
used as can be changed after the call is made, for Table Planning for example 
on DefaultExecutor.configureBatchSpecificProperties()

At the end what we need to do is something like:
if (isObjectReuseEnabled()) serializer.copy(record) else record;

So responding to your question, yes last option is ok for this but I dont see 
how to implementing it as Im missing the TypeInformation on InitContextImpl.

Best regards,

On 2023/04/21 15:04:24 "Tzu-Li (Gordon) Tai" wrote:
> Do we have to introduce `InitContext#createSerializer(TypeInformation)`
> which returns TypeSerializer, or is it sufficient to only provide
> `InitContext#createInputSerializer()` which returns TypeSerializer?
> 
> I had the impression that buffering sinks like JDBC only need the
> latter. @Joao, could you confirm?
> 
> If that's the case, +1 to adding the following method signatures to
> InitContext:
> * TypeSerializer createInputSerializer()
> * boolean isObjectReuseEnabled()
> 
> Thanks,
> Gordon
> 
> On Fri, Apr 21, 2023 at 3:04 AM Zhu Zhu  wrote:
> 
> > Good point! @Gordon
> > Introducing an `InitContext#createSerializer(TypeInformation)` looks a
> > better option to me, so we do not need to introduce an unmodifiable
> > `ExecutionConfig` at this moment.
> >
> > Hope that we can make `ExecutionConfig` a read-only interface in
> > Flink 2.0. It is exposed in `RuntimeContext` to user functions already,
> > while mutating the values at runtime is actually an undefined behavior.
> >
> > Thanks,
> > Zhu
> >
> > Tzu-Li (Gordon) Tai  于2023年4月18日周二 01:02写道：
> > >
> > > Hi,
> > >
> > > Sorry for chiming in late.
> > >
> > > I'm not so sure that exposing ExecutionConfig / ReadExecutionConfig
> > > directly through Sink#InitContext is the right thing to do.
> > >
> > > 1. A lot of the read-only getter methods on ExecutionConfig are
> > irrelevant
> > > for sinks. Expanding the scope of the InitContext interface with so many
> > > irrelevant methods is probably going to make writing unit tests a pain.
> > >
> > > 2. There's actually a few getter methods on `InitContext` that have
> > > duplicate/redundant info for what ExecutionConfig exposes. For example,
> > > InitContext#getNumberOfParallelSubtasks and InitContext#getAttemptNumber
> > > currently exist and it can be confusing if users find 2 sources of that
> > > information (either via the `InitContext` and via the wrapped
> > > `ExecutionConfig`).
> > >
> > > All in all, it feels like `Sink#InitContext` was introduced initially as
> > a
> > > means to selectively only expose certain information to sinks.
> > >
> > > It looks like right now, the only requirement is that some sinks require
> > 1)
> > > isObjectReuseEnabled, and 2) TypeSerializer for the input type. Would it
> > > make sense to follow the original intent and only selectively expose
> > these?
> > > For 1), we can just add a new method to `InitContext` and forward the
> > > information from `ExecutionConfig` accessible at the operator level.
> > > For 2), would it make sense to create the serializer at the operator
> > level
> > > and then provide it through `InitContext`?
> > >
> > > Thanks,
> > > Gordon
> > >
> > > On Mon, Apr 17, 2023 at 8:23 AM Zhu Zhu  wrote:
> > >
> > > > We can let the `InitContext` return `ExecutionConfig` in the interface.
> > > > However, a `ReadableExecutionConfig` implementation should be returned
> > > > so that exceptions will be thrown if users tries to modify the
> > > > `ExecutionConfig`.
> > > >
> > > > We can rework all the setters of `ExecutionConfig` to internally
> > invoke a
> > > > `setConfiguration(...)` method. Then the `ReadableExecutionConfig` can
> > > > just override that method. But pay attention to a few exceptional
> > > > setters, i.e. those for globalJobParameters and serializers.
> > > >
> > > > We should also explicitly state in the documentation of
> > > > `InitContext #getExecutionConfig()`, that the returned
> > `ExecutionConfig`
> > > > is unmodifiable.
> > > >
> > > > Thanks,
> > > > Zhu
> > > >
> > > > João Boto  于2023年4月17日周一 16:51写道：
> > > > >
> > > > > Hi Zhu,
> > > > >
> > > > > Thanks for you time for reviewing this.
> > > > >
> > > > > Extending ´ExecutionConfig´ will allow to modify the values in the
> > > > config (this is what we want to prevent with Option2)
> > > > >
> > > > > To extend the ExecutionConfig is not simpler to do Option1 (expose
> > > > ExecutionConfig directly).
> > > > >
> > > > > Regards
> > > > >
> > > > >
> > > > >
> > > > > On 2023/04/03 09:42:28 Zhu Zhu wro

[jira] [Created] (FLINK-31922) Port over Kinesis Client configurations for retry and backoff

[Hong Liang Teoh (Jira)]

Hong Liang Teoh created FLINK-31922:
---

 Summary: Port over Kinesis Client configurations for retry and 
backoff
 Key: FLINK-31922
 URL: https://issues.apache.org/jira/browse/FLINK-31922
 Project: Flink
  Issue Type: Sub-task
Reporter: Hong Liang Teoh


Port over the Kinesis Client configurations for GetRecords, ListShards, 
DescribeStream



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Kubernetes Operator 1.5.0 release planning

[Gyula Fóra]

Hi Devs!

According to our release cadence, it's time for our next operator minor
release.

Since 1.4.0 we have made some significant improvements for both the core
operator logic and the autoscaler.

As there are still some minor tickets in the pipeline I suggest to target
May 1 as the release cut date for 1.5.0 and we can create the first RC
early next week. The planned release date would be May 8.

I am volunteering as the release manager for the release.

Please let me know if you agree or disagree with the suggested timeline.

Cheers,
Gyula

[jira] [Created] (FLINK-31923) Connector weekly runs are only testing main branches instead of all supported branches

[Martijn Visser (Jira)]

Martijn Visser created FLINK-31923:
--

 Summary: Connector weekly runs are only testing main branches 
instead of all supported branches
 Key: FLINK-31923
 URL: https://issues.apache.org/jira/browse/FLINK-31923
 Project: Flink
  Issue Type: Bug
  Components: Build System, Connectors / Common
Reporter: Martijn Visser
Assignee: Martijn Visser


We have a weekly scheduled build for connectors. That's only triggered for the 
{{main}} branches, because that's how the Github Actions {{schedule}} works, 
per 
https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule

We can resolve that by having the Github Action flow checkout multiple branches 
as a matrix to run these weekly tests.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[SUMMARY] Flink 1.18 Release Sync 04/18/2023

[Qingsheng Ren]

Hi devs,

I'd like to share some highlights in the 1.18 release sync on 04/18/2023
(Sorry for the late summary!):

- Feature list: @contributors please add your features to the list in
release 1.18 wiki page [1] so that we could track the overall progress.
- CI instabilities: owners of issues have already been pinged.
- Version management: as 1.17 has already been released, 1.15 related
resources like CIs and docker images will be removed in the coming week.

The next release sync will be on May 2nd, 2023. Feel free and welcome to
join us[2] !

[1] https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
[2]
https://us04web.zoom.us/j/79158702091?pwd=8CXPqxMzbabWkma5b0qFXI1IcLbxBh.1

Best regards,
Jing, Konstantin, Sergey and Qingsheng

Re: [ANNOUNCE] New Apache Flink PMC Member - Qingsheng Ren

[Qingsheng Ren]

Thank you all! I'm so honored and happy to work with all contributors in
the Apache Flink community. Let's make Flink better together!

Cheers,
Qingsheng

On Mon, Apr 24, 2023 at 3:39 PM Jing Ge  wrote:

> Congrats! Qingsheng!
>
>
>
> Best regards,
>
> Jing
>
> On Mon, Apr 24, 2023 at 9:35 AM Zakelly Lan  wrote:
>
> > Congratulations, Qingsheng!
> >
> > Best regards,
> > Zakelly
> >
> > On Mon, Apr 24, 2023 at 11:52 AM Matthias Pohl
> >  wrote:
> > >
> > > Congratulations, Qingsheng! :)
> > >
> > > On Mon, Apr 24, 2023, 05:17 Yangze Guo  wrote:
> > >
> > > > Congratulations, Qingsheng!
> > > >
> > > > Best,
> > > > Yangze Guo
> > > >
> > > > On Mon, Apr 24, 2023 at 10:05 AM Shuo Cheng 
> > wrote:
> > > > >
> > > > > Congratulations, Qingsheng!
> > > > >
> > > > > Best,
> > > > > Shuo
> > > > >
> > > > > On Sun, Apr 23, 2023 at 7:43 PM Sergey Nuyanzin <
> snuyan...@gmail.com
> > >
> > > > wrote:
> > > > >
> > > > > > Congratulations, Qingsheng!
> > > > > >
> > > > > > On Sun, Apr 23, 2023 at 1:37 PM Zhipeng Zhang <
> > zhangzhipe...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Congratulations, Qingsheng!
> > > > > > >
> > > > > > > Hang Ruan  于2023年4月23日周日 19:03写道：
> > > > > > > >
> > > > > > > > Congratulations, Qingsheng!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Hang
> > > > > > > >
> > > > > > > > Yanfei Lei  于2023年4月23日周日 18:33写道：
> > > > > > > >
> > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Yanfei
> > > > > > > > >
> > > > > > > > > liu ron  于2023年4月23日周日 17:47写道：
> > > > > > > > > >
> > > > > > > > > > Congratulations, Qingsheng.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Ron
> > > > > > > > > >
> > > > > > > > > > Zhanghao Chen  于2023年4月23日周日
> > > > 17:32写道：
> > > > > > > > > >
> > > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Zhanghao Chen
> > > > > > > > > > > 
> > > > > > > > > > > From: Shammon FY 
> > > > > > > > > > > Sent: Sunday, April 23, 2023 17:22
> > > > > > > > > > > To: dev@flink.apache.org 
> > > > > > > > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member -
> > > > Qingsheng
> > > > > > Ren
> > > > > > > > > > >
> > > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Shammon FY
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Apr 23, 2023 at 4:40 PM Weihua Hu <
> > > > > > huweihua@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Weihua
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Apr 23, 2023 at 3:53 PM Yun Tang <
> > myas...@live.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best
> > > > > > > > > > > > > Yun Tang
> > > > > > > > > > > > > 
> > > > > > > > > > > > > From: weijie guo 
> > > > > > > > > > > > > Sent: Sunday, April 23, 2023 14:50
> > > > > > > > > > > > > To: dev@flink.apache.org 
> > > > > > > > > > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC
> Member -
> > > > > > > Qingsheng Ren
> > > > > > > > > > > > >
> > > > > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Weijie
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Geng Biao  于2023年4月23日周日
> > 14:29写道：
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Congrats, Qingsheng!
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Biao Geng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 获取 Outlook for iOS
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > 发件人: Wencong Liu 
> > > > > > > > > > > > > > 发送时间: Sunday, April 23, 2023 11:06:39 AM
> > > > > > > > > > > > > > 收件人: dev@flink.apache.org 
> > > > > > > > > > > > > > 主题: Re:[ANNOUNCE] New Apache Flink PMC Member -
> > > > Qingsheng
> > > > > > Ren
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Congratulations, Qingsheng!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Wencong LIu
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > At 2023-04-21 19:47:52, "Jark Wu" <
> > imj...@gmail.com

Re: [VOTE] FLIP-288: Enable Dynamic Partition Discovery by Default in Kafka Source

[Martijn Visser]

+1 (binding)

On Mon, Apr 24, 2023 at 4:10 AM Feng Jin  wrote:

> +1 (non-binding)
>
>
> Best,
> Feng
>
> On Mon, Apr 24, 2023 at 9:55 AM Hang Ruan  wrote:
>
> > +1 (non-binding)
> >
> > Best,
> > Hang
> >
> > Paul Lam  于2023年4月23日周日 11:58写道：
> >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Paul Lam
> > >
> > > > 2023年4月23日 10:57，Shammon FY  写道：
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Best,
> > > > Shammon FY
> > > >
> > > > On Sun, Apr 23, 2023 at 10:35 AM Qingsheng Ren  > > > wrote:
> > > >
> > > >> Thanks for pushing this FLIP forward, Hongshun!
> > > >>
> > > >> +1 (binding)
> > > >>
> > > >> Best,
> > > >> Qingsheng
> > > >>
> > > >> On Fri, Apr 21, 2023 at 2:52 PM Hongshun Wang <
> > loserwang1...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Dear Flink Developers,
> > > >>>
> > > >>>
> > > >>> Thank you for providing feedback on FLIP-288: Enable Dynamic
> > Partition
> > > >>> Discovery by Default in Kafka Source[1] on the discussion
> thread[2].
> > > >>>
> > > >>> The goal of the FLIP is to enable partition discovery by default
> and
> > > set
> > > >>> EARLIEST offset strategy for later discovered partitions.
> > > >>>
> > > >>>
> > > >>> I am initiating a vote for this FLIP. The vote will be open for at
> > > least
> > > >> 72
> > > >>> hours, unless there is an objection or insufficient votes.
> > > >>>
> > > >>>
> > > >>> [1]: [
> > > >>>
> > > >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source](https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> 
> > <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > >
> > > >> <
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > >
> > > >>> <
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > >
> > > >>>
> > > >>> [2]: [
> > > >>>
> > > >>>
> > > >> https://lists.apache.org/thread/581f2xq5d1tlwc8gcr27gwkp3zp0wrg6] <
> > > https://lists.apache.org/thread/581f2xq5d1tlwc8gcr27gwkp3zp0wrg6]>(
> > > https://lists.apache.org/thread/581f2xq5d1tlwc8gcr27gwkp3zp0wrg6 <
> > > https://lists.apache.org/thread/581f2xq5d1tlwc8gcr27gwkp3zp0wrg6>)
> > > >>>
> > > >>>
> > > >>> Best regards,
> > > >>> Hongshun
> > >
> > >
> >
>

[jira] [Created] (FLINK-31924) [Flink operator] Flink Autoscale - Limit the max number of scale ups

[Sriram Ganesh (Jira)]

Sriram Ganesh created FLINK-31924:
-

 Summary: [Flink operator] Flink Autoscale - Limit the max number 
of scale ups
 Key: FLINK-31924
 URL: https://issues.apache.org/jira/browse/FLINK-31924
 Project: Flink
  Issue Type: Improvement
Affects Versions: kubernetes-operator-1.4.0
Reporter: Sriram Ganesh


Found that Autoscale keeps happening even after reaching max-parallelism.

{color:#172b4d}Flink version: 1.17
{color}Source: Kafka

Configuration:

 
{code:java}
flinkConfiguration:
    kubernetes.operator.job.autoscaler.enabled: "true"
    kubernetes.operator.job.autoscaler.scaling.sources.enabled: "true"
    kubernetes.operator.job.autoscaler.target.utilization: "0.6"
    kubernetes.operator.job.autoscaler.target.utilization.boundary: "0.2"
    kubernetes.operator.job.autoscaler.stabilization.interval: "1m"
    kubernetes.operator.job.autoscaler.metrics.window: "3m"{code}

Logs:
{code:java}
2023-04-24 12:29:10,738 o.a.f.k.o.c.FlinkDeploymentController [INFO 
][my-namespace/my-pod] Starting reconciliation2023-04-24 12:29:10,740 
o.a.f.k.o.s.FlinkResourceContextFactory [INFO ][my-namespace/my-pod] Getting 
service for my-job2023-04-24 12:29:10,740 o.a.f.k.o.o.JobStatusObserver  [INFO 
][my-namespace/my-pod] Observing job status2023-04-24 12:29:10,765 
o.a.f.k.o.o.JobStatusObserver  [INFO ][my-namespace/my-pod] Job status changed 
from CREATED to RUNNING2023-04-24 12:29:10,870 o.a.f.k.o.l.AuditUtils 
[INFO ][my-namespace/my-pod] >>> Event  | Info| JOBSTATUSCHANGED | Job 
status changed from CREATED to RUNNING2023-04-24 12:29:10,938 
o.a.f.k.o.l.AuditUtils [INFO ][my-namespace/my-pod] >>> Status | Info   
 | STABLE  | The resource deployment is considered to be stable and 
won’t be rolled back2023-04-24 12:29:10,986 o.a.f.k.o.a.ScalingMetricCollector 
[INFO ][my-namespace/my-pod] Skipping metric collection during stabilization 
period until 2023-04-24T12:30:10.765Z2023-04-24 12:29:10,986 
o.a.f.k.o.r.d.AbstractFlinkResourceReconciler [INFO ][my-namespace/my-pod] 
Resource fully reconciled, nothing to do...2023-04-24 12:29:10,986 
o.a.f.k.o.c.FlinkDeploymentController [INFO ][my-namespace/my-pod] End of 
reconciliation2023-04-24 12:29:25,991 o.a.f.k.o.c.FlinkDeploymentController 
[INFO ][my-namespace/my-pod] Starting reconciliation2023-04-24 12:29:25,992 
o.a.f.k.o.s.FlinkResourceContextFactory [INFO ][my-namespace/my-pod] Getting 
service for my-job2023-04-24 12:29:25,992 o.a.f.k.o.o.JobStatusObserver  [INFO 
][my-namespace/my-pod] Observing job status2023-04-24 12:29:26,005 
o.a.f.k.o.o.JobStatusObserver  [INFO ][my-namespace/my-pod] Job status 
(RUNNING) unchanged2023-04-24 12:29:26,053 o.a.f.k.o.a.ScalingMetricCollector 
[INFO ][my-namespace/my-pod] Skipping metric collection during stabilization 
period until 2023-04-24T12:30:10.765Z2023-04-24 12:29:26,054 
o.a.f.k.o.r.d.AbstractFlinkResourceReconciler [INFO ][my-namespace/my-pod] 
Resource fully reconciled, nothing to do...2023-04-24 12:29:26,054 
o.a.f.k.o.c.FlinkDeploymentController [INFO ][my-namespace/my-pod] End of 
reconciliation2023-04-24 12:29:41,059 o.a.f.k.o.c.FlinkDeploymentController 
[INFO ][my-namespace/my-pod] Starting reconciliation2023-04-24 12:29:41,060 
o.a.f.k.o.s.FlinkResourceContextFactory [INFO ][my-namespace/my-pod] Getting 
service for my-job2023-04-24 12:29:41,061 o.a.f.k.o.o.JobStatusObserver  [INFO 
][my-namespace/my-pod] Observing job status2023-04-24 12:29:41,075 
o.a.f.k.o.o.JobStatusObserver  [INFO ][my-namespace/my-pod] Job status 
(RUNNING) unchanged2023-04-24 12:29:41,116 o.a.f.k.o.a.ScalingMetricCollector 
[INFO ][my-namespace/my-pod] Skipping metric collection during stabilization 
period until 2023-04-24T12:30:10.765Z2023-04-24 12:29:41,116 
o.a.f.k.o.r.d.AbstractFlinkResourceReconciler [INFO ][my-namespace/my-pod] 
Resource fully reconciled, nothing to do...2023-04-24 12:29:41,116 
o.a.f.k.o.c.FlinkDeploymentController [INFO ][my-namespace/my-pod] End of 
reconciliation2023-04-24 12:29:56,121 o.a.f.k.o.c.FlinkDeploymentController 
[INFO ][my-namespace/my-pod] Starting reconciliation2023-04-24 12:29:56,122 
o.a.f.k.o.s.FlinkResourceContextFactory [INFO ][my-namespace/my-pod] Getting 
service for my-job2023-04-24 12:29:56,122 o.a.f.k.o.o.JobStatusObserver  [INFO 
][my-namespace/my-pod] Observing job status2023-04-24 12:29:56,134 
o.a.f.k.o.o.JobStatusObserver  [INFO ][my-namespace/my-pod] Job status 
(RUNNING) unchanged2023-04-24 12:29:56,178 o.a.f.k.o.a.ScalingMetricCollector 
[INFO ][my-namespace/my-pod] Skipping metric collection during stabilization 
period until 2023-04-24T12:30:10.765Z2023-04-24 12:29:56,179 
o.a.f.k.o.r.d.AbstractFlinkResourceReconciler [INFO ][my-namespace/my-pod] 
Resource fully reconciled, nothing to do...2023-04-24 12:29:56,179 
o.a.f.k.o.c.FlinkDeploymentController [INFO ][my-namespace/my-pod] End of 
reconciliation2023-04-24 12:30:11,183 o.a.f.k.o.

[jira] [Created] (FLINK-31925) Sync benchmark dependency versions with the version used in Flink

[Martijn Visser (Jira)]

Martijn Visser created FLINK-31925:
--

 Summary: Sync benchmark dependency versions with the version used 
in Flink
 Key: FLINK-31925
 URL: https://issues.apache.org/jira/browse/FLINK-31925
 Project: Flink
  Issue Type: Technical Debt
  Components: Benchmarks
Reporter: Martijn Visser
Assignee: Martijn Visser






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31926) Implement rename Table in GlueCatalog

[Samrat Deb (Jira)]

Samrat Deb created FLINK-31926:
--

 Summary: Implement rename Table in GlueCatalog
 Key: FLINK-31926
 URL: https://issues.apache.org/jira/browse/FLINK-31926
 Project: Flink
  Issue Type: Sub-task
Reporter: Samrat Deb


Glue catalog don't support renaming table. 
Currently marked as unsupported operation . 

This task intend to implement it later.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31927) Cassandra source raises an exception on Flink 1.16.0 on a real Cassandra cluster

[Etienne Chauchot (Jira)]

Etienne Chauchot created FLINK-31927:


 Summary: Cassandra source raises an exception on Flink 1.16.0 on a 
real Cassandra cluster
 Key: FLINK-31927
 URL: https://issues.apache.org/jira/browse/FLINK-31927
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Cassandra
Affects Versions: 1.16.0, cassandra-3.1.0
Reporter: Etienne Chauchot


CassandraSplitEnumerator#prepareSplits() raises  
java.lang.NoClassDefFoundError: com/codahale/metrics/Gauge when calling 
cluster.getMetadata() leading to NPE in CassandraSplitEnumerator#start() async 
callback. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-connector-cassandra v3.1.0, release candidate #1

[Etienne Chauchot]

Hi,

Thanks Danny for driving this new release. It now contains the new 
source, thanks.


I'm off but I wanted to test this release still. I made a very quick job 
(1) to read from a Cassandra cluster with the new source.


I found an issue: the source raises a j"ava.lang.NoClassDefFoundError: 
com/codahale/metrics/Gauge" when trying to connect to the cluster on 
Flink 1.16.0.


As I'm on vacation right now, I don't have time to solve this now but 
I'll do within a week.


vote: -1 (non-binding)

here is the blocker ticket: 
https://issues.apache.org/jira/browse/FLINK-31927


[1] 
https://github.com/echauchot/flink-samples/blob/edf4ad1624b2ad02af380efa6b5caa26bb7a274a/src/main/java/org/example/CassandraPojoSource.java


Best

Etienne

Le 19/04/2023 à 21:07, Martijn Visser a écrit :

+1 (binding)

- Validated hashes
- Verified signature
- Verified that no binaries exist in the source archive
- Build the source with Maven
- Verified licenses
- Verified web PRs

On Fri, Apr 14, 2023 at 2:42 PM Elphas Toringepi 
wrote:


Thanks Danny

+1 (non-binding)

* Checked release notes
* Validated signature and checksum
* Apache source builds with JDK 11
* Approved website PR

Kind regards,
Elphas


On Fri, Apr 14, 2023 at 1:14 PM Danny Cranmer 
wrote:


Hi everyone,
Please review and vote on the release candidate #1 for the version 3.1.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

This version supports both Flink 1.16.x and Flink 1.17.x

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org
[2],
which are signed with the key with fingerprint
0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag v3.1.0-rc1 [5],
* website pull request listing the new release [6].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Danny

[1]



https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353030

[2]



https://dist.apache.org/repos/dist/dev/flink/flink-connector-cassandra-3.1.0-rc1/

[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4]

https://repository.apache.org/content/repositories/orgapacheflink-1627

[5]


https://github.com/apache/flink-connector-cassandra/releases/tag/v3.1.0-rc1

[6] https://github.com/apache/flink-web/pull/642

Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)

[Jing Ge]

Thanks Chesnay for working on this. Would you like to share more info about
the JDK bug?

Best regards,
Jing

On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler 
wrote:

> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
>
> On 31/03/2023 08:57, Chesnay Schepler wrote:
>
>
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
>
> Kroy themselves state that v5 likely can't read v2 data.
>
> However, both versions can be on the classpath without classpath as v5
> offers a versioned artifact that includes the version in the package.
>
> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
> purely from a read/write perspective.
>
> The bigger question is how we expose this new Kryo version in the API. If
> we stick to the versioned jar we need to either duplicate all current
> Kryo-related APIs or find a better way to integrate other serialization
> stacks.
> On 30/03/2023 17:50, Piotr Nowojski wrote:
>
> Hey,
>
> > 1. The Flink community agrees that we upgrade Kryo to a later version,
> which means breaking all checkpoint/savepoint compatibility and releasing a
> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
> dropped. This is probably the quickest way, but would still mean that we
> expose Kryo in the Flink APIs, which is the main reason why we haven't been
> able to upgrade Kryo at all.
>
> This sounds pretty bad to me.
>
> Has anyone looked into what it would take to provide a smooth migration
> from Kryo2 -> Kryo5?
>
> Best,
> Piotrek
>
> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa 
> napisał(a):
>
>> Hi Martijn,
>>
>> just to be sure, if all state-related classes use a POJO serializer, Kryo
>> will never come into play, right? Given FLINK-16686 [1], I wonder how many
>> users actually have jobs with Kryo and RocksDB, but even if there aren't
>> many, that still leaves those who don't use RocksDB for
>> checkpoints/savepoints.
>>
>> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
>> users choose between v2/v5 jars by separating them like log4j2 jars?
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-16686
>>
>> Regards,
>> Alexis.
>>
>> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
>> martijnvis...@apache.org>:
>>
>>> Hi all,
>>>
>>> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
>>> which I'm including in this discussion thread to avoid that it gets lost.
>>>
>>> From my perspective, there's two main ways to get to Java 17:
>>>
>>> 1. The Flink community agrees that we upgrade Kryo to a later version,
>>> which means breaking all checkpoint/savepoint compatibility and releasing a
>>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
>>> dropped. This is probably the quickest way, but would still mean that we
>>> expose Kryo in the Flink APIs, which is the main reason why we haven't been
>>> able to upgrade Kryo at all.
>>> 2. There's a contributor who makes a contribution that bumps Kryo, but
>>> either a) automagically reads in all old checkpoints/savepoints in using
>>> Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned
>>> in the Kryo migration guide [2][3] or b) provides an offline tool that
>>> allows users that are interested in migrating their snapshots manually
>>> before starting from a newer version. That potentially could prevent the
>>> need to introduce a new Flink major version. In both scenarios, ideally the
>>> contributor would also help with avoiding the exposure of Kryo so that we
>>> will be in a better shape in the future.
>>>
>>> It would be good to get the opinion of the community for either of these
>>> two options, or potentially for another one that I haven't mentioned. If it
>>> appears that there's an overall agreement on the direction, I would propose
>>> that a FLIP gets created which describes the entire process.
>>>
>>> Looking forward to the thoughts of others, including the Users
>>> (therefore including the User ML).
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> [1]  https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
>>> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
>>> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>>>
>>> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi 
>>> wrote:
>>>
 I agree, there are several options to mitigate the migration from v2 to
 v5.
 yet, Oracle roadmap is to end JDK 11 support in September this year.



 
 From: ConradJam 
 Sent: Thursday, March 16, 2023 4:36 AM
 To: dev@flink.apache.org 
 Subject: Re: [Discussion] - Release major Flink version to support JDK
 17 (LTS)

 EXTERNAL EMAIL



 Thanks for your start this discuss


 I have been tracking this problem for a long time, until I saw a
 conversation in ISSUSE a few days ago and learned that the Kryo version
 prob

Re: [SUMMARY] Flink 1.18 Release Sync 04/18/2023

[Jing Ge]

Hi Qingsheng,

Thanks for sharing the summary!

Best regards,
Jing

On Mon, Apr 24, 2023 at 1:50 PM Qingsheng Ren  wrote:

> Hi devs,
>
> I'd like to share some highlights in the 1.18 release sync on 04/18/2023
> (Sorry for the late summary!):
>
> - Feature list: @contributors please add your features to the list in
> release 1.18 wiki page [1] so that we could track the overall progress.
> - CI instabilities: owners of issues have already been pinged.
> - Version management: as 1.17 has already been released, 1.15 related
> resources like CIs and docker images will be removed in the coming week.
>
> The next release sync will be on May 2nd, 2023. Feel free and welcome to
> join us[2] !
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> [2]
> https://us04web.zoom.us/j/79158702091?pwd=8CXPqxMzbabWkma5b0qFXI1IcLbxBh.1
>
> Best regards,
> Jing, Konstantin, Sergey and Qingsheng
>

[jira] [Created] (FLINK-31928) flink-kubernetes works not properly in IPv6 environment in k8s

[caiyi (Jira)]

caiyi created FLINK-31928:
-

 Summary: flink-kubernetes works not properly in IPv6 environment 
in k8s
 Key: FLINK-31928
 URL: https://issues.apache.org/jira/browse/FLINK-31928
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Kubernetes, Kubernetes Operator
 Environment: Kubernetes of IPv6 stack.
Reporter: caiyi


As 
[https://github.com/square/okhttp/issues/7368|https://github.com/square/okhttp/issues/7368,]
 ,okhttp3 shaded in flink-kubernetes works not properly in IPv6 stack in k8s, 
need to upgrade okhttp3 to version 4.10.0 and dependency 
org.jetbrains.kotlin:kotlin-stdlib shaded in flink-kubernetes, and release a 
new version of flink-kubernetes-operator.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31929) HighAvailabilityServicesUtils.getWebMonitorAddress works not properly in IPv6 stack of k8s

[caiyi (Jira)]

caiyi created FLINK-31929:
-

 Summary: HighAvailabilityServicesUtils.getWebMonitorAddress works 
not properly in IPv6 stack of k8s
 Key: FLINK-31929
 URL: https://issues.apache.org/jira/browse/FLINK-31929
 Project: Flink
  Issue Type: Bug
  Components: Runtime / REST
 Environment: K8s with IPv6 stack
Reporter: caiyi
 Attachments: 1.jpg

As attachment below, String.format works not properly if address is IPv6, 
new URL(protocol, address, port, "").toString() is correct.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31930) MetricQueryService works not properly in k8s with IPv6 stack

[caiyi (Jira)]

caiyi created FLINK-31930:
-

 Summary: MetricQueryService works not properly in k8s with IPv6 
stack
 Key: FLINK-31930
 URL: https://issues.apache.org/jira/browse/FLINK-31930
 Project: Flink
  Issue Type: Bug
 Environment: 1. K8s with ipv6 stack

2. Deploy flink-kubernetes-operator

3. Deploy a standalone cluster with 3 taskmanager using kubernetes 
high-availability.
Reporter: caiyi


As attachment below, MetricQueryService works not properly in k8s with IPv6 
stack.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-288: Enable Dynamic Partition Discovery by Default in Kafka Source

[Biao Geng]

+1 (non-binding)
Best,
Biao Geng

Martijn Visser  于2023年4月24日周一 20:20写道：

> +1 (binding)
>
> On Mon, Apr 24, 2023 at 4:10 AM Feng Jin  wrote:
>
> > +1 (non-binding)
> >
> >
> > Best,
> > Feng
> >
> > On Mon, Apr 24, 2023 at 9:55 AM Hang Ruan 
> wrote:
> >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Hang
> > >
> > > Paul Lam  于2023年4月23日周日 11:58写道：
> > >
> > > > +1 (non-binding)
> > > >
> > > > Best,
> > > > Paul Lam
> > > >
> > > > > 2023年4月23日 10:57，Shammon FY  写道：
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Best,
> > > > > Shammon FY
> > > > >
> > > > > On Sun, Apr 23, 2023 at 10:35 AM Qingsheng Ren  > > > > wrote:
> > > > >
> > > > >> Thanks for pushing this FLIP forward, Hongshun!
> > > > >>
> > > > >> +1 (binding)
> > > > >>
> > > > >> Best,
> > > > >> Qingsheng
> > > > >>
> > > > >> On Fri, Apr 21, 2023 at 2:52 PM Hongshun Wang <
> > > loserwang1...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Dear Flink Developers,
> > > > >>>
> > > > >>>
> > > > >>> Thank you for providing feedback on FLIP-288: Enable Dynamic
> > > Partition
> > > > >>> Discovery by Default in Kafka Source[1] on the discussion
> > thread[2].
> > > > >>>
> > > > >>> The goal of the FLIP is to enable partition discovery by default
> > and
> > > > set
> > > > >>> EARLIEST offset strategy for later discovered partitions.
> > > > >>>
> > > > >>>
> > > > >>> I am initiating a vote for this FLIP. The vote will be open for
> at
> > > > least
> > > > >> 72
> > > > >>> hours, unless there is an objection or insufficient votes.
> > > > >>>
> > > > >>>
> > > > >>> [1]: [
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source](https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> 
> > <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > >
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > >
> > > > >> <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > > >
> > > > >>> <
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > > >
> > > > >>>
> > > > >>> [2]: [
> > > > >>>
> > > > >>>
> > > > >> https://lists.apache.org/thread/581f2xq5d1tlwc8gcr27gwkp3zp0wrg6]
> <
> > > > https://lists.apache.org/thread/581f2xq5d1tlwc8gcr27gwkp3zp0wrg6]>(
> > > > https://lists.apache.org/thread/581f2xq5d1tlwc8gcr27gwkp3zp0wrg6 <
> > > > https://lists.apache.org/thread/581f2xq5d1tlwc8gcr27gwkp3zp0wrg6>)
> > > > >>>
> > > > >>>
> > > > >>> Best regards,
> > > > >>> Hongshun
> > > >
> > > >
> > >
> >
>

Re: [SUMMARY] Flink 1.18 Release Sync 04/18/2023

[Yun Tang]

Hi Qingsheng,

Thanks for sharing the sync summary.
Since most developers in China would have a 5-day Labor Day Holiday on May 2nd, 
do we consider changing the sync meeting date after May 
4th?

Best
Yun Tang

From: Jing Ge 
Sent: Tuesday, April 25, 2023 4:30
To: Qingsheng Ren 
Cc: dev ; Konstantin Knauf ; 
snuyan...@gmail.com 
Subject: Re: [SUMMARY] Flink 1.18 Release Sync 04/18/2023

Hi Qingsheng,

Thanks for sharing the summary!

Best regards,
Jing

On Mon, Apr 24, 2023 at 1:50 PM Qingsheng Ren  wrote:

> Hi devs,
>
> I'd like to share some highlights in the 1.18 release sync on 04/18/2023
> (Sorry for the late summary!):
>
> - Feature list: @contributors please add your features to the list in
> release 1.18 wiki page [1] so that we could track the overall progress.
> - CI instabilities: owners of issues have already been pinged.
> - Version management: as 1.17 has already been released, 1.15 related
> resources like CIs and docker images will be removed in the coming week.
>
> The next release sync will be on May 2nd, 2023. Feel free and welcome to
> join us[2] !
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> [2]
> https://us04web.zoom.us/j/79158702091?pwd=8CXPqxMzbabWkma5b0qFXI1IcLbxBh.1
>
> Best regards,
> Jing, Konstantin, Sergey and Qingsheng
>

RE: [DISCUSS] Status of Statefun Project

[Andre Cloutier]

Hey all,

I share the same sentiment as others here. It would be a shame if statefun
went away. At the very least, it would be nice to see it go into
maintenance mode just to maintain compatibility with Flink releases. I do
come with a different proposal though. Feel free to let me know if I'm off
base here, I'm gradually getting ramped up on the Statefun code. I think
there's two very useful bits in there that would be useful to bring into
the Data Stream API.

The implementation behind the Embedded Module could serve as a foundation
for a stateful keyed variant of the Async I/O functionality. This would
open Data Stream up to new use cases like using Flink state as a
read-through cache for an external data store (store results in Flink state
with a ttl). Then, there's the "sink as a source" functionality. This would
allow for cyclic streams without the need to write it out to another system.

If both those capabilities existed in the Data Stream API, then Statefun
could either become a much smaller project or eliminated as it wouldn't be
too much work to wire some of these operators and a http client together to
achieve something similar.

Happy to hear any feedback about this option. I'd be interested in taking a
stab at this if the core team would be open to accepting it.

Andre

On 2023/04/03 09:02:06 Martijn Visser wrote:
> Hi everyone,
>
> I want to open a discussion on the status of the Statefun Project [1] in
> Apache Flink. As you might have noticed, there hasn't been much
development
> over the past months in the Statefun repository [2]. There is currently a
> lack of active contributors and committers who are able to help with the
> maintenance of the project.
>
> In order to improve the situation, we need to solve the lack of committers
> and the lack of contributors.
>
> On the lack of committers:
>
> 1. Ideally, there are some of the current Flink committers who have the
> bandwidth and can help with reviewing PRs and merging them.
> 2. If that's not an option, it could be a consideration that current
> committers only approve and review PRs, that are approved by those who are
> willing to contribute to Statefun and if the CI passes
>
> On the lack of contributors:
>
> 3. Next to having this discussion on the Dev and User mailing list, we can
> also create a blog with a call for new contributors on the Flink project
> website, send out some tweets on the Flink / Statefun twitter accounts,
> post messages on Slack etc. In that message, we would inform how those
that
> are interested in contributing can start and where they could reach out
for
> more information.
>
> There's also option 4. where a group of interested people would split
> Statefun from the Flink project and make it a separate top level project
> under the Apache Flink umbrella (similar as recently has happened with
> Flink Table Store, which has become Apache Paimon).
>
> If we see no improvements in the coming period, we should consider
> sunsetting Statefun and communicate that clearly to the users.
>
> I'm looking forward to your thoughts.
>
> Best regards,
>
> Martijn
>
> [1] https://nightlies.apache.org/flink/flink-statefun-docs-master/
> [2] https://github.com/apache/flink-statefun
>

Re: [VOTE] FLIP-288: Enable Dynamic Partition Discovery by Default in Kafka Source

[Rui Fan]

+1 (binding)

Best,
Rui Fan

On Tue, Apr 25, 2023 at 10:06 AM Biao Geng  wrote:

> +1 (non-binding)
> Best,
> Biao Geng
>
> Martijn Visser  于2023年4月24日周一 20:20写道：
>
> > +1 (binding)
> >
> > On Mon, Apr 24, 2023 at 4:10 AM Feng Jin  wrote:
> >
> > > +1 (non-binding)
> > >
> > >
> > > Best,
> > > Feng
> > >
> > > On Mon, Apr 24, 2023 at 9:55 AM Hang Ruan 
> > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > Paul Lam  于2023年4月23日周日 11:58写道：
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Best,
> > > > > Paul Lam
> > > > >
> > > > > > 2023年4月23日 10:57，Shammon FY  写道：
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Best,
> > > > > > Shammon FY
> > > > > >
> > > > > > On Sun, Apr 23, 2023 at 10:35 AM Qingsheng Ren <
> renqs...@gmail.com
> > > > > > wrote:
> > > > > >
> > > > > >> Thanks for pushing this FLIP forward, Hongshun!
> > > > > >>
> > > > > >> +1 (binding)
> > > > > >>
> > > > > >> Best,
> > > > > >> Qingsheng
> > > > > >>
> > > > > >> On Fri, Apr 21, 2023 at 2:52 PM Hongshun Wang <
> > > > loserwang1...@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Dear Flink Developers,
> > > > > >>>
> > > > > >>>
> > > > > >>> Thank you for providing feedback on FLIP-288: Enable Dynamic
> > > > Partition
> > > > > >>> Discovery by Default in Kafka Source[1] on the discussion
> > > thread[2].
> > > > > >>>
> > > > > >>> The goal of the FLIP is to enable partition discovery by
> default
> > > and
> > > > > set
> > > > > >>> EARLIEST offset strategy for later discovered partitions.
> > > > > >>>
> > > > > >>>
> > > > > >>> I am initiating a vote for this FLIP. The vote will be open for
> > at
> > > > > least
> > > > > >> 72
> > > > > >>> hours, unless there is an objection or insufficient votes.
> > > > > >>>
> > > > > >>>
> > > > > >>> [1]: [
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source](https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> 
> > <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > >
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > >
> > > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > > >
> > > > > >> <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > > > >
> > > > > >>> <
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > > > >
> > > > > >>>
> > > > > >>> [2]: [
> > > > > >>>
> > > > > >>>
> > > > > >>
> https://lists.apache.org/thread/581f2xq5d1tlwc8gcr27gwkp3zp0wrg6]
> > <
> > > > > https://lists.apache.org/thread/581f2xq5d1tlwc8gcr27gwkp3zp0wrg6
> ]>(
> > > > > https://lists.apache.org/thread/581f2xq5d1tlwc8gcr27gwkp3zp0wrg6 <
> > > > > https://lists.apache.org/thread/581f2xq5d1tlwc8gcr27gwkp3zp0wrg6>)
> > > > > >>>
> > > > > >>>
> > > > > >>> Best regards,
> > > > > >>> Hongshun
> > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-31931) Exception history page should not link to a non-existent TM log page.

[Junrui Li (Jira)]

Junrui Li created FLINK-31931:
-

 Summary: Exception history page should not link to a non-existent 
TM log page.
 Key: FLINK-31931
 URL: https://issues.apache.org/jira/browse/FLINK-31931
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Web Frontend
Reporter: Junrui Li
 Fix For: 1.18.0


In FLINK-30358, we supported to show the task manager ID on the exception 
history page and added a link to the task manager ID to jump to the task 
manager page. However, if the task manager no longer exists when clicking the 
link to jump, the page will continue to load and the following error log will 
be continuously printed in the JM log. This will trouble users, and should be 
optimized.
{code:java}
2023-04-25 11:40:50,109 [flink-akka.actor.default-dispatcher-35] ERROR 
org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerDetailsHandler [] 
- Unhandled exception.
org.apache.flink.runtime.resourcemanager.exceptions.UnknownTaskExecutorException:
 No TaskExecutor registered under container_01.
  at 
org.apache.flink.runtime.resourcemanager.ResourceManager.requestTaskManagerDetailsInfo(ResourceManager.java:697)
 ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
  at sun.reflect.GeneratedMethodAccessor106.invoke(Unknown Source) ~[?:?]
  at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_362]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
  at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:309)
 ~[?:?]
  at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
 ~[?:?]
  at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:307)
 ~[?:?]
  at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:222)
 ~[?:?]
  at 
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:84)
 ~[?:?]
  at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:168)
 ~[?:?]
  at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) ~[?:?]
  at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) ~[?:?]
  at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) 
~[flink-scala_2.12-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
  at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) 
~[flink-scala_2.12-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
  at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) ~[?:?]
  at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) 
~[flink-scala_2.12-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
  at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) 
~[flink-scala_2.12-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
  at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) 
~[flink-scala_2.12-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
  at akka.actor.Actor.aroundReceive(Actor.scala:537) ~[?:?]
  at akka.actor.Actor.aroundReceive$(Actor.scala:535) ~[?:?]
  at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) ~[?:?]
  at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579) ~[?:?]
  at akka.actor.ActorCell.invoke(ActorCell.scala:547) ~[?:?]
  at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) ~[?:?]
  at akka.dispatch.Mailbox.run(Mailbox.scala:231) ~[?:?]
  at akka.dispatch.Mailbox.exec(Mailbox.scala:243) ~[?:?]
  at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) 
[?:1.8.0_362]
  at 
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) 
[?:1.8.0_362]
  at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) 
[?:1.8.0_362]
  at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 
[?:1.8.0_362]
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-288: Enable Dynamic Partition Discovery by Default in Kafka Source

[Jing Ge]

+1(binding)

Best regards,
Jing

On Tue, Apr 25, 2023 at 5:17 AM Rui Fan <1996fan...@gmail.com> wrote:

> +1 (binding)
>
> Best,
> Rui Fan
>
> On Tue, Apr 25, 2023 at 10:06 AM Biao Geng  wrote:
>
> > +1 (non-binding)
> > Best,
> > Biao Geng
> >
> > Martijn Visser  于2023年4月24日周一 20:20写道：
> >
> > > +1 (binding)
> > >
> > > On Mon, Apr 24, 2023 at 4:10 AM Feng Jin 
> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > > On Mon, Apr 24, 2023 at 9:55 AM Hang Ruan 
> > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Best,
> > > > > Hang
> > > > >
> > > > > Paul Lam  于2023年4月23日周日 11:58写道：
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Best,
> > > > > > Paul Lam
> > > > > >
> > > > > > > 2023年4月23日 10:57，Shammon FY  写道：
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > Best,
> > > > > > > Shammon FY
> > > > > > >
> > > > > > > On Sun, Apr 23, 2023 at 10:35 AM Qingsheng Ren <
> > renqs...@gmail.com
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Thanks for pushing this FLIP forward, Hongshun!
> > > > > > >>
> > > > > > >> +1 (binding)
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Qingsheng
> > > > > > >>
> > > > > > >> On Fri, Apr 21, 2023 at 2:52 PM Hongshun Wang <
> > > > > loserwang1...@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Dear Flink Developers,
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Thank you for providing feedback on FLIP-288: Enable Dynamic
> > > > > Partition
> > > > > > >>> Discovery by Default in Kafka Source[1] on the discussion
> > > > thread[2].
> > > > > > >>>
> > > > > > >>> The goal of the FLIP is to enable partition discovery by
> > default
> > > > and
> > > > > > set
> > > > > > >>> EARLIEST offset strategy for later discovered partitions.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> I am initiating a vote for this FLIP. The vote will be open
> for
> > > at
> > > > > > least
> > > > > > >> 72
> > > > > > >>> hours, unless there is an objection or insufficient votes.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> [1]: [
> > > > > > >>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source](https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> 
> > <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > >
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > >
> > > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > > >
> > > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > > > >
> > > > > > >> <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > > > > >
> > > > > > >>> <
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source)
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source%5D(https://cwiki.a

[jira] [Created] (FLINK-31932) Allow to configure HA on k8s without using service account

[Arkadiusz Gasinski (Jira)]

Arkadiusz Gasinski created FLINK-31932:
--

 Summary: Allow to configure HA on k8s without using service account
 Key: FLINK-31932
 URL: https://issues.apache.org/jira/browse/FLINK-31932
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Kubernetes
Reporter: Arkadiusz Gasinski


I have quite uncommon use case where I'd like to configure job manager's high 
availability on Kubernetes, but without using a service account, but rather a 
combination of username and password for interacting with the k8s' API. The 
company's policy only allows read-only service accounts, and if I want to be 
able to manipulate k8s objects (e.g., ConfigMap creation/modification) I need 
to have a dedicated account with username/password authentication. I have such 
an account, but I wasn't yet able to configure Flink's HA with it. Any advise 
greatly appreciated. Our k8s provider is OpenShift 4.x.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)