[jira] [Created] (FLINK-30808) MultipleInputITCase failed with AdaptiveBatch Scheduler
Junrui Li created FLINK-30808:
-
Summary: MultipleInputITCase failed with AdaptiveBatch Scheduler
Key: FLINK-30808
URL: https://issues.apache.org/jira/browse/FLINK-30808
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.16.0, 1.17.0
Reporter: Junrui Li
Fix For: 1.17.0
MultipleInputITCase#testRelatedInputs failed with AdaptiveBatch Scheduler.
{code:java}
//代码占位符
java.lang.UnsupportedOperationException: Forward partitioning does not allow
change of parallelism. Upstream operation: Calc[10]-14 parallelism: 1,
downstream operation: HashJoin[15]-20 parallelism: 3 You must use another
partitioning strategy, such as broadcast, rebalance, shuffle or global. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30808) MultipleInputITCase failed with AdaptiveBatch Scheduler
[
https://issues.apache.org/jira/browse/FLINK-30808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junrui Li updated FLINK-30808:
--
Description:
MultipleInputITCase#testRelatedInputs failed with AdaptiveBatch Scheduler.
{code:java}
java.lang.UnsupportedOperationException: Forward partitioning does not allow
change of parallelism. Upstream operation: Calc[10]-14 parallelism: 1,
downstream operation: HashJoin[15]-20 parallelism: 3 You must use another
partitioning strategy, such as broadcast, rebalance, shuffle or global. {code}
was:
MultipleInputITCase#testRelatedInputs failed with AdaptiveBatch Scheduler.
{code:java}
//代码占位符
java.lang.UnsupportedOperationException: Forward partitioning does not allow
change of parallelism. Upstream operation: Calc[10]-14 parallelism: 1,
downstream operation: HashJoin[15]-20 parallelism: 3 You must use another
partitioning strategy, such as broadcast, rebalance, shuffle or global. {code}
> MultipleInputITCase failed with AdaptiveBatch Scheduler
> ---
>
> Key: FLINK-30808
> URL: https://issues.apache.org/jira/browse/FLINK-30808
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Junrui Li
>Priority: Major
> Fix For: 1.17.0
>
>
> MultipleInputITCase#testRelatedInputs failed with AdaptiveBatch Scheduler.
> {code:java}
> java.lang.UnsupportedOperationException: Forward partitioning does not allow
> change of parallelism. Upstream operation: Calc[10]-14 parallelism: 1,
> downstream operation: HashJoin[15]-20 parallelism: 3 You must use another
> partitioning strategy, such as broadcast, rebalance, shuffle or global. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30808) MultipleInputITCase failed with AdaptiveBatch Scheduler
[
https://issues.apache.org/jira/browse/FLINK-30808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17681510#comment-17681510
]
Junrui Li commented on FLINK-30808:
---
cc [~zhuzh]
> MultipleInputITCase failed with AdaptiveBatch Scheduler
> ---
>
> Key: FLINK-30808
> URL: https://issues.apache.org/jira/browse/FLINK-30808
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Junrui Li
>Priority: Major
> Fix For: 1.17.0
>
>
> MultipleInputITCase#testRelatedInputs failed with AdaptiveBatch Scheduler.
> {code:java}
> java.lang.UnsupportedOperationException: Forward partitioning does not allow
> change of parallelism. Upstream operation: Calc[10]-14 parallelism: 1,
> downstream operation: HashJoin[15]-20 parallelism: 3 You must use another
> partitioning strategy, such as broadcast, rebalance, shuffle or global. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30808) MultipleInputITCase failed with AdaptiveBatch Scheduler
[
https://issues.apache.org/jira/browse/FLINK-30808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17681514#comment-17681514
]
Junrui Li commented on FLINK-30808:
---
The failure is because currently in StreamGraph#createActualEdge, if the
partitioner is a ForwardPartitioner, then the operator parallelism of the
upstream and downstream of the partitioner will be required to be consistent.
However, for the ForConsecutiveHashPartitioner used in this case, it is not
required that the parallelism of upstream and downstream operators must be
consistent. So we can fix this bug by fixing the StreamGraph#createActualEdge.
> MultipleInputITCase failed with AdaptiveBatch Scheduler
> ---
>
> Key: FLINK-30808
> URL: https://issues.apache.org/jira/browse/FLINK-30808
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Junrui Li
>Assignee: Junrui Li
>Priority: Major
> Fix For: 1.17.0
>
>
> MultipleInputITCase#testRelatedInputs failed with AdaptiveBatch Scheduler.
> {code:java}
> java.lang.UnsupportedOperationException: Forward partitioning does not allow
> change of parallelism. Upstream operation: Calc[10]-14 parallelism: 1,
> downstream operation: HashJoin[15]-20 parallelism: 3 You must use another
> partitioning strategy, such as broadcast, rebalance, shuffle or global. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30808) MultipleInputITCase failed with AdaptiveBatch Scheduler
[
https://issues.apache.org/jira/browse/FLINK-30808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17681514#comment-17681514
]
Junrui Li edited comment on FLINK-30808 at 1/28/23 6:25 AM:
The failure is because currently in StreamGraph#createActualEdge, if the
partitioner is a ForwardPartitioner, then the operator parallelism of the
upstream and downstream of the partitioner will be required to be consistent.
However, for the ForwardForConsecutiveHashPartitioner(which is a subclass of
ForwardPartitioner) used in this case, it is not required that the parallelism
of upstream and downstream operators must be consistent. So we can fix this bug
by fixing the StreamGraph#createActualEdge.
was (Author: JIRAUSER293716):
The failure is because currently in StreamGraph#createActualEdge, if the
partitioner is a ForwardPartitioner, then the operator parallelism of the
upstream and downstream of the partitioner will be required to be consistent.
However, for the ForConsecutiveHashPartitioner used in this case, it is not
required that the parallelism of upstream and downstream operators must be
consistent. So we can fix this bug by fixing the StreamGraph#createActualEdge.
> MultipleInputITCase failed with AdaptiveBatch Scheduler
> ---
>
> Key: FLINK-30808
> URL: https://issues.apache.org/jira/browse/FLINK-30808
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Junrui Li
>Assignee: Junrui Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0
>
>
> MultipleInputITCase#testRelatedInputs failed with AdaptiveBatch Scheduler.
> {code:java}
> java.lang.UnsupportedOperationException: Forward partitioning does not allow
> change of parallelism. Upstream operation: Calc[10]-14 parallelism: 1,
> downstream operation: HashJoin[15]-20 parallelism: 3 You must use another
> partitioning strategy, such as broadcast, rebalance, shuffle or global. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30821) The optimized exec plan generated by OverAggregateTest#testDiffPartitionKeysWithDiffOrderKeys2 in the case of all-blocking is not as expected
Junrui Li created FLINK-30821:
-
Summary: The optimized exec plan generated by
OverAggregateTest#testDiffPartitionKeysWithDiffOrderKeys2 in the case of
all-blocking is not as expected
Key: FLINK-30821
URL: https://issues.apache.org/jira/browse/FLINK-30821
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.16.0, 1.17.0
Reporter: Junrui Li
Fix For: 1.17.0, 1.16.2
The optimized exec plan generated by
OverAggregateTest#testDiffPartitionKeysWithDiffOrderKeys2 in the case of
all-blocking is that
{code:java}
Calc(select=[CASE((w0$o0 > 0), w0$o1, null:INTEGER) AS EXPR$0, w1$o0 AS EXPR$1,
w2$o0 AS EXPR$2, w0$o2 AS EXPR$3, CAST((CASE((w3$o0 > 0), w3$o1, null:INTEGER)
/ w3$o0) AS INTEGER) AS EXPR$4])
+- OverAggregate(partitionBy=[c], orderBy=[a ASC], window#0=[MAX(a) AS w0$o0
RANG BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], select=[a, b, c, w0$o2,
w1$o0, w0$o1, w3$o0, w3$o1, w2$o0, w0$o0])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[c ASC, a ASC])
+- Exchange(distribution=[hash[c]])
+- OverAggregate(partitionBy=[b], orderBy=[c ASC],
window#0=[COUNT(a) AS w3$o0, $SUM0(a) AS w3$o1 RANG BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW], window#1=[RANK(*) AS w2$o0 RANG BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW], select=[a, b, c, w0$o2, w1$o0, w0$o1, w3$o0, w3$o1, w2$o0])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[b ASC, c ASC])
+- Exchange(distribution=[hash[b]])
+- OverAggregate(orderBy=[c ASC, a ASC],
window#0=[MIN(a) AS w0$o1 RANG BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW],
select=[a, b, c, w0$o2, w1$o0, w0$o1])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[c ASC, a ASC])
+- Exchange(distribution=[forward])
+- OverAggregate(orderBy=[b ASC],
window#0=[COUNT(a) AS w0$o2, $SUM0(a) AS w1$o0 RANG BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW], select=[a, b, c, w0$o2, w1$o0])
+- Sort(orderBy=[b ASC])
+- Exchange(distribution=[single])
+-
LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable,
source: [TestTableSource(a, b, c)]]], fields=[a, b, c])
{code}
However, the expected plan is that
{code:java}
Calc(select=[CASE((w0$o0 > 0), w0$o1, null:INTEGER) AS EXPR$0, w1$o0 AS EXPR$1,
w2$o0 AS EXPR$2, w0$o2 AS EXPR$3, CAST((CASE((w3$o0 > 0), w3$o1, null:INTEGER)
/ w3$o0) AS INTEGER) AS EXPR$4])
+- OverAggregate(partitionBy=[c], orderBy=[a ASC], window#0=[MAX(a) AS w0$o0
RANG BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], select=[a, b, c, w0$o2,
w1$o0, w0$o1, w3$o0, w3$o1, w2$o0, w0$o0])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[c ASC, a ASC])
+- Exchange(distribution=[hash[c]])
+- OverAggregate(partitionBy=[b], orderBy=[c ASC],
window#0=[COUNT(a) AS w3$o0, $SUM0(a) AS w3$o1 RANG BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW], window#1=[RANK(*) AS w2$o0 RANG BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW], select=[a, b, c, w0$o2, w1$o0, w0$o1, w3$o0, w3$o1, w2$o0])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[b ASC, c ASC])
+- Exchange(distribution=[hash[b]])
+- OverAggregate(orderBy=[c ASC, a ASC],
window#0=[MIN(a) AS w0$o1 RANG BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW],
select=[a, b, c, w0$o2, w1$o0, w0$o1])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[c ASC, a ASC])
+- Exchange(distribution=[forward])
+- OverAggregate(orderBy=[b ASC],
window#0=[COUNT(a) AS w0$o2, $SUM0(a) AS w1$o0 RANG BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW], select=[a, b, c, w0$o2, w1$o0])
+- Exchange (distribution=[forward])
+- Sort(orderBy=[b ASC])
+- Exchange(distribution=[single])
+-
LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable,
source: [TestTableSource(a, b, c)]]], fields=[a, b, c])
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30821) The optimized exec plan generated by OverAggregateTest#testDiffPartitionKeysWithDiffOrderKeys2 in the case of all-blocking is not as expected
[
https://issues.apache.org/jira/browse/FLINK-30821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junrui Li updated FLINK-30821:
--
Description:
The optimized exec plan generated by
OverAggregateTest#testDiffPartitionKeysWithDiffOrderKeys2 in the case of
all-blocking is that
{code:java}
Calc(select=[CASE((w0$o0 > 0), w0$o1, null:INTEGER) AS EXPR$0, w1$o0 AS EXPR$1,
w2$o0 AS EXPR$2, w0$o2 AS EXPR$3, CAST((CASE((w3$o0 > 0), w3$o1, null:INTEGER)
/ w3$o0) AS INTEGER) AS EXPR$4])
+- OverAggregate(partitionBy=[c], orderBy=[a ASC], window#0=[MAX(a) AS w0$o0
RANG BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], select=[a, b, c, w0$o2,
w1$o0, w0$o1, w3$o0, w3$o1, w2$o0, w0$o0])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[c ASC, a ASC])
+- Exchange(distribution=[hash[c]])
+- OverAggregate(partitionBy=[b], orderBy=[c ASC],
window#0=[COUNT(a) AS w3$o0, $SUM0(a) AS w3$o1 RANG BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW], window#1=[RANK(*) AS w2$o0 RANG BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW], select=[a, b, c, w0$o2, w1$o0, w0$o1, w3$o0, w3$o1, w2$o0])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[b ASC, c ASC])
+- Exchange(distribution=[hash[b]])
+- OverAggregate(orderBy=[c ASC, a ASC],
window#0=[MIN(a) AS w0$o1 RANG BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW],
select=[a, b, c, w0$o2, w1$o0, w0$o1])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[c ASC, a ASC])
+- Exchange(distribution=[forward])
+- OverAggregate(orderBy=[b ASC],
window#0=[COUNT(a) AS w0$o2, $SUM0(a) AS w1$o0 RANG BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW], select=[a, b, c, w0$o2, w1$o0])
+- Sort(orderBy=[b ASC])
+- Exchange(distribution=[single])
+-
LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable,
source: [TestTableSource(a, b, c)]]], fields=[a, b, c])
{code}
However, the expected plan is that
{code:java}
Calc(select=[CASE((w0$o0 > 0), w0$o1, null:INTEGER) AS EXPR$0, w1$o0 AS EXPR$1,
w2$o0 AS EXPR$2, w0$o2 AS EXPR$3, CAST((CASE((w3$o0 > 0), w3$o1, null:INTEGER)
/ w3$o0) AS INTEGER) AS EXPR$4])
+- OverAggregate(partitionBy=[c], orderBy=[a ASC], window#0=[MAX(a) AS w0$o0
RANG BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], select=[a, b, c, w0$o2,
w1$o0, w0$o1, w3$o0, w3$o1, w2$o0, w0$o0])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[c ASC, a ASC])
+- Exchange(distribution=[hash[c]])
+- OverAggregate(partitionBy=[b], orderBy=[c ASC],
window#0=[COUNT(a) AS w3$o0, $SUM0(a) AS w3$o1 RANG BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW], window#1=[RANK(*) AS w2$o0 RANG BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW], select=[a, b, c, w0$o2, w1$o0, w0$o1, w3$o0, w3$o1, w2$o0])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[b ASC, c ASC])
+- Exchange(distribution=[hash[b]])
+- OverAggregate(orderBy=[c ASC, a ASC],
window#0=[MIN(a) AS w0$o1 RANG BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW],
select=[a, b, c, w0$o2, w1$o0, w0$o1])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[c ASC, a ASC])
+- Exchange(distribution=[forward])
+- OverAggregate(orderBy=[b ASC],
window#0=[COUNT(a) AS w0$o2, $SUM0(a) AS w1$o0 RANG BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW], select=[a, b, c, w0$o2, w1$o0])
+- Exchange (distribution=[forward])
+- Sort(orderBy=[b ASC])
+- Exchange(distribution=[single])
+-
LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable,
source: [TestTableSource(a, b, c)]]], fields=[a, b, c])
{code}
was:
The optimized exec plan generated by
OverAggregateTest#testDiffPartitionKeysWithDiffOrderKeys2 in the case of
all-blocking is that
{code:java}
Calc(select=[CASE((w0$o0 > 0), w0$o1, null:INTEGER) AS EXPR$0, w1$o0 AS EXPR$1,
w2$o0 AS EXPR$2, w0$o2 AS EXPR$3, CAST((CASE((w3$o0 > 0), w3$o1, null:INTEGER)
/ w3$o0) AS INTEGER) AS EXPR$4])
+- OverAggregate(partitionBy=[c], orderBy=[a ASC], window#0=[MAX(a) AS w0$o0
RANG BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], select=[a, b, c, w0$o2,
w1$o0, w0$o1, w3$o0, w3$o1, w2$o0, w0$o0])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[c ASC, a ASC])
+- Exchange(distribution=[hash[c]])
+- OverAggregate(partitionBy=[b], or
[jira] [Commented] (FLINK-30821) The optimized exec plan generated by OverAggregateTest#testDiffPartitionKeysWithDiffOrderKeys2 in the case of all-blocking is not as expected
[
https://issues.apache.org/jira/browse/FLINK-30821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17681853#comment-17681853
]
Junrui Li commented on FLINK-30821:
---
cc [~zhuzh] [~godfrey]
> The optimized exec plan generated by
> OverAggregateTest#testDiffPartitionKeysWithDiffOrderKeys2 in the case of
> all-blocking is not as expected
> -
>
> Key: FLINK-30821
> URL: https://issues.apache.org/jira/browse/FLINK-30821
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Junrui Li
>Priority: Major
> Fix For: 1.17.0, 1.16.2
>
>
> The optimized exec plan generated by
> OverAggregateTest#testDiffPartitionKeysWithDiffOrderKeys2 in the case of
> all-blocking is that
> {code:java}
> Calc(select=[CASE((w0$o0 > 0), w0$o1, null:INTEGER) AS EXPR$0, w1$o0 AS
> EXPR$1, w2$o0 AS EXPR$2, w0$o2 AS EXPR$3, CAST((CASE((w3$o0 > 0), w3$o1,
> null:INTEGER) / w3$o0) AS INTEGER) AS EXPR$4])
> +- OverAggregate(partitionBy=[c], orderBy=[a ASC], window#0=[MAX(a) AS w0$o0
> RANG BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], select=[a, b, c, w0$o2,
> w1$o0, w0$o1, w3$o0, w3$o1, w2$o0, w0$o0])
>+- Exchange(distribution=[forward])
> +- Sort(orderBy=[c ASC, a ASC])
> +- Exchange(distribution=[hash[c]])
> +- OverAggregate(partitionBy=[b], orderBy=[c ASC],
> window#0=[COUNT(a) AS w3$o0, $SUM0(a) AS w3$o1 RANG BETWEEN UNBOUNDED
> PRECEDING AND CURRENT ROW], window#1=[RANK(*) AS w2$o0 RANG BETWEEN UNBOUNDED
> PRECEDING AND CURRENT ROW], select=[a, b, c, w0$o2, w1$o0, w0$o1, w3$o0,
> w3$o1, w2$o0])
>+- Exchange(distribution=[forward])
> +- Sort(orderBy=[b ASC, c ASC])
> +- Exchange(distribution=[hash[b]])
> +- OverAggregate(orderBy=[c ASC, a ASC],
> window#0=[MIN(a) AS w0$o1 RANG BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW],
> select=[a, b, c, w0$o2, w1$o0, w0$o1])
>+- Exchange(distribution=[forward])
> +- Sort(orderBy=[c ASC, a ASC])
> +- Exchange(distribution=[forward])
> +- OverAggregate(orderBy=[b ASC],
> window#0=[COUNT(a) AS w0$o2, $SUM0(a) AS w1$o0 RANG BETWEEN UNBOUNDED
> PRECEDING AND CURRENT ROW], select=[a, b, c, w0$o2, w1$o0])
>+- Sort(orderBy=[b ASC])
> +- Exchange(distribution=[single])
> +-
> LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [TestTableSource(a, b, c)]]], fields=[a, b, c])
> {code}
> However, the expected plan is that
> {code:java}
> Calc(select=[CASE((w0$o0 > 0), w0$o1, null:INTEGER) AS EXPR$0, w1$o0 AS
> EXPR$1, w2$o0 AS EXPR$2, w0$o2 AS EXPR$3, CAST((CASE((w3$o0 > 0), w3$o1,
> null:INTEGER) / w3$o0) AS INTEGER) AS EXPR$4])
> +- OverAggregate(partitionBy=[c], orderBy=[a ASC], window#0=[MAX(a) AS w0$o0
> RANG BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], select=[a, b, c, w0$o2,
> w1$o0, w0$o1, w3$o0, w3$o1, w2$o0, w0$o0])
>+- Exchange(distribution=[forward])
> +- Sort(orderBy=[c ASC, a ASC])
> +- Exchange(distribution=[hash[c]])
> +- OverAggregate(partitionBy=[b], orderBy=[c ASC],
> window#0=[COUNT(a) AS w3$o0, $SUM0(a) AS w3$o1 RANG BETWEEN UNBOUNDED
> PRECEDING AND CURRENT ROW], window#1=[RANK(*) AS w2$o0 RANG BETWEEN UNBOUNDED
> PRECEDING AND CURRENT ROW], select=[a, b, c, w0$o2, w1$o0, w0$o1, w3$o0,
> w3$o1, w2$o0])
>+- Exchange(distribution=[forward])
> +- Sort(orderBy=[b ASC, c ASC])
> +- Exchange(distribution=[hash[b]])
> +- OverAggregate(orderBy=[c ASC, a ASC],
> window#0=[MIN(a) AS w0$o1 RANG BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW],
> select=[a, b, c, w0$o2, w1$o0, w0$o1])
>+- Exchange(distribution=[forward])
> +- Sort(orderBy=[c ASC, a ASC])
> +- Exchange(distribution=[forward])
> +- OverAggregate(orderBy=[b ASC],
> window#0=[COUNT(a) AS w0$o2, $SUM0(a) AS w1$o0 RANG BETWEEN UNBOUNDED
> PRECEDING AND CURRENT ROW], select=[a, b, c, w0$o2, w1$o0])
>+- Exchange (distribution=[forward])
>
> +- Sort(orderBy=[b ASC])
> +-
> Exchange(distribution=[single])
> +-
[jira] [Commented] (FLINK-29427) LookupJoinITCase failed with classloader problem
[
https://issues.apache.org/jira/browse/FLINK-29427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682331#comment-17682331
]
Junrui Li commented on FLINK-29427:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45438&view=logs&j=0c940707-2659-5648-cbe6-a1ad63045f0a&t=075c2716-8010-5565-fe08-3c4bb45824a4
> LookupJoinITCase failed with classloader problem
>
>
> Key: FLINK-29427
> URL: https://issues.apache.org/jira/browse/FLINK-29427
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Huang Xingbo
>Assignee: Qingsheng Ren
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.17.0
>
>
> {code:java}
> 2022-09-27T02:49:20.9501313Z Sep 27 02:49:20 Caused by:
> org.codehaus.janino.InternalCompilerException: Compiling
> "KeyProjection$108341": Trying to access closed classloader. Please check if
> you store classloaders directly or indirectly in static fields. If the
> stacktrace suggests that the leak occurs in a third party library and cannot
> be fixed immediately, you can disable this check with the configuration
> 'classloader.check-leaked-classloader'.
> 2022-09-27T02:49:20.9502654Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382)
> 2022-09-27T02:49:20.9503366Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237)
> 2022-09-27T02:49:20.9504044Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465)
> 2022-09-27T02:49:20.9504704Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:216)
> 2022-09-27T02:49:20.9505341Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207)
> 2022-09-27T02:49:20.9505965Z Sep 27 02:49:20 at
> org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80)
> 2022-09-27T02:49:20.9506584Z Sep 27 02:49:20 at
> org.codehaus.commons.compiler.Cookable.cook(Cookable.java:75)
> 2022-09-27T02:49:20.9507261Z Sep 27 02:49:20 at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:104)
> 2022-09-27T02:49:20.9507883Z Sep 27 02:49:20 ... 30 more
> 2022-09-27T02:49:20.9509266Z Sep 27 02:49:20 Caused by:
> java.lang.IllegalStateException: Trying to access closed classloader. Please
> check if you store classloaders directly or indirectly in static fields. If
> the stacktrace suggests that the leak occurs in a third party library and
> cannot be fixed immediately, you can disable this check with the
> configuration 'classloader.check-leaked-classloader'.
> 2022-09-27T02:49:20.9510835Z Sep 27 02:49:20 at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:184)
> 2022-09-27T02:49:20.9511760Z Sep 27 02:49:20 at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:192)
> 2022-09-27T02:49:20.9512456Z Sep 27 02:49:20 at
> java.lang.Class.forName0(Native Method)
> 2022-09-27T02:49:20.9513014Z Sep 27 02:49:20 at
> java.lang.Class.forName(Class.java:348)
> 2022-09-27T02:49:20.9513649Z Sep 27 02:49:20 at
> org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:89)
> 2022-09-27T02:49:20.9514339Z Sep 27 02:49:20 at
> org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:312)
> 2022-09-27T02:49:20.9514990Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:8556)
> 2022-09-27T02:49:20.9515659Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6749)
> 2022-09-27T02:49:20.9516337Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6594)
> 2022-09-27T02:49:20.9516989Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6573)
> 2022-09-27T02:49:20.9517632Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.access$13900(UnitCompiler.java:215)
> 2022-09-27T02:49:20.9518319Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22$1.visitReferenceType(UnitCompiler.java:6481)
> 2022-09-27T02:49:20.9519018Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22$1.visitReferenceType(UnitCompiler.java:6476)
> 2022-09-27T02:49:20.9519680Z Sep 27 02:49:20 at
> org.codehaus.janino.Java$ReferenceType.accept(Java.java:3928)
> 2022-09-27T02:49:20.9520386Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22.visitType(UnitCompiler.java:6476)
> 2022-09-27T02:49:20.9521042Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22.visitType(UnitCompiler.java:6469)
> 2022-09-27T02:49:2
[jira] [Commented] (FLINK-29405) InputFormatCacheLoaderTest is unstable
[
https://issues.apache.org/jira/browse/FLINK-29405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682374#comment-17682374
]
Junrui Li commented on FLINK-29405:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45443&view=logs&j=0c940707-2659-5648-cbe6-a1ad63045f0a&t=075c2716-8010-5565-fe08-3c4bb45824a4
> InputFormatCacheLoaderTest is unstable
> --
>
> Key: FLINK-29405
> URL: https://issues.apache.org/jira/browse/FLINK-29405
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Chesnay Schepler
>Assignee: Alexander Smirnov
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.17.0
>
>
> #testExceptionDuringReload/#testCloseAndInterruptDuringReload fail reliably
> when run in a loop.
> {code}
> java.lang.AssertionError:
> Expecting AtomicInteger(0) to have value:
> 0
> but did not.
> at
> org.apache.flink.table.runtime.functions.table.fullcache.inputformat.InputFormatCacheLoaderTest.testCloseAndInterruptDuringReload(InputFormatCacheLoaderTest.java:161)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30838) Update documentation about the AdaptiveBatchScheduler
Junrui Li created FLINK-30838: - Summary: Update documentation about the AdaptiveBatchScheduler Key: FLINK-30838 URL: https://issues.apache.org/jira/browse/FLINK-30838 Project: Flink Issue Type: Sub-task Components: Documentation Reporter: Junrui Li Fix For: 1.17.0 Documentation is needed to update to help users how to enable the AdaptiveBatchScheduler and properly configuring it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30846) SpeculativeSchedulerITCase.testSpeculativeExecutionOfInputFormatSource fails
[
https://issues.apache.org/jira/browse/FLINK-30846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682866#comment-17682866
]
Junrui Li commented on FLINK-30846:
---
[~wanglijie] This issue will not be a block to production availability, it will
only appear in the test. I will fix it immediately, thanks!
> SpeculativeSchedulerITCase.testSpeculativeExecutionOfInputFormatSource fails
>
>
> Key: FLINK-30846
> URL: https://issues.apache.org/jira/browse/FLINK-30846
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: Junrui Li
>Priority: Blocker
> Labels: test-stability
>
> {{SpeculativeSchedulerITCase.testSpeculativeExecutionOfInputFormatSource}} is
> timing out
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45440&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=8599
> {code}
> Jan 31 02:02:28 "ForkJoinPool-1-worker-25" #27 daemon prio=5 os_prio=0
> tid=0x7fcf74f2b800 nid=0x5476 waiting on condition [0x7fce2b078000]
> Jan 31 02:02:28java.lang.Thread.State: WAITING (parking)
> Jan 31 02:02:28 at sun.misc.Unsafe.park(Native Method)
> Jan 31 02:02:28 - parking to wait for <0xa22933e0> (a
> java.util.concurrent.CompletableFuture$Signaller)
> Jan 31 02:02:28 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> Jan 31 02:02:28 at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> Jan 31 02:02:28 at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
> Jan 31 02:02:28 at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> Jan 31 02:02:28 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> Jan 31 02:02:28 at
> org.apache.flink.test.scheduling.SpeculativeSchedulerITCase.executeJob(SpeculativeSchedulerITCase.java:216)
> Jan 31 02:02:28 at
> org.apache.flink.test.scheduling.SpeculativeSchedulerITCase.testSpeculativeExecutionOfInputFormatSource(SpeculativeSchedulerITCase.java:162)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30875) Remove legacy AdaptiveBatchScheduler min parallelism configuration call.
Junrui Li created FLINK-30875: - Summary: Remove legacy AdaptiveBatchScheduler min parallelism configuration call. Key: FLINK-30875 URL: https://issues.apache.org/jira/browse/FLINK-30875 Project: Flink Issue Type: Bug Components: Runtime / Coordination Affects Versions: 1.17.0 Reporter: Junrui Li Fix For: 1.17.0 In FLINK-30686, we deprecated the JobManagerOptions's AdaptiveBatchScheduler configuration. However, these configuration items still have some calls. And we should change these calls to new configuration. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30875) Remove legacy AdaptiveBatchScheduler configuration call.
[ https://issues.apache.org/jira/browse/FLINK-30875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-30875: -- Summary: Remove legacy AdaptiveBatchScheduler configuration call. (was: Remove legacy AdaptiveBatchScheduler min parallelism configuration call. ) > Remove legacy AdaptiveBatchScheduler configuration call. > - > > Key: FLINK-30875 > URL: https://issues.apache.org/jira/browse/FLINK-30875 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.17.0 >Reporter: Junrui Li >Priority: Critical > Fix For: 1.17.0 > > > In FLINK-30686, we deprecated the JobManagerOptions's AdaptiveBatchScheduler > configuration. However, these configuration items still have some calls. And > we should change these calls to new configuration. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30889) Rename the configuration key SPECULATIVE_ENABLED.
Junrui Li created FLINK-30889: - Summary: Rename the configuration key SPECULATIVE_ENABLED. Key: FLINK-30889 URL: https://issues.apache.org/jira/browse/FLINK-30889 Project: Flink Issue Type: Bug Components: Runtime / Configuration Affects Versions: 1.17.0 Reporter: Junrui Li Fix For: 1.17.0 In FLINK-30686 we refined the adaptive batch configuration. However, the key of a configuration item `SPECULATIVE_ENABLED` has been modified incorrectly. According to flip-283, it should be modified to `execution.adaptive.speculative.enabled`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30889) Rename the configuration key SPECULATIVE_ENABLED.
[ https://issues.apache.org/jira/browse/FLINK-30889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-30889: -- Description: In FLINK-30686 we refined the adaptive batch configuration. However, the key of a configuration item `SPECULATIVE_ENABLED` has been modified incorrectly. According to flip-283, it should be modified to `execution.batch.speculative.enabled`. (was: In FLINK-30686 we refined the adaptive batch configuration. However, the key of a configuration item `SPECULATIVE_ENABLED` has been modified incorrectly. According to flip-283, it should be modified to `execution.adaptive.speculative.enabled`.) > Rename the configuration key SPECULATIVE_ENABLED. > - > > Key: FLINK-30889 > URL: https://issues.apache.org/jira/browse/FLINK-30889 > Project: Flink > Issue Type: Bug > Components: Runtime / Configuration >Affects Versions: 1.17.0 >Reporter: Junrui Li >Priority: Critical > Fix For: 1.17.0 > > > In FLINK-30686 we refined the adaptive batch configuration. However, the key > of a configuration item `SPECULATIVE_ENABLED` has been modified incorrectly. > According to flip-283, it should be modified to > `execution.batch.speculative.enabled`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30889) Fix the key of configuration SPECULATIVE_ENABLED.
[ https://issues.apache.org/jira/browse/FLINK-30889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-30889: -- Summary: Fix the key of configuration SPECULATIVE_ENABLED. (was: Rename the configuration key SPECULATIVE_ENABLED.) > Fix the key of configuration SPECULATIVE_ENABLED. > - > > Key: FLINK-30889 > URL: https://issues.apache.org/jira/browse/FLINK-30889 > Project: Flink > Issue Type: Bug > Components: Runtime / Configuration >Affects Versions: 1.17.0 >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Critical > Fix For: 1.17.0 > > > In FLINK-30686 we refined the adaptive batch configuration. However, the key > of a configuration item `SPECULATIVE_ENABLED` has been modified incorrectly. > According to flip-283, it should be modified to > `execution.batch.speculative.enabled`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30889) Fix the key of configuration SPECULATIVE_ENABLED.
[ https://issues.apache.org/jira/browse/FLINK-30889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683756#comment-17683756 ] Junrui Li commented on FLINK-30889: --- Thanks [~mapohl] , I'll close this issue. > Fix the key of configuration SPECULATIVE_ENABLED. > - > > Key: FLINK-30889 > URL: https://issues.apache.org/jira/browse/FLINK-30889 > Project: Flink > Issue Type: Bug > Components: Runtime / Configuration >Affects Versions: 1.17.0 >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.0 > > > In FLINK-30686 we refined the adaptive batch configuration. However, the key > of a configuration item `SPECULATIVE_ENABLED` has been modified incorrectly. > According to flip-283, it should be modified to > `execution.batch.speculative.enabled`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30889) Fix the key of configuration SPECULATIVE_ENABLED.
[ https://issues.apache.org/jira/browse/FLINK-30889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li closed FLINK-30889. - Release Note: 1.17.0 Resolution: Fixed > Fix the key of configuration SPECULATIVE_ENABLED. > - > > Key: FLINK-30889 > URL: https://issues.apache.org/jira/browse/FLINK-30889 > Project: Flink > Issue Type: Bug > Components: Runtime / Configuration >Affects Versions: 1.17.0 >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.0 > > > In FLINK-30686 we refined the adaptive batch configuration. However, the key > of a configuration item `SPECULATIVE_ENABLED` has been modified incorrectly. > According to flip-283, it should be modified to > `execution.batch.speculative.enabled`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30889) Fix the key of configuration SPECULATIVE_ENABLED.
[ https://issues.apache.org/jira/browse/FLINK-30889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683784#comment-17683784 ] Junrui Li commented on FLINK-30889: --- [~mapohl] Thank you for reminding me. I accidentally made a mistake.:D > Fix the key of configuration SPECULATIVE_ENABLED. > - > > Key: FLINK-30889 > URL: https://issues.apache.org/jira/browse/FLINK-30889 > Project: Flink > Issue Type: Bug > Components: Runtime / Configuration >Affects Versions: 1.17.0 >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.0 > > > In FLINK-30686 we refined the adaptive batch configuration. However, the key > of a configuration item `SPECULATIVE_ENABLED` has been modified incorrectly. > According to flip-283, it should be modified to > `execution.batch.speculative.enabled`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-26839) SessionDispatcherLeaderProcessTest.recoverJobs_withJobIdRecoveryFailure_failsFatally failed on azures
[
https://issues.apache.org/jira/browse/FLINK-26839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683791#comment-17683791
]
Junrui Li commented on FLINK-26839:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45675&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=24c3384f-1bcb-57b3-224f-51bf973bbee8&l=8444
>
> SessionDispatcherLeaderProcessTest.recoverJobs_withJobIdRecoveryFailure_failsFatally
> failed on azures
> --
>
> Key: FLINK-26839
> URL: https://issues.apache.org/jira/browse/FLINK-26839
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> {code:java}
> Mar 24 04:11:15 [ERROR]
> SessionDispatcherLeaderProcessTest.recoverJobs_withJobIdRecoveryFailure_failsFatally:667->runJobRecoveryFailureTest:677
>
> Mar 24 04:11:15 Expecting
> Mar 24 04:11:15java.util.concurrent.CompletionException:
> org.apache.flink.util.FlinkRuntimeException: Could not retrieve job ids of
> persisted jobs.
> Mar 24 04:11:15 at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> Mar 24 04:11:15 at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> Mar 24 04:11:15 at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:618)
> Mar 24 04:11:15 ...(5 remaining lines not displayed - this can be
> changed with Assertions.setMaxStackTraceElementsDisplayed)]>
> Mar 24 04:11:15 to be completed within 100L Millis.
> Mar 24 04:11:15
> Mar 24 04:11:15 exception caught while trying to get the future result:
> java.util.concurrent.TimeoutException
> Mar 24 04:11:15 at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
> Mar 24 04:11:15 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> Mar 24 04:11:15 at
> org.assertj.core.internal.Futures.assertSucceededWithin(Futures.java:100)
> Mar 24 04:11:15 at
> org.assertj.core.api.AbstractCompletableFutureAssert.internalSucceedsWithin(AbstractCompletableFutureAssert.java:438)
> Mar 24 04:11:15 at
> org.assertj.core.api.AbstractCompletableFutureAssert.succeedsWithin(AbstractCompletableFutureAssert.java:434)
> Mar 24 04:11:15 at
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcessTest.runJobRecoveryFailureTest(SessionDispatcherLeaderProcessTest.java:677)
> Mar 24 04:11:15 at
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcessTest.recoverJobs_withJobIdRecoveryFailure_failsFatally(SessionDispatcherLeaderProcessTest.java:667)
> Mar 24 04:11:15 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Mar 24 04:11:15 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Mar 24 04:11:15 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Mar 24 04:11:15 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 24 04:11:15 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> Mar 24 04:11:15 at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> Mar 24 04:11:15 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> Mar 24 04:11:15 at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> Mar 24 04:11:15 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> Mar 24 04:11:15 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> Mar 24 04:11:15 at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> Mar 24 04:11:15 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> Mar 24 04:11:15 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> Mar 24 04:11:15 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> Mar 24 04:11:15 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> Mar 24 04:11:15 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(Invocati
[jira] [Created] (FLINK-30901) Fix set jobVertex parallelismConfigured don't contain chained sources.
Junrui Li created FLINK-30901: - Summary: Fix set jobVertex parallelismConfigured don't contain chained sources. Key: FLINK-30901 URL: https://issues.apache.org/jira/browse/FLINK-30901 Project: Flink Issue Type: Bug Components: Runtime / Coordination Affects Versions: 1.17.0 Reporter: Junrui Li Fix For: 1.17.0 When create OperatorChainInfo in StreamingJobGenerator, the chained source don't include in the chainedNodes and they will not be added to chainInfo via addNodeToChain. This will cause the jobVertex's parallelismConfigured property to be wrong. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30903) Fix the maxParallelism of DefaultVertexParallelismAndInputInfosDecider fallback to default parallelism
Junrui Li created FLINK-30903: - Summary: Fix the maxParallelism of DefaultVertexParallelismAndInputInfosDecider fallback to default parallelism Key: FLINK-30903 URL: https://issues.apache.org/jira/browse/FLINK-30903 Project: Flink Issue Type: Bug Components: Runtime / Coordination Affects Versions: 1.17.0 Reporter: Junrui Li Fix For: 1.17.0 In FLINK-30684 we mark the vertices which use the default parallelism, and in AdaptiveBatchScheduler we allow users to use parallelism.default as the max parallelism if they don't configure the configuration item "execution.batch.adaptive.auto-parallelism.max-parallelism". This issue will fix the fallback logic. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30903) Fix the maxParallelism of DefaultVertexParallelismAndInputInfosDecider fallback to default parallelism
[ https://issues.apache.org/jira/browse/FLINK-30903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-30903: -- Priority: Critical (was: Major) > Fix the maxParallelism of DefaultVertexParallelismAndInputInfosDecider > fallback to default parallelism > -- > > Key: FLINK-30903 > URL: https://issues.apache.org/jira/browse/FLINK-30903 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.17.0 >Reporter: Junrui Li >Priority: Critical > Fix For: 1.17.0 > > > In FLINK-30684 we mark the vertices which use the default parallelism, and in > AdaptiveBatchScheduler we allow users to use parallelism.default as the max > parallelism if they don't configure the configuration item > "execution.batch.adaptive.auto-parallelism.max-parallelism". This issue will > fix the fallback logic. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30903) The max parallelism of adaptive batch scheduler doesn't fallback to default parallelism
[ https://issues.apache.org/jira/browse/FLINK-30903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-30903: -- Summary: The max parallelism of adaptive batch scheduler doesn't fallback to default parallelism (was: Fix the maxParallelism of DefaultVertexParallelismAndInputInfosDecider fallback to default parallelism) > The max parallelism of adaptive batch scheduler doesn't fallback to default > parallelism > --- > > Key: FLINK-30903 > URL: https://issues.apache.org/jira/browse/FLINK-30903 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.17.0 >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.0 > > > In FLINK-30684 we mark the vertices which use the default parallelism, and in > AdaptiveBatchScheduler we allow users to use parallelism.default as the max > parallelism if they don't configure the configuration item > "execution.batch.adaptive.auto-parallelism.max-parallelism". This issue will > fix the fallback logic. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30901) The jobVertex's parallelismConfigured is incorrect when chaining with source operators
[ https://issues.apache.org/jira/browse/FLINK-30901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-30901: -- Summary: The jobVertex's parallelismConfigured is incorrect when chaining with source operators (was: Fix set jobVertex parallelismConfigured don't contain chained sources.) > The jobVertex's parallelismConfigured is incorrect when chaining with source > operators > -- > > Key: FLINK-30901 > URL: https://issues.apache.org/jira/browse/FLINK-30901 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.17.0 >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > When create OperatorChainInfo in StreamingJobGenerator, the chained source > don't include in the chainedNodes and they will not be added to chainInfo via > addNodeToChain. > This will cause the jobVertex's parallelismConfigured property to be wrong. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30903) The max parallelism used in adaptive batch scheduler doesn't fallbacks to default parallelism
[ https://issues.apache.org/jira/browse/FLINK-30903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-30903: -- Summary: The max parallelism used in adaptive batch scheduler doesn't fallbacks to default parallelism (was: The max parallelism of adaptive batch scheduler doesn't fallback to default parallelism) > The max parallelism used in adaptive batch scheduler doesn't fallbacks to > default parallelism > - > > Key: FLINK-30903 > URL: https://issues.apache.org/jira/browse/FLINK-30903 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.17.0 >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.0 > > > In FLINK-30684 we mark the vertices which use the default parallelism, and in > AdaptiveBatchScheduler we allow users to use parallelism.default as the max > parallelism if they don't configure the configuration item > "execution.batch.adaptive.auto-parallelism.max-parallelism". This issue will > fix the fallback logic. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31055) The dynamic flag of stream graph does not take effect when translating the transformations
[ https://issues.apache.org/jira/browse/FLINK-31055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688357#comment-17688357 ] Junrui Li commented on FLINK-31055: --- [~wanglijie] Thanks for creating this issue. As you said, I think it is a bug. Could you fix it？Thanks! > The dynamic flag of stream graph does not take effect when translating the > transformations > -- > > Key: FLINK-31055 > URL: https://issues.apache.org/jira/browse/FLINK-31055 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Reporter: Lijie Wang >Priority: Major > Fix For: 1.17.0 > > > Currently, the dynamic flag of stream graph is not set when [translate > transformations|https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphGenerator.java#L324]. > However, the dynamic flag will be used > ([here|https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraph.java#L696]) > when translating, we should set the dynamic flag before the translating. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31077) Trigger checkpoint failed but it were shown as COMPLETED by rest API
Junrui Li created FLINK-31077: - Summary: Trigger checkpoint failed but it were shown as COMPLETED by rest API Key: FLINK-31077 URL: https://issues.apache.org/jira/browse/FLINK-31077 Project: Flink Issue Type: Bug Components: Runtime / Checkpointing Affects Versions: 1.16.1, 1.15.3, 1.17.0 Reporter: Junrui Li Fix For: 1.17.0, 1.15.4, 1.16.2 Currently, we can trigger a checkpoint and poll the status of the checkpoint until it is finished by rest according to FLINK-27101. However, even if the checkpoint status returned by rest is completed, it does not mean that the checkpoint is really completed. If an exception occurs after marking the pendingCheckpoint completed([here|https://github.com/apache/flink/blob/bf0ad52cbcb052961c54c94c7013f5ac0110ef8a/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1309]), the checkpoint is not written to the HA service and we can not failover from this checkpoint. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31077) Trigger checkpoint failed but it were shown as COMPLETED by rest API
[ https://issues.apache.org/jira/browse/FLINK-31077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688841#comment-17688841 ] Junrui Li commented on FLINK-31077: --- cc [~gaoyunhaii] > Trigger checkpoint failed but it were shown as COMPLETED by rest API > > > Key: FLINK-31077 > URL: https://issues.apache.org/jira/browse/FLINK-31077 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.17.0, 1.15.3, 1.16.1 >Reporter: Junrui Li >Priority: Major > Fix For: 1.17.0, 1.15.4, 1.16.2 > > > Currently, we can trigger a checkpoint and poll the status of the checkpoint > until it is finished by rest according to FLINK-27101. However, even if the > checkpoint status returned by rest is completed, it does not mean that the > checkpoint is really completed. If an exception occurs after marking the > pendingCheckpoint > completed([here|https://github.com/apache/flink/blob/bf0ad52cbcb052961c54c94c7013f5ac0110ef8a/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1309]), > the checkpoint is not written to the HA service and we can not failover from > this checkpoint. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31077) Trigger checkpoint failed but it were shown as COMPLETED by rest API
[ https://issues.apache.org/jira/browse/FLINK-31077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688860#comment-17688860 ] Junrui Li commented on FLINK-31077: --- [~zhuzh] Sure, I'll fix it. > Trigger checkpoint failed but it were shown as COMPLETED by rest API > > > Key: FLINK-31077 > URL: https://issues.apache.org/jira/browse/FLINK-31077 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.17.0, 1.15.3, 1.16.1 >Reporter: Junrui Li >Priority: Major > Fix For: 1.17.0, 1.15.4, 1.16.2 > > > Currently, we can trigger a checkpoint and poll the status of the checkpoint > until it is finished by rest according to FLINK-27101. However, even if the > checkpoint status returned by rest is completed, it does not mean that the > checkpoint is really completed. If an exception occurs after marking the > pendingCheckpoint > completed([here|https://github.com/apache/flink/blob/bf0ad52cbcb052961c54c94c7013f5ac0110ef8a/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1309]), > the checkpoint is not written to the HA service and we can not failover from > this checkpoint. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31061) Release Testing: Verify FLINK-30376 Introduce a new flink bushy join reorder rule which based on greedy algorithm
[ https://issues.apache.org/jira/browse/FLINK-31061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1767#comment-1767 ] Junrui Li commented on FLINK-31061: --- Thanks [~337361...@qq.com] for creating this issue, I'd like to do this testing work. > Release Testing: Verify FLINK-30376 Introduce a new flink bushy join reorder > rule which based on greedy algorithm > - > > Key: FLINK-31061 > URL: https://issues.apache.org/jira/browse/FLINK-31061 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.17.0 >Reporter: Yunhong Zheng >Priority: Major > Fix For: 1.17.0 > > > This issue aims to verify FLINK-30376: [Introduce a new flink bushy join > reorder rule which based on greedy > algorithm|https://issues.apache.org/jira/browse/FLINK-30376]. > In Flink-1.17, bushy join reorder strategy is the default join reorder > strategy, and this strategy can be disable by setting factor ' > table.optimizer.bushy-join-reorder-threshold' smaller that the table number > need to be reordered. If disabled, the Lopt join reorder strategy, which is > default join reorder strategy in Flink-1.16, will be choosen. > We can verify it in SQL client after we build the flink-dist package. > # Firstly, we need to create several tables (The best case is that these > tables have table and column statistics). > # Secondly, we need to set 'table.optimizer.join-reorder-enabled = true' to > open join reorder. > # Verify bushy join reorder (The default bushy join reorder threshold is 12, > so if the number of table smaller than 12, the join reorder strategy is bushy > join reorder). > # Compare the results of bushy join reorder and Lopt join reorder strategy. > Need to be same. > # If we want to create a bushy join tree after join reorder, we need to add > statistics. Like：'JoinReorderITCaseBase.testBushyTreeJoinReorder'. > If you meet any problems, it's welcome to ping me directly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31052) Release Testing: Verify FLINK-30707 Improve slow task detection
[ https://issues.apache.org/jira/browse/FLINK-31052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688895#comment-17688895 ] Junrui Li commented on FLINK-31052: --- [~zhuzh] Thanks for creating this issue, I'd like to do this testing work. > Release Testing: Verify FLINK-30707 Improve slow task detection > --- > > Key: FLINK-31052 > URL: https://issues.apache.org/jira/browse/FLINK-31052 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Zhu Zhu >Assignee: Junrui Li >Priority: Blocker > Fix For: 1.17.0 > > > This task aims to verify > [FLINK-30707|https://issues.apache.org/jira/browse/FLINK-30707] which > improves the slow task detection. > The slow task detection now takes the input data volume of tasks into > account. Tasks which has a longer execution time but consumes more data may > not be considered as slow. This improvement helps to eliminate the negative > impacts of data skew on slow task detecting. > The documentation of speculative execution can be found > [here|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/speculative_execution/#speculative-execution] > . > One can verify it by creating intended data skew. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31093) NullpointerException when restoring a FlinkSQL job from a savepoint
[
https://issues.apache.org/jira/browse/FLINK-31093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17689666#comment-17689666
]
Junrui Li commented on FLINK-31093:
---
[~mapohl] According to FLINK-29379,
[ExecutionConfig:486|https://github.com/apache/flink/blob/143464d82814e342aa845f3ac976ae2854fc892f/flink-core/src/main/java/org/apache/flink/api/common/ExecutionConfig.java#L486]
was introduced in 1.17. Could you confirm whether the version of sql client
and cluster are both 1.17?
> NullpointerException when restoring a FlinkSQL job from a savepoint
> ---
>
> Key: FLINK-31093
> URL: https://issues.apache.org/jira/browse/FLINK-31093
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Blocker
> Attachments: flink-conf.yaml,
> flink-mapohl-standalonesession-0-aiven-mapohl.log
>
>
> I tried to restore a FlinkSQL job from a savepoint and ran into a
> {{NullPointerException}}:
> {code}
> 2023-02-15 16:38:24,835 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Initializing job 'collect'
> (0263d02536654102f2aa903f843cacd1).
> 2023-02-15 16:38:24,858 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job
> 0263d02536654102f2aa903f843cacd1 reached terminal state FAILED.
> org.apache.flink.runtime.client.JobInitializationException: Could not start
> the JobMaster.
> at
> org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1609)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: java.util.concurrent.CompletionException:
> java.lang.NullPointerException
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
> ... 3 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.api.common.ExecutionConfig.getNumberOfExecutionRetries(ExecutionConfig.java:486)
> at
> org.apache.flink.api.common.ExecutionConfig.getRestartStrategy(ExecutionConfig.java:459)
> at
> org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:99)
> at
> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:119)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:371)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:348)
> at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:123)
> at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:95)
> at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> ... 3 more
> {code}
> The SQL job was submitted through the SQL client:
> {code}
> $ -- table created in Flink 1.16.1
> $ CREATE TABLE MyTable (
> >a bigint,
> >b int not null,
> >c varchar,
> >d timestamp(3)
> > ) with ('connector' = 'datagen', 'rows-per-second' = '1', 'fields.a.kind' =
> > 'sequence', 'fields.a.start' = '0', 'fields.a.end' = '100');
> $ -- SELECT statement ran in Flink 1.16.1 session cluster
> $ SELECT a FROM MyTable WHERE a = 1 or a = 2 or a IS NOT NULL;
> {code}
> The job was stopped with a savepoint from the command line:
> {code}
> $ ./bin/flink stop --type native --savepointPath ../1.16.1-savepoint
> 6029e8e5632a9852c630b1b0e4b62477
> {code}
> A new 1.17-SNAPSHOT (commit: {{21158c06}}) session cluster was started and
> the following SQL code was executed from within the SQL client:
> {code}
> $ SET 'execution.savepoint.path
[jira] [Commented] (FLINK-31061) Release Testing: Verify FLINK-30376 Introduce a new flink bushy join reorder rule which based on greedy algorithm
[ https://issues.apache.org/jira/browse/FLINK-31061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691004#comment-17691004 ] Junrui Li commented on FLINK-31061: --- I have tested it and it looks good to me. I used sql client to do the test by connecting it to Hive and set factor 'table.optimizer.bushy-join-reorder-threshold' = 1 to disable bushy join reorder and set 'table.optimizer.bushy-join-reorder-threshold' = 1000 to enable bushy join reorder. By running testing jobs, I can see that bushy join reorder is taking effect: the join operator plan is different and the job result is also as expected. > Release Testing: Verify FLINK-30376 Introduce a new flink bushy join reorder > rule which based on greedy algorithm > - > > Key: FLINK-31061 > URL: https://issues.apache.org/jira/browse/FLINK-31061 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.17.0 >Reporter: Yunhong Zheng >Assignee: Junrui Li >Priority: Major > Fix For: 1.17.0 > > > This issue aims to verify FLINK-30376: [Introduce a new flink bushy join > reorder rule which based on greedy > algorithm|https://issues.apache.org/jira/browse/FLINK-30376]. > In Flink-1.17, bushy join reorder strategy is the default join reorder > strategy, and this strategy can be disable by setting factor ' > table.optimizer.bushy-join-reorder-threshold' smaller that the table number > need to be reordered. If disabled, the Lopt join reorder strategy, which is > default join reorder strategy in Flink-1.16, will be choosen. > We can verify it in SQL client after we build the flink-dist package. > # Firstly, we need to create several tables (The best case is that these > tables have table and column statistics). > # Secondly, we need to set 'table.optimizer.join-reorder-enabled = true' to > open join reorder. > # Verify bushy join reorder (The default bushy join reorder threshold is 12, > so if the number of table smaller than 12, the join reorder strategy is bushy > join reorder). > # Compare the results of bushy join reorder and Lopt join reorder strategy. > Need to be same. > # If we want to create a bushy join tree after join reorder, we need to add > statistics. Like：'JoinReorderITCaseBase.testBushyTreeJoinReorder'. > If you meet any problems, it's welcome to ping me directly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31052) Release Testing: Verify FLINK-30707 Improve slow task detection
[ https://issues.apache.org/jira/browse/FLINK-31052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691110#comment-17691110 ] Junrui Li commented on FLINK-31052: --- I have tested it and it looks good to me. I created two simple batch jobs with only source and sink operator connected by hash edges. Job 1 has data skew, while Job 2 has no data skew but has a taskManager that processes data very slowly. And I set the `execution.batch.speculative.enabled` = `true` and `slow-task-detector.execution-time.baseline-lower-bound` = `0s` and `slow-task-detector.execution-time.baseline-ratio` = `0.1` to enable speculative scheduler and slow task detection more qucikly. As expected job 1 did not detect slow tasks, while job 2 did. > Release Testing: Verify FLINK-30707 Improve slow task detection > --- > > Key: FLINK-31052 > URL: https://issues.apache.org/jira/browse/FLINK-31052 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Zhu Zhu >Assignee: Junrui Li >Priority: Blocker > Fix For: 1.17.0 > > > This task aims to verify > [FLINK-30707|https://issues.apache.org/jira/browse/FLINK-30707] which > improves the slow task detection. > The slow task detection now takes the input data volume of tasks into > account. Tasks which has a longer execution time but consumes more data may > not be considered as slow. This improvement helps to eliminate the negative > impacts of data skew on slow task detecting. > The documentation of speculative execution can be found > [here|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/speculative_execution/#speculative-execution] > . > One can verify it by creating intended data skew. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31071) Release Testing: Verify FLIP-283 Use adaptive batch scheduler as default scheduler for batch jobs
[
https://issues.apache.org/jira/browse/FLINK-31071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691141#comment-17691141
]
Junrui Li commented on FLINK-31071:
---
Thanks [~xiasun] and [~wanglijie] .
This situation is by design, there are four ways for setting parallelism
([here|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution/parallel/]).
Except for the operator level, the others are to set the global default
parallelism, and the priority is that system < cli < environment < operator.
> Release Testing: Verify FLIP-283 Use adaptive batch scheduler as default
> scheduler for batch jobs
> -
>
> Key: FLINK-31071
> URL: https://issues.apache.org/jira/browse/FLINK-31071
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Lijie Wang
>Assignee: xingbe
>Priority: Blocker
> Fix For: 1.17.0
>
>
> This task aims to verify FLIP-283 Use adaptive batch scheduler as default
> scheduler for batch jobs.
> The documentation of adaptive batch scheduler can be found
> [here|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#adaptive-batch-scheduler]
> .
> Things to verify:
> 1. Verify the adaptive batch scheduler is the default scheduler of batch
> jobs. By default, Flink will automatically decide parallelism for operators
> of batch jobs if the scheduler is not specified.
> 2. Verify the configuration
> options({{{}execution.batch.adaptive.auto-parallelism.xxx{}}}) take effect.
> Besides, it is also necessary to verify that the default parallelism set via
> {{parallelism.default}} or {{StreamExecutionEnvironment#setParallelism()}}
> will be used as upper bound of allowed parallelism if the
> execution.batch.adaptive.auto-parallelism.max-parallelism is not configured.
> 3. Verify the final data produced are correct.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31144) Slow scheduling on large-scale batch jobs
[ https://issues.apache.org/jira/browse/FLINK-31144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17696030#comment-17696030 ] Junrui Li commented on FLINK-31144: --- Hi, [~zhuzh]. I'd like to fix this issue, could you help to assign this ticket to me? > Slow scheduling on large-scale batch jobs > -- > > Key: FLINK-31144 > URL: https://issues.apache.org/jira/browse/FLINK-31144 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Reporter: Julien Tournay >Priority: Major > Attachments: flink-1.17-snapshot-1676473798013.nps, > image-2023-02-21-10-29-49-388.png > > > When executing a complex job graph at high parallelism > `DefaultPreferredLocationsRetriever.getPreferredLocationsBasedOnInputs` can > get slow and cause long pauses where the JobManager becomes unresponsive and > all the taskmanagers just wait. I've attached a VisualVM snapshot to > illustrate the problem.[^flink-1.17-snapshot-1676473798013.nps] > At Spotify we have complex jobs where this issue can cause batch "pause" of > 40+ minutes and make the overall execution 30% slower or more. > More importantly this prevent us from running said jobs on larger cluster as > adding resources to the cluster worsen the issue. > We have successfully tested a modified Flink version where > `DefaultPreferredLocationsRetriever.getPreferredLocationsBasedOnInputs` was > completely commented and simply returns an empty collection and confirmed it > solves the issue. > In the same spirit as a recent change > ([https://github.com/apache/flink/blob/43f419d0eccba86ecc8040fa6f521148f1e358ff/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultPreferredLocationsRetriever.java#L98-L102)] > there could be a mechanism in place to detect when Flink run into this > specific issue and just skip the call to `getInputLocationFutures` > [https://github.com/apache/flink/blob/43f419d0eccba86ecc8040fa6f521148f1e358ff/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultPreferredLocationsRetriever.java#L105-L108.] > I'm not familiar enough with the internals of Flink to propose a more > advanced fix, however it seems like a configurable threshold on the number of > consumer vertices above which the preferred location is not computed would > do. If this solution is good enough, I'd be happy to submit a PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31401) testTransformationSetParallelism fails on 10 core machines
[
https://issues.apache.org/jira/browse/FLINK-31401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699072#comment-17699072
]
Junrui Li commented on FLINK-31401:
---
[~mbalassi] Thanks so much for your volunteering and contribution!:D As you
said, this is a bug: testTransformationSetParallelism implies that the map and
source should have different parallelism so that they will not be chained
together, but when the default parallelism is 10, it will break the rule.
> testTransformationSetParallelism fails on 10 core machines
> --
>
> Key: FLINK-31401
> URL: https://issues.apache.org/jira/browse/FLINK-31401
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Tests
>Affects Versions: 1.17.0
>Reporter: Márton Balassi
>Assignee: Márton Balassi
>Priority: Minor
> Labels: pull-request-available
>
> StreamingJobGraphGenerator#testTransformationSetParallelism fails if it is
> run in an environment where the default parallelism is 10:
> {noformat}
> org.opentest4j.AssertionFailedError:
> expected: 3
> but was: 2
> Expected :3
> Actual :2
> {noformat}
> The fix is trivial, we need to make an implicit assumption in the test about
> paralellisms explicit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31425) Support submitting a job with streamgraph
[ https://issues.apache.org/jira/browse/FLINK-31425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699925#comment-17699925 ] Junrui Li commented on FLINK-31425: --- [~tophei] Thanks for creating this issue. IIUC, according to the description, I think this issue can be divided into two parts: 1. Submit and run jobs by submitting a streamGraph, which is very meaningful. Submitting and running jobs through streamGraph can expand the dynamic adjustment capability of flink runtime. This is a relatively large and very meaningful change, and we are currently doing related research. 2. Make StreamGraph public so that users can get streamGraph. I'm not sure if this part has enough meaning to expose an internal interface to users, which may bring additional burden. > Support submitting a job with streamgraph > -- > > Key: FLINK-31425 > URL: https://issues.apache.org/jira/browse/FLINK-31425 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Reporter: Jeff >Priority: Major > > Currently, we have rest api to submit a job via jobgraph, which is aligned to > the way of flink cli running the entry class locally and submit the compiled > binary to remote cluster for execution. > This is convenient in its own right. However it also seems to bring in some > confusion and 'blackbox' feeling in that the payload of rest api is a binary > object and thus not self-descriptive and it's relative a low-level > presentation of the job executions whose interface is more likely to change > as version evolves. > Do you think it make more sense to build an api that accepts streamgraph as > input which may be presented with a json(just like visualizer did for an > execution plan visualization) plus additional runtime related configs and > resources? This may make the rest interface more descriptive. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31437) 'lookup.cache.caching-missing-key' change should be configured as 'lookup.partial-cache.caching-missing-key'
[ https://issues.apache.org/jira/browse/FLINK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699960#comment-17699960 ] Junrui Li commented on FLINK-31437: --- [~gaara] Thanks for creating this issue, and *[xuzhiwen1255|https://github.com/xuzhiwen1255]* has proposed a hot-fix pr to fix this bug: https://github.com/apache/flink/pull/22167 > 'lookup.cache.caching-missing-key' change should be configured as > 'lookup.partial-cache.caching-missing-key' > > > Key: FLINK-31437 > URL: https://issues.apache.org/jira/browse/FLINK-31437 > Project: Flink > Issue Type: Bug >Reporter: Zhimin Geng >Priority: Blocker > Attachments: image-2023-03-14-05-45-06-230.png, > image-2023-03-14-05-45-44-616.png > > > 'lookup.cache.caching-missing-key' change should be configured as > 'lookup.partial-cache.caching-missing-key'. > An error occurred when I configured a dimension table. > The configuration given by the official website is not available. > !image-2023-03-14-05-45-06-230.png! > !image-2023-03-14-05-45-44-616.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31437) 'lookup.cache.caching-missing-key' change should be configured as 'lookup.partial-cache.caching-missing-key'
[ https://issues.apache.org/jira/browse/FLINK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699967#comment-17699967 ] Junrui Li commented on FLINK-31437: --- [~gaara] flink-connector-jdbc connector has been removed in release 1.17: https://issues.apache.org/jira/browse/FLINK-30465 > 'lookup.cache.caching-missing-key' change should be configured as > 'lookup.partial-cache.caching-missing-key' > > > Key: FLINK-31437 > URL: https://issues.apache.org/jira/browse/FLINK-31437 > Project: Flink > Issue Type: Bug >Reporter: Zhimin Geng >Priority: Blocker > Attachments: image-2023-03-14-05-45-06-230.png, > image-2023-03-14-05-45-44-616.png > > > 'lookup.cache.caching-missing-key' change should be configured as > 'lookup.partial-cache.caching-missing-key'. > An error occurred when I configured a dimension table. > The configuration given by the official website is not available. > !image-2023-03-14-05-45-06-230.png! > !image-2023-03-14-05-45-44-616.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31437) 'lookup.cache.caching-missing-key' change should be configured as 'lookup.partial-cache.caching-missing-key'
[ https://issues.apache.org/jira/browse/FLINK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699967#comment-17699967 ] Junrui Li edited comment on FLINK-31437 at 3/14/23 8:26 AM: [~gaara] -flink-connector-jdbc connector has been removed in release 1.17:- -https://issues.apache.org/jira/browse/FLINK-30465- Sorry, I made a mistake, flink-connector-jdbc was moved to a separate repository, so you can fix it in flink version 1.16.2, and also fix in [https://github.com/apache/flink-connector.] cc [~renqs] was (Author: JIRAUSER293716): [~gaara] flink-connector-jdbc connector has been removed in release 1.17: https://issues.apache.org/jira/browse/FLINK-30465 > 'lookup.cache.caching-missing-key' change should be configured as > 'lookup.partial-cache.caching-missing-key' > > > Key: FLINK-31437 > URL: https://issues.apache.org/jira/browse/FLINK-31437 > Project: Flink > Issue Type: Bug >Reporter: Zhimin Geng >Priority: Major > Attachments: image-2023-03-14-05-45-06-230.png, > image-2023-03-14-05-45-44-616.png > > > 'lookup.cache.caching-missing-key' change should be configured as > 'lookup.partial-cache.caching-missing-key'. > An error occurred when I configured a dimension table. > The configuration given by the official website is not available. > !image-2023-03-14-05-45-06-230.png! > !image-2023-03-14-05-45-44-616.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31437) 'lookup.cache.caching-missing-key' change should be configured as 'lookup.partial-cache.caching-missing-key'
[ https://issues.apache.org/jira/browse/FLINK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699967#comment-17699967 ] Junrui Li edited comment on FLINK-31437 at 3/14/23 8:30 AM: [~gaara] -flink-connector-jdbc connector has been removed in release 1.17:- -https://issues.apache.org/jira/browse/FLINK-30465- Sorry, I made a mistake, flink-connector-jdbc was moved to a separate repository, so you can fix it in flink version 1.16.2, and also fix in https://github.com/apache/flink-connector-jdbc. cc [~renqs] was (Author: JIRAUSER293716): [~gaara] -flink-connector-jdbc connector has been removed in release 1.17:- -https://issues.apache.org/jira/browse/FLINK-30465- Sorry, I made a mistake, flink-connector-jdbc was moved to a separate repository, so you can fix it in flink version 1.16.2, and also fix in [https://github.com/apache/flink-connector.] cc [~renqs] > 'lookup.cache.caching-missing-key' change should be configured as > 'lookup.partial-cache.caching-missing-key' > > > Key: FLINK-31437 > URL: https://issues.apache.org/jira/browse/FLINK-31437 > Project: Flink > Issue Type: Bug >Reporter: Zhimin Geng >Priority: Major > Attachments: image-2023-03-14-05-45-06-230.png, > image-2023-03-14-05-45-44-616.png > > > 'lookup.cache.caching-missing-key' change should be configured as > 'lookup.partial-cache.caching-missing-key'. > An error occurred when I configured a dimension table. > The configuration given by the official website is not available. > !image-2023-03-14-05-45-06-230.png! > !image-2023-03-14-05-45-44-616.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31437) 'lookup.cache.caching-missing-key' change should be configured as 'lookup.partial-cache.caching-missing-key'
[ https://issues.apache.org/jira/browse/FLINK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700017#comment-17700017 ] Junrui Li commented on FLINK-31437: --- [~gaara] Yes, you're right. I made a mistake and I've updated the comments. You can fix flink-connector-jdbc docs in flink version 1.16.2, and also fix in [https://github.com/apache/flink-connector-jdbc]. :D > 'lookup.cache.caching-missing-key' change should be configured as > 'lookup.partial-cache.caching-missing-key' > > > Key: FLINK-31437 > URL: https://issues.apache.org/jira/browse/FLINK-31437 > Project: Flink > Issue Type: Bug >Reporter: Zhimin Geng >Priority: Major > Attachments: image-2023-03-14-05-45-06-230.png, > image-2023-03-14-05-45-44-616.png > > > 'lookup.cache.caching-missing-key' change should be configured as > 'lookup.partial-cache.caching-missing-key'. > An error occurred when I configured a dimension table. > The configuration given by the official website is not available. > !image-2023-03-14-05-45-06-230.png! > !image-2023-03-14-05-45-44-616.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31437) 'lookup.cache.caching-missing-key' change should be configured as 'lookup.partial-cache.caching-missing-key'
[ https://issues.apache.org/jira/browse/FLINK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700025#comment-17700025 ] Junrui Li commented on FLINK-31437: --- [~zhuzh] Could I take this tickets? Thx! > 'lookup.cache.caching-missing-key' change should be configured as > 'lookup.partial-cache.caching-missing-key' > > > Key: FLINK-31437 > URL: https://issues.apache.org/jira/browse/FLINK-31437 > Project: Flink > Issue Type: Bug >Reporter: Zhimin Geng >Priority: Major > Attachments: image-2023-03-14-05-45-06-230.png, > image-2023-03-14-05-45-44-616.png > > > 'lookup.cache.caching-missing-key' change should be configured as > 'lookup.partial-cache.caching-missing-key'. > An error occurred when I configured a dimension table. > The configuration given by the official website is not available. > !image-2023-03-14-05-45-06-230.png! > !image-2023-03-14-05-45-44-616.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31457) Support waiting for required resources in DefaultScheduler during job restart
[ https://issues.apache.org/jira/browse/FLINK-31457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700503#comment-17700503 ] Junrui Li commented on FLINK-31457: --- [~a.pilipenko] I'm not sure what is the scenario where `NoResourceAvailableException` will be reported after job restart? Can you describe it in detail? IIUC, if it is a session cluster, the slot may be occupied by other jobs after slot idle timeout. Maybe you can increase the slot.idle.timeout. In addition, the adaptive scheduler has a mechanism to wait for resources because it can dynamically adjust the parallelism, and run jobs with a small parallelism when resources are insufficient, while the default scheduler does not have such a capability, so when resources are insufficient, it will report `NoResourceAvailableException`. If you want to run jobs even when resources are insufficient, you can use the adaptive scheduler in stream job. > Support waiting for required resources in DefaultScheduler during job restart > - > > Key: FLINK-31457 > URL: https://issues.apache.org/jira/browse/FLINK-31457 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.15.3 >Reporter: Aleksandr Pilipenko >Priority: Major > > Currently Flink support [waiting for required resources to become > available|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#jobmanager-adaptive-scheduler-resource-stabilization-timeout] > during job restart only while using adaptive scheduler. > On the other hand, if cluster is using default scheduler and there is not > enough slots available - restart attempts will fail with > `NoResourceAvailableException` until resource requirements are satisfied. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-21126) Reconsider FLINK_PROPERTIES
[
https://issues.apache.org/jira/browse/FLINK-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810859#comment-17810859
]
Junrui Li commented on FLINK-21126:
---
After reviewing the considerations for deprecating the FLINK_PROPERTIES
environment variable in Docker scripts for Flink 2.0, we have decided to retain
this feature for the time being.
The upcoming code freeze for Flink-1.19 on January 25, 2024, does not leave us
with sufficient time to undergo the required community discussions and voting
process, considering that FLINK_PROPERTIES is part of the public API.
Moreover, removing support for FLINK_PROPERTIES would not significantly ease
the user experience or reduce code complexity. In fact, it could introduce
unnecessary migration costs for our users.
Given these points, we plan not to include this change in the scope of
Flink-2.0.
> Reconsider FLINK_PROPERTIES
> ---
>
> Key: FLINK-21126
> URL: https://issues.apache.org/jira/browse/FLINK-21126
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Scripts
>Reporter: Chesnay Schepler
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor
> Fix For: 2.0.0
>
>
> The docker scripts support a {{FLINK_PROPERTIES}} environment variable that
> contains a multi-line string with config options, that is piped into the
> configuration.
> This variable is somewhat redundant, because docker users can also specify a
> list of dynamic properties when starting a container, and Kubernetes users
> are advised to use config maps instead.
> FLIP-161 might also re-introduce new ways to configure Flink via environment
> variables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33577) Make "config.yaml" as the default Flink configuration file
[ https://issues.apache.org/jira/browse/FLINK-33577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-33577: -- Summary: Make "config.yaml" as the default Flink configuration file (was: Make "conf.yaml" as the default Flink configuration file) > Make "config.yaml" as the default Flink configuration file > -- > > Key: FLINK-33577 > URL: https://issues.apache.org/jira/browse/FLINK-33577 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Configuration >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > This update ensures that the flink-dist package in FLINK will include the new > configuration file "conf.yaml" by default. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33577) Make "config.yaml" as the default Flink configuration file
[ https://issues.apache.org/jira/browse/FLINK-33577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-33577: -- Description: This update ensures that the flink-dist package in FLINK will include the new configuration file "config.yaml" by default. (was: This update ensures that the flink-dist package in FLINK will include the new configuration file "conf.yaml" by default.) > Make "config.yaml" as the default Flink configuration file > -- > > Key: FLINK-33577 > URL: https://issues.apache.org/jira/browse/FLINK-33577 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Configuration >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > This update ensures that the flink-dist package in FLINK will include the new > configuration file "config.yaml" by default. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34247) Document FLIP-366: Support standard YAML for FLINK configuration
Junrui Li created FLINK-34247: - Summary: Document FLIP-366: Support standard YAML for FLINK configuration Key: FLINK-34247 URL: https://issues.apache.org/jira/browse/FLINK-34247 Project: Flink Issue Type: Sub-task Components: Documentation Reporter: Junrui Li Fix For: 1.19.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33297) FLIP-366: Support standard YAML for FLINK configuration
[ https://issues.apache.org/jira/browse/FLINK-33297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17811331#comment-17811331 ] Junrui Li commented on FLINK-33297: --- [~lincoln.86xy] Not yet, sorry for the misleading information. I missed creating the JIRA for the document this FLIP. have now added that. Once the documentation is completed, I will close this JIRA. > FLIP-366: Support standard YAML for FLINK configuration > --- > > Key: FLINK-33297 > URL: https://issues.apache.org/jira/browse/FLINK-33297 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: 2.0-related, pull-request-available > Fix For: 1.19.0 > > > Support standard YAML for FLINK configuration -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34245) CassandraSinkTest.test_cassandra_sink fails under JDK17 and JDK21 due to InaccessibleObjectException
[
https://issues.apache.org/jira/browse/FLINK-34245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17811562#comment-17811562
]
Junrui Li commented on FLINK-34245:
---
[~Sergey Nuyanzin] Thanks for kindly reminder, it's a bug that handle standard
yaml config file in Pyflink gateway, I'll prepare a fix as soon as possible.
> CassandraSinkTest.test_cassandra_sink fails under JDK17 and JDK21 due to
> InaccessibleObjectException
>
>
> Key: FLINK-34245
> URL: https://issues.apache.org/jira/browse/FLINK-34245
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Connectors / Cassandra
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Priority: Blocker
> Labels: test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56942&view=logs&j=b53e1644-5cb4-5a3b-5d48-f523f39bcf06&t=b68c9f5c-04c9-5c75-3862-a3a27aabbce3]
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56942&view=logs&j=60960eae-6f09-579e-371e-29814bdd1adc&t=7a70c083-6a74-5348-5106-30a76c29d8fa&l=63680]
> {code:java}
> Jan 26 01:29:27 E py4j.protocol.Py4JJavaError: An error
> occurred while calling
> z:org.apache.flink.python.util.PythonConfigUtil.configPythonOperator.
> Jan 26 01:29:27 E :
> java.lang.reflect.InaccessibleObjectException: Unable to make field final
> java.util.Map java.util.Collections$UnmodifiableMap.m accessible: module
> java.base does not "opens java.util" to unnamed module @17695df3
> Jan 26 01:29:27 E at
> java.base/java.lang.reflect.AccessibleObject.throwInaccessibleObjectException(AccessibleObject.java:391)
> Jan 26 01:29:27 E at
> java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:367)
> Jan 26 01:29:27 E at
> java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:315)
> Jan 26 01:29:27 E at
> java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:183)
> Jan 26 01:29:27 E at
> java.base/java.lang.reflect.Field.setAccessible(Field.java:177)
> Jan 26 01:29:27 E at
> org.apache.flink.python.util.PythonConfigUtil.registerPythonBroadcastTransformationTranslator(PythonConfigUtil.java:357)
> Jan 26 01:29:27 E at
> org.apache.flink.python.util.PythonConfigUtil.configPythonOperator(PythonConfigUtil.java:101)
> Jan 26 01:29:27 E at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> Jan 26 01:29:27 E at
> java.base/java.lang.reflect.Method.invoke(Method.java:580)
> Jan 26 01:29:27 E at
> org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> Jan 26 01:29:27 E at
> org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
> Jan 26 01:29:27 E at
> org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
> Jan 26 01:29:27 E at
> org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> Jan 26 01:29:27 E at
> org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
> Jan 26 01:29:27 E at
> org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
> Jan 26 01:29:27 E at
> java.base/java.lang.Thread.run(Thread.java:1583) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34257) Update Flink YAML Parser to Support YAML 1.2 Specification
Junrui Li created FLINK-34257: - Summary: Update Flink YAML Parser to Support YAML 1.2 Specification Key: FLINK-34257 URL: https://issues.apache.org/jira/browse/FLINK-34257 Project: Flink Issue Type: Bug Components: API / Core Reporter: Junrui Li Fix For: 1.19.0 FLINK-33297 and FLINK-33577 added snakeyaml and pyyaml dependencies to support a standard YAML parser. However, these parsers support the YAML 1.1 specification, not the YAML 1.2 specification. Therefore, we need to update these dependencies that support YAML 1.2. The updated dependencies are as follows: 1. For Java: change from snakeyaml to snakeyaml-engine 2. For Python: change from pyyaml to ruamel.yaml -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34257) Update Flink YAML Parser to Support YAML 1.2 Specification
[ https://issues.apache.org/jira/browse/FLINK-34257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-34257: -- Description: FLINK-33364 and FLINK-33577 added snakeyaml and pyyaml dependencies to support a standard YAML parser. However, these parsers support the YAML 1.1 specification, not the YAML 1.2 specification. Therefore, we need to update these dependencies that support YAML 1.2. The updated dependencies are as follows: 1. For Java: change from snakeyaml to snakeyaml-engine 2. For Python: change from pyyaml to ruamel.yaml was: FLINK-33297 and FLINK-33577 added snakeyaml and pyyaml dependencies to support a standard YAML parser. However, these parsers support the YAML 1.1 specification, not the YAML 1.2 specification. Therefore, we need to update these dependencies that support YAML 1.2. The updated dependencies are as follows: 1. For Java: change from snakeyaml to snakeyaml-engine 2. For Python: change from pyyaml to ruamel.yaml > Update Flink YAML Parser to Support YAML 1.2 Specification > --- > > Key: FLINK-34257 > URL: https://issues.apache.org/jira/browse/FLINK-34257 > Project: Flink > Issue Type: Bug > Components: API / Core >Reporter: Junrui Li >Priority: Major > Fix For: 1.19.0 > > > FLINK-33364 and FLINK-33577 added snakeyaml and pyyaml dependencies to > support a standard YAML parser. However, these parsers support the YAML 1.1 > specification, not the YAML 1.2 specification. Therefore, we need to update > these dependencies that support YAML 1.2. > The updated dependencies are as follows: > 1. For Java: change from snakeyaml to snakeyaml-engine > 2. For Python: change from pyyaml to ruamel.yaml -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34257) Update Flink YAML Parser to Support YAML 1.2 Specification
[ https://issues.apache.org/jira/browse/FLINK-34257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-34257: -- Description: FLINK-33364 and FLINK-33577 added snakeyaml and pyyaml dependencies to support a standard YAML parser. However, these parsers support the YAML 1.1 specification rather than the YAML 1.2 specification, which is the version referenced by [FLINK official website|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#configuration]. Therefore, we need to update these dependencies that support YAML 1.2. The updated dependencies are as follows: 1. For Java: change from snakeyaml to snakeyaml-engine 2. For Python: change from pyyaml to ruamel.yaml was: FLINK-33364 and FLINK-33577 added snakeyaml and pyyaml dependencies to support a standard YAML parser. However, these parsers support the YAML 1.1 specification, not the YAML 1.2 specification. Therefore, we need to update these dependencies that support YAML 1.2. The updated dependencies are as follows: 1. For Java: change from snakeyaml to snakeyaml-engine 2. For Python: change from pyyaml to ruamel.yaml > Update Flink YAML Parser to Support YAML 1.2 Specification > --- > > Key: FLINK-34257 > URL: https://issues.apache.org/jira/browse/FLINK-34257 > Project: Flink > Issue Type: Bug > Components: API / Core, Runtime / Configuration >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > FLINK-33364 and FLINK-33577 added snakeyaml and pyyaml dependencies to > support a standard YAML parser. However, these parsers support the YAML 1.1 > specification rather than the YAML 1.2 specification, which is the version > referenced by [FLINK official > website|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#configuration]. > Therefore, we need to update these dependencies that support YAML 1.2. > The updated dependencies are as follows: > 1. For Java: change from snakeyaml to snakeyaml-engine > 2. For Python: change from pyyaml to ruamel.yaml -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33581) FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects
[ https://issues.apache.org/jira/browse/FLINK-33581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-33581: -- Release Note: The non-ConfigOption objects in the StreamExecutionEnvironment, CheckpointConfig, and ExecutionConfig and their corresponding getter/setter interfaces is now be deprecated in FLINK-1.19. And these objects and methods is planned to be removed in Flink-2.0. Detailed information regarding the deprecations is as follows: 1. RestartStrategy: Class: org.apache.flink.api.common.restartstrategy.RestartStrategies org.apache.flink.api.common.restartstrategy.RestartStrategies.RestartStrategyConfiguration org.apache.flink.api.common.restartstrategy.RestartStrategies.FixedDelayRestartStrategyConfiguration org.apache.flink.api.common.restartstrategy.RestartStrategies.ExponentialDelayRestartStrategyConfiguration org.apache.flink.api.common.restartstrategy.RestartStrategies.FailureRateRestartStrategyConfiguration org.apache.flink.api.common.restartstrategy.RestartStrategies.FallbackRestartStrategyConfiguration Method: org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#setRestartStrategy(RestartStrategies.RestartStrategyConfiguration restartStrategyConfiguration) org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#getRestartStrategy() org.apache.flink.api.common.ExecutionConfig#getRestartStrategy() org.apache.flink.api.common.ExecutionConfig#setRestartStrategy(RestartStrategies.RestartStrategyConfiguration restartStrategyConfiguration) pyflink.common.execution_config.ExecutionConfig.set_restart_strategy(self, restart_strategy_configuration: RestartStrategyConfiguration) pyflink.datastream.stream_execution_environment.StreamExecutionEnvironment.set_restart_strategy(self, restart_strategy_configuration: RestartStrategyConfiguration) Field: org.apache.flink.api.common.ExecutionConfig#restartStrategyConfiguration Suggested alternative: Users can configure the RestartStrategyOptions related ConfigOptions, such as "restart-strategy.type", in the configuration, instead of passing a RestartStrategyConfiguration object. 2.CheckpointStorage: Method: org.apache.flink.streaming.api.environment.CheckpointConfig#setCheckpointStorage(CheckpointStorage storage) org.apache.flink.streaming.api.environment.CheckpointConfig#setCheckpointStorage(String checkpointDirectory) org.apache.flink.streaming.api.environment.CheckpointConfig#setCheckpointStorage(URI checkpointDirectory) org.apache.flink.streaming.api.environment.CheckpointConfig#setCheckpointStorage(Path checkpointDirectory) org.apache.flink.streaming.api.environment.CheckpointConfig#getCheckpointStorage() pyflink.datastream.checkpoint_config.CheckpointConfig.set_checkpoint_storage(self, storage: CheckpointStorage) pyflink.datastream.checkpoint_config.CheckpointConfig.set_checkpoint_storage_dir(self, checkpoint_path: str) pyflink.datastream.checkpoint_config.CheckpointConfig.get_checkpoint_storage(self) Suggested alternative: Users can configure "state.checkpoint-storage" in the configuration as the fully qualified name of the checkpoint storage or use some FLINK-provided checkpoint storage shortcut names such as "jobmanager" and "filesystem", and provide the necessary configuration options for building that storage, instead of passing a CheckpointStorage object. 3.StateBackend: Method: org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#setStateBackend(StateBackend backend) org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#getStateBackend() pyflink.datastream.stream_execution_environment.StreamExecutionEnvironment.set_state_backend(self, state_backend: StateBackend) Field: org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#defaultStateBackend Suggested alternative: Users can configure "state.backend.type" in the configuration as the fully qualified name of the state backend or use some FLINK-provided state backend shortcut names such as "hashmap" and "rocksdb", and provide the necessary configuration options for building that StateBackend, instead of passing a StateBackend object. More details can be found at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992. > FLIP-381: Deprecate configuration getters/setters that return/set complex > Java objects > -- > > Key: FLINK-33581 > URL: https://issues.apache.org/jira/browse/FLINK-33581 > Project: Flink > Issue Type: Technical Debt > Components: API / DataStream >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > Attachments: image-2023-11-30-17-59-42-650.png > > > Deprecate the non-ConfigOption objects i
[jira] [Commented] (FLINK-33581) FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects
[ https://issues.apache.org/jira/browse/FLINK-33581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812525#comment-17812525 ] Junrui Li commented on FLINK-33581: --- Thanks [~martijnvisser] for the kind reminder. I've updated the release notes > FLIP-381: Deprecate configuration getters/setters that return/set complex > Java objects > -- > > Key: FLINK-33581 > URL: https://issues.apache.org/jira/browse/FLINK-33581 > Project: Flink > Issue Type: Technical Debt > Components: API / DataStream >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > Attachments: image-2023-11-30-17-59-42-650.png > > > Deprecate the non-ConfigOption objects in the StreamExecutionEnvironment, > CheckpointConfig, and ExecutionConfig, and ultimately removing them in FLINK > 2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33581) FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects
[ https://issues.apache.org/jira/browse/FLINK-33581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-33581: -- Release Note: The non-ConfigOption objects in the StreamExecutionEnvironment, CheckpointConfig, and ExecutionConfig and their corresponding getter/setter interfaces is now be deprecated in FLINK-1.19. And these objects and methods is planned to be removed in Flink-2.0. Detailed information regarding the deprecations is as follows: 1. RestartStrategy: Class: org.apache.flink.api.common.restartstrategy.RestartStrategies org.apache.flink.api.common.restartstrategy.RestartStrategies.RestartStrategyConfiguration org.apache.flink.api.common.restartstrategy.RestartStrategies.FixedDelayRestartStrategyConfiguration org.apache.flink.api.common.restartstrategy.RestartStrategies.ExponentialDelayRestartStrategyConfiguration org.apache.flink.api.common.restartstrategy.RestartStrategies.FailureRateRestartStrategyConfiguration org.apache.flink.api.common.restartstrategy.RestartStrategies.FallbackRestartStrategyConfiguration Method: org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#setRestartStrategy(RestartStrategies.RestartStrategyConfiguration restartStrategyConfiguration) org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#getRestartStrategy() org.apache.flink.api.common.ExecutionConfig#getRestartStrategy() org.apache.flink.api.common.ExecutionConfig#setRestartStrategy(RestartStrategies.RestartStrategyConfiguration restartStrategyConfiguration) pyflink.common.execution_config.ExecutionConfig#set_restart_strategy(self, restart_strategy_configuration: RestartStrategyConfiguration) pyflink.datastream.stream_execution_environment.StreamExecutionEnvironment#set_restart_strategy(self, restart_strategy_configuration: RestartStrategyConfiguration) Field: org.apache.flink.api.common.ExecutionConfig#restartStrategyConfiguration Suggested alternative: Users can configure the RestartStrategyOptions related ConfigOptions, such as "restart-strategy.type", in the configuration, instead of passing a RestartStrategyConfiguration object. 2.CheckpointStorage: Method: org.apache.flink.streaming.api.environment.CheckpointConfig#setCheckpointStorage(CheckpointStorage storage) org.apache.flink.streaming.api.environment.CheckpointConfig#setCheckpointStorage(String checkpointDirectory) org.apache.flink.streaming.api.environment.CheckpointConfig#setCheckpointStorage(URI checkpointDirectory) org.apache.flink.streaming.api.environment.CheckpointConfig#setCheckpointStorage(Path checkpointDirectory) org.apache.flink.streaming.api.environment.CheckpointConfig#getCheckpointStorage() pyflink.datastream.checkpoint_config.CheckpointConfig#set_checkpoint_storage(self, storage: CheckpointStorage) pyflink.datastream.checkpoint_config.CheckpointConfig#set_checkpoint_storage_dir(self, checkpoint_path: str) pyflink.datastream.checkpoint_config.CheckpointConfig#get_checkpoint_storage(self) Suggested alternative: Users can configure "state.checkpoint-storage" in the configuration as the fully qualified name of the checkpoint storage or use some FLINK-provided checkpoint storage shortcut names such as "jobmanager" and "filesystem", and provide the necessary configuration options for building that storage, instead of passing a CheckpointStorage object. 3.StateBackend: Method: org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#setStateBackend(StateBackend backend) org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#getStateBackend() pyflink.datastream.stream_execution_environment.StreamExecutionEnvironment#set_state_backend(self, state_backend: StateBackend) Field: org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#defaultStateBackend Suggested alternative: Users can configure "state.backend.type" in the configuration as the fully qualified name of the state backend or use some FLINK-provided state backend shortcut names such as "hashmap" and "rocksdb", and provide the necessary configuration options for building that StateBackend, instead of passing a StateBackend object. More details can be found at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992. was: The non-ConfigOption objects in the StreamExecutionEnvironment, CheckpointConfig, and ExecutionConfig and their corresponding getter/setter interfaces is now be deprecated in FLINK-1.19. And these objects and methods is planned to be removed in Flink-2.0. Detailed information regarding the deprecations is as follows: 1. RestartStrategy: Class: org.apache.flink.api.common.restartstrategy.RestartStrategies org.apache.flink.api.common.restartstrategy.RestartStrategies.RestartStrategyConfiguration org.apache.flink.api.common.restartstrategy.RestartStrategies.FixedDelayRestartStrategyConfiguration org.apache.flink.api.common.restartstrategy.RestartStrategies.
[jira] [Updated] (FLINK-34126) Correct the description of jobmanager.scheduler
[ https://issues.apache.org/jira/browse/FLINK-34126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-34126: -- Component/s: Documentation (was: Runtime / Configuration) > Correct the description of jobmanager.scheduler > --- > > Key: FLINK-34126 > URL: https://issues.apache.org/jira/browse/FLINK-34126 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: Junrui Li >Priority: Major > Labels: pull-request-available > > Now the config option jobmanager.scheduler has description: > _Determines which scheduler implementation is used to schedule tasks. > Accepted values are:_ > * _'Default': Default scheduler_ > * _'Adaptive': Adaptive scheduler. More details can be found > [here|https://nightlies.apache.org/flink/flink-docs-master/zh/docs/deployment/elastic_scaling#adaptive-scheduler]._ > * _'AdaptiveBatch': Adaptive batch scheduler. More details can be found > [here|https://nightlies.apache.org/flink/flink-docs-master/zh/docs/deployment/elastic_scaling#adaptive-batch-scheduler]._ > _Possible values:_ > * _"Default"_ > * _"Adaptive"_ > * _"AdaptiveBatch"_ > > However, after FLIP-283 we changed the default scheduler for batch job to > AdaptiveBatchScheduler. This config option description will mislead users > that the 'DefaultScheduler' is the universal fallback for both batch and > streaming jobs. > We should update this description. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34296) Release Testing Instructions: Verify FLINK-33581 Deprecate configuration getters/setters that return/set complex Java objects
[ https://issues.apache.org/jira/browse/FLINK-34296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17813466#comment-17813466 ] Junrui Li commented on FLINK-34296: --- [~lincoln.86xy] This change doesn't need cross testing. I think we could close this ticket. > Release Testing Instructions: Verify FLINK-33581 Deprecate configuration > getters/setters that return/set complex Java objects > - > > Key: FLINK-34296 > URL: https://issues.apache.org/jira/browse/FLINK-34296 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Configuration >Affects Versions: 1.19.0 >Reporter: lincoln lee >Assignee: Junrui Li >Priority: Blocker > Labels: release-testing > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34295) Release Testing Instructions: Verify FLINK-33712 Deprecate RuntimeContext#getExecutionConfig
[ https://issues.apache.org/jira/browse/FLINK-34295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17813465#comment-17813465 ] Junrui Li commented on FLINK-34295: --- [~lincoln.86xy] This change doesn't need cross testing. I think we could close this ticket. > Release Testing Instructions: Verify FLINK-33712 Deprecate > RuntimeContext#getExecutionConfig > > > Key: FLINK-34295 > URL: https://issues.apache.org/jira/browse/FLINK-34295 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Configuration >Affects Versions: 1.19.0 >Reporter: lincoln lee >Assignee: Junrui Li >Priority: Blocker > Labels: release-testing > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33297) FLIP-366: Support standard YAML for FLINK configuration
[ https://issues.apache.org/jira/browse/FLINK-33297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-33297: -- Release Note: Starting with Flink-1.19, Flink has officially introduced full support for the standard YAML 1.2 syntax (FLIP-366). The default configuration file has been changed to `config.yaml` and placed in the `conf/` directory. Users should directly modify this file to configure Flink. If users want to use the legacy configuration file `flink-conf.yaml`, users just need to copy this file into the `conf/` directory. Once the legacy configuration file `flink-conf.yaml` is detected, Flink will prioritize using it as the configuration file. And in the upcoming Flink 2.0, the `flink-conf.yaml` configuration file will no longer work. More details could be found at Flink website (https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#flink-configuration-file) > FLIP-366: Support standard YAML for FLINK configuration > --- > > Key: FLINK-33297 > URL: https://issues.apache.org/jira/browse/FLINK-33297 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: 2.0-related, pull-request-available > Fix For: 1.19.0 > > > Support standard YAML for FLINK configuration -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33297) FLIP-366: Support standard YAML for FLINK configuration
[ https://issues.apache.org/jira/browse/FLINK-33297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li closed FLINK-33297. - Resolution: Fixed > FLIP-366: Support standard YAML for FLINK configuration > --- > > Key: FLINK-33297 > URL: https://issues.apache.org/jira/browse/FLINK-33297 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: 2.0-related, pull-request-available > Fix For: 1.19.0 > > > Support standard YAML for FLINK configuration -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34294) Release Testing Instructions: Verify FLINK-33297 Support standard YAML for FLINK configuration
[ https://issues.apache.org/jira/browse/FLINK-34294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814569#comment-17814569 ] Junrui Li commented on FLINK-34294: --- [~lincoln.86xy] Yes, this feature does require cross-team testing. I will prepare the test instructions and create a new ticket follow the steps. > Release Testing Instructions: Verify FLINK-33297 Support standard YAML for > FLINK configuration > -- > > Key: FLINK-34294 > URL: https://issues.apache.org/jira/browse/FLINK-34294 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Configuration >Affects Versions: 1.19.0 >Reporter: lincoln lee >Assignee: Junrui Li >Priority: Blocker > Labels: release-testing > Fix For: 1.19.0 > > Attachments: screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34377) Release Testing : Verify FLINK-33297 Support standard YAML for FLINK configuration
Junrui Li created FLINK-34377: - Summary: Release Testing : Verify FLINK-33297 Support standard YAML for FLINK configuration Key: FLINK-34377 URL: https://issues.apache.org/jira/browse/FLINK-34377 Project: Flink Issue Type: Sub-task Components: Runtime / Configuration Affects Versions: 1.19.0 Reporter: Junrui Li Assignee: Junrui Li Fix For: 1.19.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34377) Release Testing : Verify FLINK-33297 Support standard YAML for FLINK configuration
[ https://issues.apache.org/jira/browse/FLINK-34377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-34377: -- Description: This issue aims to verify FLIP-366. Starting with version 1.19, Flink has officially introduced full support for the standard YAML 1.2 syntax. For detailed information, please refer to the Flink website:https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#flink-configuration-file We may need to cover the following two types of test cases: Test 1: For newly created jobs, utilize a config.yaml file to set up the Flink cluster. We need to verify that the job runs as expected with this new configuration. Test 2: For existing jobs, migrate the legacy flink-conf.yaml to the new config.yaml. Test the job runs just like before post-migration. > Release Testing : Verify FLINK-33297 Support standard YAML for FLINK > configuration > -- > > Key: FLINK-34377 > URL: https://issues.apache.org/jira/browse/FLINK-34377 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Configuration >Affects Versions: 1.19.0 >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Blocker > Labels: release-testing > Fix For: 1.19.0 > > > This issue aims to verify FLIP-366. > Starting with version 1.19, Flink has officially introduced full support for > the standard YAML 1.2 syntax. For detailed information, please refer to the > Flink > website:https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#flink-configuration-file > We may need to cover the following two types of test cases: > Test 1: For newly created jobs, utilize a config.yaml file to set up the > Flink cluster. We need to verify that the job runs as expected with this new > configuration. > Test 2: For existing jobs, migrate the legacy flink-conf.yaml to the new > config.yaml. Test the job runs just like before post-migration. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33581) FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects
[ https://issues.apache.org/jira/browse/FLINK-33581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-33581: -- Release Note: The non-ConfigOption objects in the StreamExecutionEnvironment, CheckpointConfig, and ExecutionConfig and their corresponding getter/setter interfaces is now be deprecated in FLINK-1.19. And these objects and methods is planned to be removed in Flink-2.0. Detailed information regarding the deprecations is as follows: 1. RestartStrategy: Suggested alternative: Users can configure the RestartStrategyOptions related ConfigOptions, such as "restart-strategy.type", in the configuration, instead of passing a RestartStrategyConfiguration object. 2.CheckpointStorage: Suggested alternative: Users can configure "state.checkpoint-storage" in the configuration as the fully qualified name of the checkpoint storage or use some FLINK-provided checkpoint storage shortcut names such as "jobmanager" and "filesystem", and provide the necessary configuration options for building that storage, instead of passing a CheckpointStorage object. 3.StateBackend: Suggested alternative: Users can configure "state.backend.type" in the configuration as the fully qualified name of the state backend or use some FLINK-provided state backend shortcut names such as "hashmap" and "rocksdb", and provide the necessary configuration options for building that StateBackend, instead of passing a StateBackend object. More details can be found at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992. was: The non-ConfigOption objects in the StreamExecutionEnvironment, CheckpointConfig, and ExecutionConfig and their corresponding getter/setter interfaces is now be deprecated in FLINK-1.19. And these objects and methods is planned to be removed in Flink-2.0. Detailed information regarding the deprecations is as follows: 1. RestartStrategy: Class: org.apache.flink.api.common.restartstrategy.RestartStrategies org.apache.flink.api.common.restartstrategy.RestartStrategies.RestartStrategyConfiguration org.apache.flink.api.common.restartstrategy.RestartStrategies.FixedDelayRestartStrategyConfiguration org.apache.flink.api.common.restartstrategy.RestartStrategies.ExponentialDelayRestartStrategyConfiguration org.apache.flink.api.common.restartstrategy.RestartStrategies.FailureRateRestartStrategyConfiguration org.apache.flink.api.common.restartstrategy.RestartStrategies.FallbackRestartStrategyConfiguration Method: org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#setRestartStrategy(RestartStrategies.RestartStrategyConfiguration restartStrategyConfiguration) org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#getRestartStrategy() org.apache.flink.api.common.ExecutionConfig#getRestartStrategy() org.apache.flink.api.common.ExecutionConfig#setRestartStrategy(RestartStrategies.RestartStrategyConfiguration restartStrategyConfiguration) pyflink.common.execution_config.ExecutionConfig#set_restart_strategy(self, restart_strategy_configuration: RestartStrategyConfiguration) pyflink.datastream.stream_execution_environment.StreamExecutionEnvironment#set_restart_strategy(self, restart_strategy_configuration: RestartStrategyConfiguration) Field: org.apache.flink.api.common.ExecutionConfig#restartStrategyConfiguration Suggested alternative: Users can configure the RestartStrategyOptions related ConfigOptions, such as "restart-strategy.type", in the configuration, instead of passing a RestartStrategyConfiguration object. 2.CheckpointStorage: Method: org.apache.flink.streaming.api.environment.CheckpointConfig#setCheckpointStorage(CheckpointStorage storage) org.apache.flink.streaming.api.environment.CheckpointConfig#setCheckpointStorage(String checkpointDirectory) org.apache.flink.streaming.api.environment.CheckpointConfig#setCheckpointStorage(URI checkpointDirectory) org.apache.flink.streaming.api.environment.CheckpointConfig#setCheckpointStorage(Path checkpointDirectory) org.apache.flink.streaming.api.environment.CheckpointConfig#getCheckpointStorage() pyflink.datastream.checkpoint_config.CheckpointConfig#set_checkpoint_storage(self, storage: CheckpointStorage) pyflink.datastream.checkpoint_config.CheckpointConfig#set_checkpoint_storage_dir(self, checkpoint_path: str) pyflink.datastream.checkpoint_config.CheckpointConfig#get_checkpoint_storage(self) Suggested alternative: Users can configure "state.checkpoint-storage" in the configuration as the fully qualified name of the checkpoint storage or use some FLINK-provided checkpoint storage shortcut names such as "jobmanager" and "filesystem", and provide the necessary configuration options for building that storage, instead of passing a CheckpointStorage object. 3.StateBackend: Method: org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#setStateBackend(StateBackend backend) org.apache.flink
[jira] [Updated] (FLINK-33581) FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects
[ https://issues.apache.org/jira/browse/FLINK-33581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-33581: -- Release Note: The non-ConfigOption objects in the StreamExecutionEnvironment, CheckpointConfig, and ExecutionConfig and their corresponding getter/setter interfaces is now be deprecated in FLINK-1.19. And these objects and methods is planned to be removed in Flink-2.0. Detailed information regarding the deprecations is as follows: 1. RestartStrategy: Suggested alternative: Users can configure the RestartStrategyOptions related ConfigOptions, such as "restart-strategy.type", in the configuration, instead of passing a RestartStrategyConfiguration object. 2.CheckpointStorage: Suggested alternative: Users can configure "state.checkpoint-storage" in the configuration as the fully qualified name of the checkpoint storage or use some FLINK-provided checkpoint storage shortcut names such as "jobmanager" and "filesystem", and provide the necessary configuration options for building that storage, instead of passing a CheckpointStorage object. 3.StateBackend: Suggested alternative: Users can configure "state.backend.type" in the configuration as the fully qualified name of the state backend or use some FLINK-provided state backend shortcut names such as "hashmap" and "rocksdb", and provide the necessary configuration options for building that StateBackend, instead of passing a StateBackend object. More details can be found at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992. was: The non-ConfigOption objects in the StreamExecutionEnvironment, CheckpointConfig, and ExecutionConfig and their corresponding getter/setter interfaces is now be deprecated in FLINK-1.19. And these objects and methods is planned to be removed in Flink-2.0. Detailed information regarding the deprecations is as follows: 1. RestartStrategy: Suggested alternative: Users can configure the RestartStrategyOptions related ConfigOptions, such as "restart-strategy.type", in the configuration, instead of passing a RestartStrategyConfiguration object. 2.CheckpointStorage: Suggested alternative: Users can configure "state.checkpoint-storage" in the configuration as the fully qualified name of the checkpoint storage or use some FLINK-provided checkpoint storage shortcut names such as "jobmanager" and "filesystem", and provide the necessary configuration options for building that storage, instead of passing a CheckpointStorage object. 3.StateBackend: Suggested alternative: Users can configure "state.backend.type" in the configuration as the fully qualified name of the state backend or use some FLINK-provided state backend shortcut names such as "hashmap" and "rocksdb", and provide the necessary configuration options for building that StateBackend, instead of passing a StateBackend object. More details can be found at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992. > FLIP-381: Deprecate configuration getters/setters that return/set complex > Java objects > -- > > Key: FLINK-33581 > URL: https://issues.apache.org/jira/browse/FLINK-33581 > Project: Flink > Issue Type: Technical Debt > Components: API / DataStream >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > Attachments: image-2023-11-30-17-59-42-650.png > > > Deprecate the non-ConfigOption objects in the StreamExecutionEnvironment, > CheckpointConfig, and ExecutionConfig, and ultimately removing them in FLINK > 2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33581) FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects
[ https://issues.apache.org/jira/browse/FLINK-33581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-33581: -- Release Note: The non-ConfigOption objects in the StreamExecutionEnvironment, CheckpointConfig, and ExecutionConfig and their corresponding getter/setter interfaces is now be deprecated in FLINK-1.19. And these objects and methods is planned to be removed in Flink-2.0. The deprecated interfaces include the getter and setter methods of RestartStrategy, CheckpointStorage, and StateBackend. More details can be found at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992. was: The non-ConfigOption objects in the StreamExecutionEnvironment, CheckpointConfig, and ExecutionConfig and their corresponding getter/setter interfaces is now be deprecated in FLINK-1.19. And these objects and methods is planned to be removed in Flink-2.0. Detailed information regarding the deprecations is as follows: 1. RestartStrategy: Suggested alternative: Users can configure the RestartStrategyOptions related ConfigOptions, such as "restart-strategy.type", in the configuration, instead of passing a RestartStrategyConfiguration object. 2.CheckpointStorage: Suggested alternative: Users can configure "state.checkpoint-storage" in the configuration as the fully qualified name of the checkpoint storage or use some FLINK-provided checkpoint storage shortcut names such as "jobmanager" and "filesystem", and provide the necessary configuration options for building that storage, instead of passing a CheckpointStorage object. 3.StateBackend: Suggested alternative: Users can configure "state.backend.type" in the configuration as the fully qualified name of the state backend or use some FLINK-provided state backend shortcut names such as "hashmap" and "rocksdb", and provide the necessary configuration options for building that StateBackend, instead of passing a StateBackend object. More details can be found at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992. > FLIP-381: Deprecate configuration getters/setters that return/set complex > Java objects > -- > > Key: FLINK-33581 > URL: https://issues.apache.org/jira/browse/FLINK-33581 > Project: Flink > Issue Type: Technical Debt > Components: API / DataStream >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > Attachments: image-2023-11-30-17-59-42-650.png > > > Deprecate the non-ConfigOption objects in the StreamExecutionEnvironment, > CheckpointConfig, and ExecutionConfig, and ultimately removing them in FLINK > 2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33581) FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects
[ https://issues.apache.org/jira/browse/FLINK-33581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-33581: -- Release Note: The non-ConfigOption objects in the StreamExecutionEnvironment, CheckpointConfig, and ExecutionConfig and their corresponding getter/setter interfaces are now be deprecated in FLINK-1.19. And these objects and methods are planned to be removed in Flink-2.0. The deprecated interfaces include the getter and setter methods of RestartStrategy, CheckpointStorage, and StateBackend. More details can be found at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992. was: The non-ConfigOption objects in the StreamExecutionEnvironment, CheckpointConfig, and ExecutionConfig and their corresponding getter/setter interfaces is now be deprecated in FLINK-1.19. And these objects and methods is planned to be removed in Flink-2.0. The deprecated interfaces include the getter and setter methods of RestartStrategy, CheckpointStorage, and StateBackend. More details can be found at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992. > FLIP-381: Deprecate configuration getters/setters that return/set complex > Java objects > -- > > Key: FLINK-33581 > URL: https://issues.apache.org/jira/browse/FLINK-33581 > Project: Flink > Issue Type: Technical Debt > Components: API / DataStream >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > Attachments: image-2023-11-30-17-59-42-650.png > > > Deprecate the non-ConfigOption objects in the StreamExecutionEnvironment, > CheckpointConfig, and ExecutionConfig, and ultimately removing them in FLINK > 2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34356) Release Testing: Verify FLINK-33768 Support dynamic source parallelism inference for batch jobs
[ https://issues.apache.org/jira/browse/FLINK-34356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819132#comment-17819132 ] Junrui Li commented on FLINK-34356: --- I'd like to take this ticket. [~zhuzh] Could you help to assign this to me? > Release Testing: Verify FLINK-33768 Support dynamic source parallelism > inference for batch jobs > - > > Key: FLINK-34356 > URL: https://issues.apache.org/jira/browse/FLINK-34356 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.19.0 >Reporter: lincoln lee >Priority: Blocker > Labels: release-testing > Fix For: 1.19.0 > > > This issue aims to verify FLIP-379. > New Source can implement the interface DynamicParallelismInference to enable > dynamic parallelism inference. For detailed information, please refer to the > documentation. > We may need to cover the following two types of test cases: > Test 1: FileSource has implemented the dynamic source parallelism inference. > Test the automatic parallelism inference of FileSource. > Test 2: Test the dynamic source parallelism inference of a custom Source. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34356) Release Testing: Verify FLINK-33768 Support dynamic source parallelism inference for batch jobs
[ https://issues.apache.org/jira/browse/FLINK-34356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820544#comment-17820544 ] Junrui Li commented on FLINK-34356: --- I have conducted tests for both scenarios as instructed: # With the FileSource, when parallelism.default is greater than the number of files, the parallelism is set to the number of files. Conversely, when parallelism.default is less than the number of files, the parallelism.default value is used. # With a custom source that implemented the DynamicParallelismInference interface, the results indicate that the parallelism is determined by the DynamicParallelismInference interface. All tests completed successfully. The test results are consistent with expectations. > Release Testing: Verify FLINK-33768 Support dynamic source parallelism > inference for batch jobs > - > > Key: FLINK-34356 > URL: https://issues.apache.org/jira/browse/FLINK-34356 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.19.0 >Reporter: lincoln lee >Assignee: Junrui Li >Priority: Blocker > Labels: release-testing > Fix For: 1.19.0 > > > This issue aims to verify FLIP-379. > New Source can implement the interface DynamicParallelismInference to enable > dynamic parallelism inference. For detailed information, please refer to the > documentation. > We may need to cover the following two types of test cases: > Test 1: FileSource has implemented the dynamic source parallelism inference. > Test the automatic parallelism inference of FileSource. > Test 2: Test the dynamic source parallelism inference of a custom Source. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33986) Extend shuffleMaster to support batch snapshot.
[ https://issues.apache.org/jira/browse/FLINK-33986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-33986: -- Summary: Extend shuffleMaster to support batch snapshot. (was: Introduce SupportsBatchSnapshot for shuffle master) > Extend shuffleMaster to support batch snapshot. > --- > > Key: FLINK-33986 > URL: https://issues.apache.org/jira/browse/FLINK-33986 > Project: Flink > Issue Type: Sub-task >Reporter: Junrui Li >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33986) Extend shuffleMaster to support batch snapshot.
[ https://issues.apache.org/jira/browse/FLINK-33986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-33986: -- Description: Extend shuffleMaster to support batch snapshot as follows: # Add method supportsBatchSnapshot to identify whether the shuffle master supports taking snapshot in batch scenarios # Add method snapshotState and restoreState to snapshot and restore the shuffle master's state. > Extend shuffleMaster to support batch snapshot. > --- > > Key: FLINK-33986 > URL: https://issues.apache.org/jira/browse/FLINK-33986 > Project: Flink > Issue Type: Sub-task >Reporter: Junrui Li >Priority: Major > > Extend shuffleMaster to support batch snapshot as follows: > # Add method supportsBatchSnapshot to identify whether the shuffle master > supports taking snapshot in batch scenarios > # Add method snapshotState and restoreState to snapshot and restore the > shuffle master's state. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33985) Extend ShuffleMaster to fetch all partition
[ https://issues.apache.org/jira/browse/FLINK-33985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17822106#comment-17822106 ] Junrui Li commented on FLINK-33985: --- This ticket can be closed as its contents will be covered by another ticket. > Extend ShuffleMaster to fetch all partition > --- > > Key: FLINK-33985 > URL: https://issues.apache.org/jira/browse/FLINK-33985 > Project: Flink > Issue Type: Sub-task >Reporter: Junrui Li >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33985) Extend ShuffleMaster to fetch all partition
[ https://issues.apache.org/jira/browse/FLINK-33985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li closed FLINK-33985. - Resolution: Duplicate > Extend ShuffleMaster to fetch all partition > --- > > Key: FLINK-33985 > URL: https://issues.apache.org/jira/browse/FLINK-33985 > Project: Flink > Issue Type: Sub-task >Reporter: Junrui Li >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30358) Show the task manager id on the exception history page
[ https://issues.apache.org/jira/browse/FLINK-30358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JunRui Li updated FLINK-30358: -- Attachment: 8371ECF0-4F7F-45EE-93B0-7EC6F7F57D72.png > Show the task manager id on the exception history page > -- > > Key: FLINK-30358 > URL: https://issues.apache.org/jira/browse/FLINK-30358 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Reporter: JunRui Li >Assignee: JunRui Li >Priority: Major > Attachments: 8371ECF0-4F7F-45EE-93B0-7EC6F7F57D72.png > > > At present, the web UI exception history page only displays the TM host and > port. However, we generally need to search for problems according to the pod > name or container ID. > Therefore, it is more convenient to add resource id (pod name on k8s, > container id on yarn) to the location column and a link to the task manager > id to jump to the task manager page. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-23620) Introduce proper YAML parsing to Flink's configuration
[
https://issues.apache.org/jira/browse/FLINK-23620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17737499#comment-17737499
]
Junrui Li commented on FLINK-23620:
---
Hi [~autophagy], are you still working on this issue? If not, I would like to
take this issue.
IIUC, FLINK-29372 solves the problem that the key of some configuration items
is the prefix of the key of other configuration items.
And there are still two problems:
1. The flink-conf parser will read row-by-row and just skipped invalid yaml
rows. But the standard YAML parser will discard the whole file as long as there
are any invalid lines.
2. Even if there are no errors in parsing, use standard YAML parser may lead to
inconsistencies.
For example, if the user configures: "key1: value1#comment1" in
flink-conf.yaml, the old parser will regard "value1" as value and "#comment1"
as comment. But the standard YAML parser requried "Comments must be separated
from other tokens by white space characters", so the standard YAML parser will
regard "value1#comment1" as value. This can lead to an unexpected behavior
change and users may not even be able to notice it.
So I plan to introduce a flink-config.yaml file which is parsed using standard
YAML parser. As long as there is this file under the "conf" folder, it will be
used as the flink configuration file, otherwise flink-conf.yaml will be used
and the old parsing method will be used.
If there is no concern about this proposal, I will create a FLIP for it.
cc [~zhuzh] [~chesnay]
> Introduce proper YAML parsing to Flink's configuration
> --
>
> Key: FLINK-23620
> URL: https://issues.apache.org/jira/browse/FLINK-23620
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration
>Reporter: Mika Naylor
>Assignee: Mika Naylor
>Priority: Minor
> Labels: pull-request-available, stale-assigned
>
> At the moment, the YAML parsing for Flink's configuration file
> ({{conf/flink-conf.yaml)}} is pretty basic. It only supports basic key value
> pairs, such as:
> {code:java}
> a.b.c: a value
> a.b.d: another value{code}
> As well as supporting some invalid YAML syntax, such as:
> {code:java}
> a: b: value{code}
>
> Introducing proper YAML parsing to the configuration component would let
> Flink users use features such as nested keys, such as:
> {code:java}
> a:
> b:
> c: a value
> d: another value{code}
> as well as make it easier to integrate configuration tools/languages that
> compile to YAML, such as Dhall.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32669) Support range-port for taskmanager data port
[ https://issues.apache.org/jira/browse/FLINK-32669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17746897#comment-17746897 ] Junrui Li commented on FLINK-32669: --- IIUC, the default value of the configuration item taskmanager.data.port is 0, which means that an idle port is randomly selected, so generally speaking, no port conflicts will occur. Could you please explain in detail the scenario where port conflicts will occur if this configuration item is not set? > Support range-port for taskmanager data port > > > Key: FLINK-32669 > URL: https://issues.apache.org/jira/browse/FLINK-32669 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Reporter: chenyuzhi >Priority: Major > > We can setup range-port for taskmanager rpc port to avoid occupying an > unexpected port(such as the port of datanode service). > > However, we can't setup range-port for taskmanager data port(config-key: > taskmanager.data.port). In production env, it's unreasonable to setup a > specify port, thus we usually not setup this configuration key. > > The problem is without setup taskmanager data port, it's possible to conflict > with port of other services. Thus is it necessary to support range-port for > taskmanager data port ? > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-32669) Support range-port for taskmanager data port
[ https://issues.apache.org/jira/browse/FLINK-32669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17746897#comment-17746897 ] Junrui Li edited comment on FLINK-32669 at 7/25/23 10:19 AM: - IIUC, the default value of the configuration item taskmanager.data.port is 0, which means that an idle port will be randomly selected, so generally speaking, no port conflicts will occur. Could you please explain in detail the scenario where port conflicts will occur if this configuration item is not set? was (Author: JIRAUSER293716): IIUC, the default value of the configuration item taskmanager.data.port is 0, which means that an idle port is randomly selected, so generally speaking, no port conflicts will occur. Could you please explain in detail the scenario where port conflicts will occur if this configuration item is not set? > Support range-port for taskmanager data port > > > Key: FLINK-32669 > URL: https://issues.apache.org/jira/browse/FLINK-32669 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Reporter: chenyuzhi >Priority: Major > > We can setup range-port for taskmanager rpc port to avoid occupying an > unexpected port(such as the port of datanode service). > > However, we can't setup range-port for taskmanager data port(config-key: > taskmanager.data.port). In production env, it's unreasonable to setup a > specify port, thus we usually not setup this configuration key. > > The problem is without setup taskmanager data port, it's possible to conflict > with port of other services. Thus is it necessary to support range-port for > taskmanager data port ? > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32680) Job vertex names get messed up once there is a source vertex chained with a MultipleInput vertex in job graph
[
https://issues.apache.org/jira/browse/FLINK-32680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17747409#comment-17747409
]
Junrui Li commented on FLINK-32680:
---
This bug is because the global ChainedSources are used when generating the
JobVertex name
([here|https://github.com/apache/flink/blob/c8ae39d4ac73f81873e1d8ac37e17c29ae330b23/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java#L903]).
But in fact, it should be filtered according to the id of the current node. An
example can refer to is
[here.|https://github.com/apache/flink/blob/c8ae39d4ac73f81873e1d8ac37e17c29ae330b23/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java#L1052]
If the idea is correct, I will prepare a PR to fix this issue.
> Job vertex names get messed up once there is a source vertex chained with a
> MultipleInput vertex in job graph
> -
>
> Key: FLINK-32680
> URL: https://issues.apache.org/jira/browse/FLINK-32680
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.16.2, 1.18.0, 1.17.1
>Reporter: Lijie Wang
>Priority: Major
> Attachments: image-2023-07-26-15-23-29-551.png,
> image-2023-07-26-15-24-24-077.png
>
>
> Take the following test(put it to {{MultipleInputITCase}}) as example:
> {code:java}
> @Test
> public void testMultipleInputDoesNotChainedWithSource() throws Exception {
> testJobVertexName(false);
> }
>
> @Test
> public void testMultipleInputChainedWithSource() throws Exception {
> testJobVertexName(true);
> }
> public void testJobVertexName(boolean chain) throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setParallelism(1);
> TestListResultSink resultSink = new TestListResultSink<>();
> DataStream source1 = env.fromSequence(0L, 3L).name("source1");
> DataStream source2 = env.fromElements(4L, 6L).name("source2");
> DataStream source3 = env.fromElements(7L, 9L).name("source3");
> KeyedMultipleInputTransformation transform =
> new KeyedMultipleInputTransformation<>(
> "MultipleInput",
> new KeyedSumMultipleInputOperatorFactory(),
> BasicTypeInfo.LONG_TYPE_INFO,
> 1,
> BasicTypeInfo.LONG_TYPE_INFO);
> if (chain) {
> transform.setChainingStrategy(ChainingStrategy.HEAD_WITH_SOURCES);
> }
> KeySelector keySelector = (KeySelector) value
> -> value % 3;
> env.addOperator(
> transform
> .addInput(source1.getTransformation(), keySelector)
> .addInput(source2.getTransformation(), keySelector)
> .addInput(source3.getTransformation(), keySelector));
> new
> MultipleConnectedStreams(env).transform(transform).rebalance().addSink(resultSink).name("sink");
> env.execute();
> }{code}
>
> When we run {{testMultipleInputDoesNotChainedWithSource}} , all job vertex
> names are normal:
> !image-2023-07-26-15-24-24-077.png|width=494,height=246!
> When we run {{testMultipleInputChainedWithSource}} (the MultipleInput chained
> with source1), job vertex names get messed up (all job vertex names contain
> {{{}Source: source1{}}}):
> !image-2023-07-26-15-23-29-551.png|width=515,height=182!
>
> I think it's a bug.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30682) FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
JunRui Li created FLINK-30682: - Summary: FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs Key: FLINK-30682 URL: https://issues.apache.org/jira/browse/FLINK-30682 Project: Flink Issue Type: Improvement Components: Runtime / Coordination Reporter: JunRui Li Fix For: 1.17.0 To further use the adaptive batch scheduler to improve flink's batch capability, in this FLIP we aim to make the adaptive batch scheduler as the default batch scheduler and optimize the current adaptive batch scheduler configuration. More details see [FLIP-283|https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30683) Make adaptive batch scheduler as the default batch scheduler
Junrui Li created FLINK-30683: - Summary: Make adaptive batch scheduler as the default batch scheduler Key: FLINK-30683 URL: https://issues.apache.org/jira/browse/FLINK-30683 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Reporter: Junrui Li Fix For: 1.17.0 Based on the [FLIP-283|https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs], this issue mainly focuses on the first issue. This change proposes to make AdaptiveBatchScheduler as the default batch scheduler and user can use it without explicitly configuring it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30684) Use the default parallelism as a flag for vertices that can automatically derive parallelism.
Junrui Li created FLINK-30684: - Summary: Use the default parallelism as a flag for vertices that can automatically derive parallelism. Key: FLINK-30684 URL: https://issues.apache.org/jira/browse/FLINK-30684 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Reporter: Junrui Li Fix For: 1.17.0 This change proposes to add a *parallelismConfigured* property as a flag to identify whether the parallelism of node is used "parallelism.default" or not. If the vertex's *parallelismConfigured* is true, the AdaptiveBatchScheduler will not automatically deciding parallelisms for it. Otherwise, AdaptiveBatchScheduler will automatically deciding parallelisms and use the "parallelism.default" as an alternative value for the "jobmanager.adaptive-batch-scheduler.max-parallelism". This change will make user do not need to configure "parallelism.default" as "-1" to automatically deciding parallelisms for vertices. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30685) Support mark the transformations whose parallelism is infected by the input transformation
Junrui Li created FLINK-30685: - Summary: Support mark the transformations whose parallelism is infected by the input transformation Key: FLINK-30685 URL: https://issues.apache.org/jira/browse/FLINK-30685 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Reporter: Junrui Li Fix For: 1.17.0 In order to chain operators together as much as possible, many downstream operators will use the parallelism of upstream input operators in the table planner. If some operators need to have their own defined parallelism, the parallelism will be explicitly set. Therefore, the operator that takes the parallelism of the upstream operator as its own parallelism should be automatically derived by the AdaptiveBatchScheduler. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30686) Simplify the configuration of adaptive batch scheduler
Junrui Li created FLINK-30686: - Summary: Simplify the configuration of adaptive batch scheduler Key: FLINK-30686 URL: https://issues.apache.org/jira/browse/FLINK-30686 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Reporter: Junrui Li Fix For: 1.17.0 Based on the [FLIP-283|[https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs]], this issue mainly focuses on the second issue. This change includes three parts: 1.Introduce "execution.batch.adaptive.auto-parallelism.enabled" as a switch for automatic parallelism derivation 2.Modify adaptive batch scheduler configuration default values 3.Rename the configuration of adaptive batch scheduler -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27756) Fix intermittently failing test in AsyncSinkWriterTest.checkLoggedSendTimesAreWithinBounds
[
https://issues.apache.org/jira/browse/FLINK-27756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798795#comment-17798795
]
Junrui Li commented on FLINK-27756:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55673&view=logs&j=2e8cb2f7-b2d3-5c62-9c05-cd756d33a819&t=2dd510a3-5041-5201-6dc3-54d310f68906
> Fix intermittently failing test in
> AsyncSinkWriterTest.checkLoggedSendTimesAreWithinBounds
> --
>
> Key: FLINK-27756
> URL: https://issues.apache.org/jira/browse/FLINK-27756
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.15.0, 1.17.0, 1.19.0
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Critical
> Labels: test-stability
> Fix For: 1.16.0
>
>
> h2. Motivation
> - One of the integration tests ({{checkLoggedSendTimesAreWithinBounds}}) of
> {{AsyncSinkWriterTest}} has been reported to fail intermittently on build
> pipeline causing blocking of new changes.
> - Reporting build is [linked
> |https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=36009&view=logs&j=aa18c3f6-13b8-5f58-86bb-c1cffb239496&t=502fb6c0-30a2-5e49-c5c2-a00fa3acb203]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33980) Reorganize job configuration
Junrui Li created FLINK-33980: - Summary: Reorganize job configuration Key: FLINK-33980 URL: https://issues.apache.org/jira/browse/FLINK-33980 Project: Flink Issue Type: Technical Debt Components: API / Core Reporter: Junrui Li Currently, job configuration in FLINK is spread out across different components, including StreamExecutionEnvironment, CheckpointConfig, and ExecutionConfig. This distribution leads to inconsistencies among the configurations stored within these components. Furthermore, the methods used to configure these components vary; some rely on complex Java objects, while others use ConfigOption, which is a key-value configuration approach. This variation complicates the effective management of job configurations. Additionally, passing complex Java objects (e.g., StateBackend and CheckpointStorage) between the environment, StreamGraph, and JobGraph adds complexity to development. With the completion of FLIP-381, it is now time to standardize and unify job configuration in FLINK. The goals of this JIRA are as follows: # Migrate configuration from non-ConfigOption objects to use ConfigOption. # Adopt a single Configuration object to house all configurations. # Create complex Java objects, such as RestartBackoffTimeStrategyFactory, CheckpointStorage, and StateBackend, directly from the configuration on the JM side. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33982) Introduce new config options for Job Recovery.
Junrui Li created FLINK-33982: - Summary: Introduce new config options for Job Recovery. Key: FLINK-33982 URL: https://issues.apache.org/jira/browse/FLINK-33982 Project: Flink Issue Type: Sub-task Components: API / Core Reporter: Junrui Li -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33983) Introduce JobEvent and JobEventStore for Job Recovery
Junrui Li created FLINK-33983: - Summary: Introduce JobEvent and JobEventStore for Job Recovery Key: FLINK-33983 URL: https://issues.apache.org/jira/browse/FLINK-33983 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Reporter: Junrui Li -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33982) Introduce new config options for Batch Job Recovery
[ https://issues.apache.org/jira/browse/FLINK-33982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-33982: -- Summary: Introduce new config options for Batch Job Recovery (was: Introduce new config options for Job Recovery.) > Introduce new config options for Batch Job Recovery > --- > > Key: FLINK-33982 > URL: https://issues.apache.org/jira/browse/FLINK-33982 > Project: Flink > Issue Type: Sub-task > Components: API / Core >Reporter: Junrui Li >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33983) Introduce JobEvent and JobEventStore for Batch Job Recovery
[ https://issues.apache.org/jira/browse/FLINK-33983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-33983: -- Summary: Introduce JobEvent and JobEventStore for Batch Job Recovery (was: Introduce JobEvent and JobEventStore for Job Recovery) > Introduce JobEvent and JobEventStore for Batch Job Recovery > --- > > Key: FLINK-33983 > URL: https://issues.apache.org/jira/browse/FLINK-33983 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Junrui Li >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33984) Introduce SupportsBatchSnapshot for operator coordinator
Junrui Li created FLINK-33984: - Summary: Introduce SupportsBatchSnapshot for operator coordinator Key: FLINK-33984 URL: https://issues.apache.org/jira/browse/FLINK-33984 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Reporter: Junrui Li -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33985) Extend ShuffleMaster to fetch all partition
Junrui Li created FLINK-33985: - Summary: Extend ShuffleMaster to fetch all partition Key: FLINK-33985 URL: https://issues.apache.org/jira/browse/FLINK-33985 Project: Flink Issue Type: Sub-task Reporter: Junrui Li -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33986) Introduce SupportsBatchSnapshot for shuffle master
Junrui Li created FLINK-33986: - Summary: Introduce SupportsBatchSnapshot for shuffle master Key: FLINK-33986 URL: https://issues.apache.org/jira/browse/FLINK-33986 Project: Flink Issue Type: Sub-task Reporter: Junrui Li -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34015) Setting `execution.savepoint.ignore-unclaimed-state` does not take effect when passing this parameter by dynamic properties
[
https://issues.apache.org/jira/browse/FLINK-34015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804140#comment-17804140
]
Junrui Li commented on FLINK-34015:
---
If the jobmanager log reflects your actual input and it shows a space after
{{{}-D{}}}, then the usage is incorrect. The correct format should be:
{code:java}
-Dexecution.savepoint.ignore-unclaimed-state=true{code}
> Setting `execution.savepoint.ignore-unclaimed-state` does not take effect
> when passing this parameter by dynamic properties
> ---
>
> Key: FLINK-34015
> URL: https://issues.apache.org/jira/browse/FLINK-34015
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.17.0
>Reporter: Renxiang Zhou
>Priority: Critical
> Labels: ignore-unclaimed-state-invalid
> Attachments: image-2024-01-08-14-22-09-758.png,
> image-2024-01-08-14-24-30-665.png
>
>
> We set `execution.savepoint.ignore-unclaimed-state` to true and use -D option
> to submit the job, but unfortunately we found the value is still false in
> jobmanager log.
> Pic 1: we set `execution.savepoint.ignore-unclaimed-state` to true in
> submiting job.
> !image-2024-01-08-14-22-09-758.png|width=1012,height=222!
> Pic 2: The value is still false in jmlog.
> !image-2024-01-08-14-24-30-665.png|width=651,height=51!
>
> Besides, the parameter `execution.savepoint-restore-mode` has the same
> problem since when we pass it by -D option.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34052) Missing TopSpeedWindowing and SessionWindowing JARs in Flink Maven Repository
Junrui Li created FLINK-34052: - Summary: Missing TopSpeedWindowing and SessionWindowing JARs in Flink Maven Repository Key: FLINK-34052 URL: https://issues.apache.org/jira/browse/FLINK-34052 Project: Flink Issue Type: Bug Components: Build System Affects Versions: 1.18.0 Reporter: Junrui Li As a result of the changes implemented in FLINK-32821, the build process no longer produces artifacts with the names flink-examples-streaming-1.x-TopSpeedWindowing.jar and flink-examples-streaming-1.x-SessionWindowing.jar. This has led to the absence of these specific JAR files in the Maven repository (https://repo1.maven.org/maven2/org/apache/flink/flink-examples-streaming/1.18.0/). These artifacts were previously available and may still be expected by users as part of their application dependencies. Their removal could potentially break existing build pipelines and applications that depend on these example JARs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34052) Missing TopSpeedWindowing and SessionWindowing JARs in Flink Maven Repository
[ https://issues.apache.org/jira/browse/FLINK-34052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805173#comment-17805173 ] Junrui Li commented on FLINK-34052: --- Hi [~Zhanghao Chen] , Could you have a look? > Missing TopSpeedWindowing and SessionWindowing JARs in Flink Maven Repository > - > > Key: FLINK-34052 > URL: https://issues.apache.org/jira/browse/FLINK-34052 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.18.0 >Reporter: Junrui Li >Priority: Major > > As a result of the changes implemented in FLINK-32821, the build process no > longer produces artifacts with the names > flink-examples-streaming-1.x-TopSpeedWindowing.jar and > flink-examples-streaming-1.x-SessionWindowing.jar. This has led to the > absence of these specific JAR files in the Maven repository > (https://repo1.maven.org/maven2/org/apache/flink/flink-examples-streaming/1.18.0/). > These artifacts were previously available and may still be expected by users > as part of their application dependencies. Their removal could potentially > break existing build pipelines and applications that depend on these example > JARs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34052) Missing TopSpeedWindowing and SessionWindowing JARs in Flink Maven Repository
[ https://issues.apache.org/jira/browse/FLINK-34052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li updated FLINK-34052: -- Component/s: Examples > Missing TopSpeedWindowing and SessionWindowing JARs in Flink Maven Repository > - > > Key: FLINK-34052 > URL: https://issues.apache.org/jira/browse/FLINK-34052 > Project: Flink > Issue Type: Bug > Components: Build System, Examples >Affects Versions: 1.18.0 >Reporter: Junrui Li >Priority: Major > > As a result of the changes implemented in FLINK-32821, the build process no > longer produces artifacts with the names > flink-examples-streaming-1.x-TopSpeedWindowing.jar and > flink-examples-streaming-1.x-SessionWindowing.jar. This has led to the > absence of these specific JAR files in the Maven repository > (https://repo1.maven.org/maven2/org/apache/flink/flink-examples-streaming/1.18.0/). > These artifacts were previously available and may still be expected by users > as part of their application dependencies. Their removal could potentially > break existing build pipelines and applications that depend on these example > JARs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
