[jira] [Commented] (FLINK-34898) Cannot create named STRUCT with a single field
[
https://issues.apache.org/jira/browse/FLINK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829401#comment-17829401
]
Feng Jin commented on FLINK-34898:
--
You can try using
{code:java}
SELECT ROW(1)
{code}
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/systemfunctions/#value-construction-functions
> The implicit row constructor requires at least two fields. The explicit row
> constructor can deal with an arbitrary number of fields. Both of them support
> arbitrary expressions as fields.
> Cannot create named STRUCT with a single field
> --
>
> Key: FLINK-34898
> URL: https://issues.apache.org/jira/browse/FLINK-34898
> Project: Flink
> Issue Type: Bug
>Reporter: Chloe He
>Priority: Major
> Attachments: image-2024-03-21-12-00-00-183.png
>
>
> I'm trying to create named structs using Flink SQL and I found a previous
> ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use
> of the following syntax:
> {code:java}
> SELECT CAST(('a', 1) as ROW) AS row1;
> {code}
> However, my named struct has a single field and effectively it should look
> something like `\{"a": 1}`. I can't seem to be able to find a way to
> construct this. I have experimented with a few different syntax and it either
> throws parsing error or casting error:
> {code:java}
> Cast function cannot convert value of type INTEGER to type
> RecordType(VARCHAR(2147483647) a) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34901][connectors/jdbc]Improve the performance of the update Postgres database [flink-connector-jdbc]
boring-cyborg[bot] commented on PR #108: URL: https://github.com/apache/flink-connector-jdbc/pull/108#issuecomment-2011343599 Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34668][checkpoint] Report state handle of file merging directory to JM [flink]
masteryhx commented on code in PR #24513:
URL: https://github.com/apache/flink/pull/24513#discussion_r155369
##
flink-runtime/src/test/java/org/apache/flink/runtime/state/OperatorStateBackendTest.java:
##
@@ -418,6 +430,62 @@ void testSnapshotEmpty() throws Exception {
assertThat(stateHandle).isNull();
}
+@Test
+void testFileMergingSnapshotEmpty(@TempDir File tmpFolder) throws
Exception {
Review Comment:
Could we also test the registery of the new handle ?
Or test that the subsumed checkpoint will discard correctly.
##
flink-runtime/src/main/java/org/apache/flink/runtime/state/filemerging/DirectoryStreamStateHandle.java:
##
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.filemerging;
+
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.runtime.state.DirectoryStateHandle;
+import org.apache.flink.runtime.state.PhysicalStateHandleID;
+import org.apache.flink.runtime.state.SharedStateRegistryKey;
+import org.apache.flink.runtime.state.StreamStateHandle;
+import org.apache.flink.util.FileUtils;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Optional;
+
+/** Wrap {@link DirectoryStateHandle} to a {@link StreamStateHandle}. */
+public class DirectoryStreamStateHandle extends DirectoryStateHandle
implements StreamStateHandle {
+
+private static final long serialVersionUID = -6453596108675892492L;
+
+public DirectoryStreamStateHandle(@Nonnull Path directory, long
directorySize) {
+super(directory, directorySize);
+}
+
+@Override
+public FSDataInputStream openInputStream() {
+throw new UnsupportedOperationException();
+}
+
+@Override
+public Optional asBytesIfInMemory() {
+return Optional.empty();
+}
+
+@Override
+public PhysicalStateHandleID getStreamStateHandleID() {
+return new PhysicalStateHandleID(getDirectory().toString());
+}
+
+@Override
+public boolean equals(Object o) {
+if (this == o) {
+return true;
+}
+if (o == null || getClass() != o.getClass()) {
+return false;
+}
+
+DirectoryStreamStateHandle that = (DirectoryStreamStateHandle) o;
+
+return getDirectory().equals(that.getDirectory());
+}
+
+@Override
+public String toString() {
+return "DirectoryStreamStateHandle{" + "directory=" + getDirectory() +
'}';
+}
+
+public static DirectoryStreamStateHandle forPathWithSize(@Nonnull Path
directory) {
+long size;
+try {
+size = FileUtils.getDirectoryFilesSize(directory);
+} catch (IOException e) {
+size = 0L;
+}
+return new DirectoryStreamStateHandle(directory, size);
+}
+
+public static SharedStateRegistryKey createStateRegistryKey(
Review Comment:
Could this be a member function ?
##
flink-runtime/src/main/java/org/apache/flink/runtime/state/filemerging/SegmentFileStateHandle.java:
##
@@ -62,19 +77,58 @@ public class SegmentFileStateHandle implements
StreamStateHandle {
* @param scope The state's scope, whether it is exclusive or shared.
*/
public SegmentFileStateHandle(
-Path filePath, long startPos, long stateSize,
CheckpointedStateScope scope) {
+Path directoryPath,
+Path filePath,
+long startPos,
+long stateSize,
+CheckpointedStateScope scope) {
this.filePath = filePath;
this.stateSize = stateSize;
this.startPos = startPos;
this.scope = scope;
+this.directoryStateHandle =
+DirectoryStreamStateHandle.forPathWithSize(
+new File(directoryPath.getPath()).toPath());
Review Comment:
+1, At least we should not calculate for every SegmentFileStateHandle in the
same directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues
[jira] [Commented] (FLINK-34898) Cannot create named STRUCT with a single field
[
https://issues.apache.org/jira/browse/FLINK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829402#comment-17829402
]
Chloe He commented on FLINK-34898:
--
[~hackergin] Thanks for the pointer.
{code:java}
SELECT CAST(ROW(1) as ROW) AS row1; {code}
works for me.
I want to also wrap this in an ARRAY, so that the data in this cell looks like
`[\{"a": 1}]` (i.e., it's an ARRAY constructed from named STRUCTs).
{code:java}
SELECT ARRAY[CAST(ROW(1) as ROW)] AS row1; {code}
does not work and it seems that the only way that I can get this to work
properly is to wrap this in two ROWs, i.e.,
{code:java}
SELECT ROW(ROW(CAST(ROW(1) as ROW))) AS row1; {code}
Is this the only way to achieve this?
> Cannot create named STRUCT with a single field
> --
>
> Key: FLINK-34898
> URL: https://issues.apache.org/jira/browse/FLINK-34898
> Project: Flink
> Issue Type: Bug
>Reporter: Chloe He
>Priority: Major
> Attachments: image-2024-03-21-12-00-00-183.png
>
>
> I'm trying to create named structs using Flink SQL and I found a previous
> ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use
> of the following syntax:
> {code:java}
> SELECT CAST(('a', 1) as ROW) AS row1;
> {code}
> However, my named struct has a single field and effectively it should look
> something like `\{"a": 1}`. I can't seem to be able to find a way to
> construct this. I have experimented with a few different syntax and it either
> throws parsing error or casting error:
> {code:java}
> Cast function cannot convert value of type INTEGER to type
> RecordType(VARCHAR(2147483647) a) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34902) INSERT INTO column mismatch leads to IndexOutOfBoundsException
Timo Walther created FLINK-34902:
Summary: INSERT INTO column mismatch leads to
IndexOutOfBoundsException
Key: FLINK-34902
URL: https://issues.apache.org/jira/browse/FLINK-34902
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Reporter: Timo Walther
SQL:
{code}
INSERT INTO t (a, b) SELECT 1;
{code}
Stack trace:
{code}
org.apache.flink.table.api.ValidationException: SQL validation failed. Index 1
out of bounds for length 1
at
org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:200)
at
org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:117)
at
Caused by: java.lang.IndexOutOfBoundsException: Index 1 out of bounds for
length 1
at
java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
at
java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
at
java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
at java.base/java.util.Objects.checkIndex(Objects.java:374)
at java.base/java.util.ArrayList.get(ArrayList.java:459)
at
org.apache.flink.table.planner.calcite.PreValidateReWriter$.$anonfun$reorder$1(PreValidateReWriter.scala:355)
at
org.apache.flink.table.planner.calcite.PreValidateReWriter$.$anonfun$reorder$1$adapted(PreValidateReWriter.scala:355)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34731][runtime] Remove SpeculativeScheduler and incorporate its features into AdaptiveBatchScheduler. [flink]
zhuzhurk commented on code in PR #24524:
URL: https://github.com/apache/flink/pull/24524#discussion_r1533346303
##
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultSchedulerBuilder.java:
##
@@ -322,39 +321,14 @@ public DefaultScheduler build() throws Exception {
}
public AdaptiveBatchScheduler buildAdaptiveBatchJobScheduler() throws
Exception {
-return new AdaptiveBatchScheduler(
-log,
-jobGraph,
-ioExecutor,
-jobMasterConfiguration,
-componentMainThreadExecutor -> {},
-delayExecutor,
-userCodeLoader,
-checkpointCleaner,
-checkpointRecoveryFactory,
-jobManagerJobMetricGroup,
-new
VertexwiseSchedulingStrategy.Factory(inputConsumableDeciderFactory),
-failoverStrategyFactory,
-restartBackoffTimeStrategy,
-executionOperations,
-executionVertexVersioner,
-executionSlotAllocatorFactory,
-System.currentTimeMillis(),
-mainThreadExecutor,
-jobStatusListener,
-failureEnrichers,
-createExecutionGraphFactory(true),
-shuffleMaster,
-rpcTimeout,
-vertexParallelismAndInputInfosDecider,
-defaultMaxParallelism,
-hybridPartitionDataConsumeConstraint,
-
ForwardGroupComputeUtil.computeForwardGroupsAndCheckParallelism(
-jobGraph.getVerticesSortedTopologicallyFromSources()));
+return buildAdaptiveBatchJobScheduler(false);
}
-public SpeculativeScheduler buildSpeculativeScheduler() throws Exception {
-return new SpeculativeScheduler(
+public AdaptiveBatchScheduler buildAdaptiveBatchJobScheduler(boolean
enableSpeculativeExecution)
+throws Exception {
+jobMasterConfiguration.set(
+BatchExecutionOptions.SPECULATIVE_ENABLED,
enableSpeculativeExecution);
+return new AdaptiveBatchScheduler(
Review Comment:
Is it possible that we use `AdaptiveBatchSchedulerFactory` to create
scheduler, so that more production code can be covered by this test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-34731) Remove SpeculativeScheduler and incorporate its features into AdaptiveBatchScheduler
[ https://issues.apache.org/jira/browse/FLINK-34731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34731: --- Labels: pull-request-available (was: ) > Remove SpeculativeScheduler and incorporate its features into > AdaptiveBatchScheduler > > > Key: FLINK-34731 > URL: https://issues.apache.org/jira/browse/FLINK-34731 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > > Presently, speculative execution is exposed to users as a feature of the > AdaptiveBatchScheduler. > To streamline our codebase and reduce maintenance overhead, this ticket will > consolidate the SpeculativeScheduler into the AdaptiveBatchScheduler, > eliminating the need for a separate SpeculativeScheduler class. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-26088][Connectors/ElasticSearch] Add Elasticsearch 8.0 support [flink-connector-elasticsearch]
drorventura commented on PR #53: URL: https://github.com/apache/flink-connector-elasticsearch/pull/53#issuecomment-2011373385 when is the next release planned? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-34903) Add mysql-pipeline-connector with table.exclude.list option to exclude unnecessary tables
shiyuyang created FLINK-34903:
-
Summary: Add mysql-pipeline-connector with table.exclude.list
option to exclude unnecessary tables
Key: FLINK-34903
URL: https://issues.apache.org/jira/browse/FLINK-34903
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Reporter: shiyuyang
Fix For: cdc-3.1.0
When using the MySQL Pipeline connector for whole-database synchronization,
users currently cannot exclude unnecessary tables. Taking reference from
Debezium's parameters, specifically the {*}table.exclude.list{*}, if the
*table.include.list* is declared, then the *table.exclude.list* parameter will
not take effect. However, the tables specified in the tables parameter of the
MySQL Pipeline connector are effectively added to the *table.include.list* in
Debezium's context.
In summary, it is necessary to introduce an externally-exposed
*table.exclude.list* parameter within the MySQL Pipeline connector to
facilitate the exclusion of tables. This is because the current setup does not
allow for excluding unnecessary tables when including others through the tables
parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34903) Add mysql-pipeline-connector with table.exclude.list option to exclude unnecessary tables
[
https://issues.apache.org/jira/browse/FLINK-34903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829418#comment-17829418
]
Thorne commented on FLINK-34903:
I will take a pr for this
> Add mysql-pipeline-connector with table.exclude.list option to exclude
> unnecessary tables
> ---
>
> Key: FLINK-34903
> URL: https://issues.apache.org/jira/browse/FLINK-34903
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
>Reporter: Thorne
>Priority: Major
> Labels: cdc
> Fix For: cdc-3.1.0
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> When using the MySQL Pipeline connector for whole-database
> synchronization, users currently cannot exclude unnecessary tables. Taking
> reference from Debezium's parameters, specifically the
> {*}table.exclude.list{*}, if the *table.include.list* is declared, then the
> *table.exclude.list* parameter will not take effect. However, the tables
> specified in the tables parameter of the MySQL Pipeline connector are
> effectively added to the *table.include.list* in Debezium's context.
> In summary, it is necessary to introduce an externally-exposed
> *table.exclude.list* parameter within the MySQL Pipeline connector to
> facilitate the exclusion of tables. This is because the current setup does
> not allow for excluding unnecessary tables when including others through the
> tables parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34903) Add mysql-pipeline-connector with table.exclude.list option to exclude unnecessary tables
[
https://issues.apache.org/jira/browse/FLINK-34903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thorne updated FLINK-34903:
---
Docs Text: (was: i will take a pr for this)
> Add mysql-pipeline-connector with table.exclude.list option to exclude
> unnecessary tables
> ---
>
> Key: FLINK-34903
> URL: https://issues.apache.org/jira/browse/FLINK-34903
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
>Reporter: Thorne
>Priority: Major
> Labels: cdc
> Fix For: cdc-3.1.0
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> When using the MySQL Pipeline connector for whole-database
> synchronization, users currently cannot exclude unnecessary tables. Taking
> reference from Debezium's parameters, specifically the
> {*}table.exclude.list{*}, if the *table.include.list* is declared, then the
> *table.exclude.list* parameter will not take effect. However, the tables
> specified in the tables parameter of the MySQL Pipeline connector are
> effectively added to the *table.include.list* in Debezium's context.
> In summary, it is necessary to introduce an externally-exposed
> *table.exclude.list* parameter within the MySQL Pipeline connector to
> facilitate the exclusion of tables. This is because the current setup does
> not allow for excluding unnecessary tables when including others through the
> tables parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34903) Add mysql-pipeline-connector with table.exclude.list option to exclude unnecessary tables
[
https://issues.apache.org/jira/browse/FLINK-34903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thorne updated FLINK-34903:
---
Attachment: screenshot-1.png
> Add mysql-pipeline-connector with table.exclude.list option to exclude
> unnecessary tables
> ---
>
> Key: FLINK-34903
> URL: https://issues.apache.org/jira/browse/FLINK-34903
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
>Reporter: Thorne
>Priority: Major
> Labels: cdc
> Fix For: cdc-3.1.0
>
> Attachments: screenshot-1.png
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> When using the MySQL Pipeline connector for whole-database
> synchronization, users currently cannot exclude unnecessary tables. Taking
> reference from Debezium's parameters, specifically the
> {*}table.exclude.list{*}, if the *table.include.list* is declared, then the
> *table.exclude.list* parameter will not take effect. However, the tables
> specified in the tables parameter of the MySQL Pipeline connector are
> effectively added to the *table.include.list* in Debezium's context.
> In summary, it is necessary to introduce an externally-exposed
> *table.exclude.list* parameter within the MySQL Pipeline connector to
> facilitate the exclusion of tables. This is because the current setup does
> not allow for excluding unnecessary tables when including others through the
> tables parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34903) Add mysql-pipeline-connector with table.exclude.list option to exclude unnecessary tables
[
https://issues.apache.org/jira/browse/FLINK-34903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thorne updated FLINK-34903:
---
Attachment: screenshot-2.png
> Add mysql-pipeline-connector with table.exclude.list option to exclude
> unnecessary tables
> ---
>
> Key: FLINK-34903
> URL: https://issues.apache.org/jira/browse/FLINK-34903
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
>Reporter: Thorne
>Priority: Major
> Labels: cdc
> Fix For: cdc-3.1.0
>
> Attachments: screenshot-1.png, screenshot-2.png
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> When using the MySQL Pipeline connector for whole-database
> synchronization, users currently cannot exclude unnecessary tables. Taking
> reference from Debezium's parameters, specifically the
> {*}table.exclude.list{*}, if the *table.include.list* is declared, then the
> *table.exclude.list* parameter will not take effect. However, the tables
> specified in the tables parameter of the MySQL Pipeline connector are
> effectively added to the *table.include.list* in Debezium's context.
> In summary, it is necessary to introduce an externally-exposed
> *table.exclude.list* parameter within the MySQL Pipeline connector to
> facilitate the exclusion of tables. This is because the current setup does
> not allow for excluding unnecessary tables when including others through the
> tables parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34903) Add mysql-pipeline-connector with table.exclude.list option to exclude unnecessary tables
[
https://issues.apache.org/jira/browse/FLINK-34903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thorne updated FLINK-34903:
---
Attachment: screenshot-3.png
> Add mysql-pipeline-connector with table.exclude.list option to exclude
> unnecessary tables
> ---
>
> Key: FLINK-34903
> URL: https://issues.apache.org/jira/browse/FLINK-34903
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
>Reporter: Thorne
>Priority: Major
> Labels: cdc
> Fix For: cdc-3.1.0
>
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> When using the MySQL Pipeline connector for whole-database
> synchronization, users currently cannot exclude unnecessary tables. Taking
> reference from Debezium's parameters, specifically the
> {*}table.exclude.list{*}, if the *table.include.list* is declared, then the
> *table.exclude.list* parameter will not take effect. However, the tables
> specified in the tables parameter of the MySQL Pipeline connector are
> effectively added to the *table.include.list* in Debezium's context.
> In summary, it is necessary to introduce an externally-exposed
> *table.exclude.list* parameter within the MySQL Pipeline connector to
> facilitate the exclusion of tables. This is because the current setup does
> not allow for excluding unnecessary tables when including others through the
> tables parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [Flink 32701] [cep] Fix CEP Operator Memory Leak Issue [flink]
dawidwys merged PR #24084: URL: https://github.com/apache/flink/pull/24084 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-32701) Potential Memory Leak in Flink CEP due to Persistent Starting States in NFAState
[
https://issues.apache.org/jira/browse/FLINK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Wysakowicz reassigned FLINK-32701:
Assignee: Puneet Duggal
> Potential Memory Leak in Flink CEP due to Persistent Starting States in
> NFAState
>
>
> Key: FLINK-32701
> URL: https://issues.apache.org/jira/browse/FLINK-32701
> Project: Flink
> Issue Type: Bug
> Components: Library / CEP
>Affects Versions: 1.17.0, 1.16.1, 1.16.2, 1.17.1
>Reporter: Puneet Duggal
>Assignee: Puneet Duggal
>Priority: Major
> Labels: CEP, auto-deprioritized-critical, cep
> Attachments: Screenshot 2023-07-26 at 11.45.06 AM.png, Screenshot
> 2023-07-26 at 11.50.28 AM.png
>
>
> Our team has encountered a potential memory leak issue while working with the
> Complex Event Processing (CEP) library in Flink v1.17.
> h2. Context
> The CEP Operator maintains a keyed state called NFAState, which holds two
> queues: one for partial matches and one for completed matches. When a key is
> first encountered, the CEP creates a starting computation state and stores it
> in the partial matches queue. As more events occur that match the defined
> conditions (e.g., a TAKE condition), additional computation states get added
> to the queue, with their specific type (normal, pending, end) depending on
> the pattern sequence.
> However, I have noticed that the starting computation state remains in the
> partial matches queue even after the pattern sequence has been completely
> matched. This is also the case for keys that have already timed out. As a
> result, the state gets stored for all keys that the CEP ever encounters,
> leading to a continual increase in the checkpoint size.
> h2. How to reproduce this
> # Pattern Sequence - A not_followed_by B within 5 mins
> # Time Characteristic - EventTime
> # StateBackend - HashMapStateBackend
> On my local machine, I started this pipeline and started sending events at
> the rate of 10 events per second (only A) and as expected after 5 mins, CEP
> started sending pattern matched output with the same rate. But the issue was
> that after every 2 mins (checkpoint interval), checkpoint size kept on
> increasing. Expectation was that after 5 mins (2-3 checkpoints), checkpoint
> size will remain constant since any window of 5 mins will consist of the same
> number of unique keys (older ones will get matched or timed out hence removed
> from state). But as you can see below attached images, checkpoint size kept
> on increasing till 40 checkpoints (around 1.5hrs).
> P.S. - After 3 checkpoints (6 mins), the checkpoint size was around 1.78MB.
> Hence assumption is that ideal checkpoint size for a 5 min window should be
> less than 1.78MB.
> As you can see after 39 checkpoints, I triggered a savepoint for this
> pipeline. After that I used a savepoint reader to investigate what all is
> getting stored in CEP states. Below code investigates NFAState of CEPOperator
> for potential memory leak.
> {code:java}
> import lombok.AllArgsConstructor;
> import lombok.Data;
> import lombok.NoArgsConstructor;
> import org.apache.flink.api.common.state.ValueState;
> import org.apache.flink.api.common.state.ValueStateDescriptor;
> import org.apache.flink.cep.nfa.NFAState;
> import org.apache.flink.cep.nfa.NFAStateSerializer;
> import org.apache.flink.configuration.Configuration;
> import org.apache.flink.runtime.state.filesystem.FsStateBackend;
> import org.apache.flink.state.api.OperatorIdentifier;
> import org.apache.flink.state.api.SavepointReader;
> import org.apache.flink.state.api.functions.KeyedStateReaderFunction;
> import org.apache.flink.streaming.api.datastream.DataStream;
> import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
> import org.apache.flink.util.Collector;
> import org.junit.jupiter.api.Test;
> import java.io.Serializable;
> import java.util.Objects;
> public class NFAStateReaderTest {
> private static final String NFA_STATE_NAME = "nfaStateName";
> @Test
> public void testNfaStateReader() throws Exception {
> StreamExecutionEnvironment environment =
> StreamExecutionEnvironment.getExecutionEnvironment();
> SavepointReader savepointReader =
> SavepointReader.read(environment,
> "file:///opt/flink/savepoints/savepoint-093404-9bc0a38654df", new
> FsStateBackend("file:///abc"));
> DataStream stream =
> savepointReader.readKeyedState(OperatorIdentifier.forUid("select_pattern_events"),
> new NFAStateReaderTest.NFAStateReaderFunction());
> stream.print();
> environment.execute();
> }
> static class NFAStateReaderFunction extends
> KeyedStateReaderFunction {
> private ValueState computationStates;
[jira] [Closed] (FLINK-32701) Potential Memory Leak in Flink CEP due to Persistent Starting States in NFAState
[
https://issues.apache.org/jira/browse/FLINK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Wysakowicz closed FLINK-32701.
Fix Version/s: 1.20.0
Resolution: Fixed
Fixed in a9cde49118bab4b32b2d1ae1f97beb94eb967f9b
> Potential Memory Leak in Flink CEP due to Persistent Starting States in
> NFAState
>
>
> Key: FLINK-32701
> URL: https://issues.apache.org/jira/browse/FLINK-32701
> Project: Flink
> Issue Type: Bug
> Components: Library / CEP
>Affects Versions: 1.17.0, 1.16.1, 1.16.2, 1.17.1
>Reporter: Puneet Duggal
>Assignee: Puneet Duggal
>Priority: Major
> Labels: CEP, auto-deprioritized-critical, cep
> Fix For: 1.20.0
>
> Attachments: Screenshot 2023-07-26 at 11.45.06 AM.png, Screenshot
> 2023-07-26 at 11.50.28 AM.png
>
>
> Our team has encountered a potential memory leak issue while working with the
> Complex Event Processing (CEP) library in Flink v1.17.
> h2. Context
> The CEP Operator maintains a keyed state called NFAState, which holds two
> queues: one for partial matches and one for completed matches. When a key is
> first encountered, the CEP creates a starting computation state and stores it
> in the partial matches queue. As more events occur that match the defined
> conditions (e.g., a TAKE condition), additional computation states get added
> to the queue, with their specific type (normal, pending, end) depending on
> the pattern sequence.
> However, I have noticed that the starting computation state remains in the
> partial matches queue even after the pattern sequence has been completely
> matched. This is also the case for keys that have already timed out. As a
> result, the state gets stored for all keys that the CEP ever encounters,
> leading to a continual increase in the checkpoint size.
> h2. How to reproduce this
> # Pattern Sequence - A not_followed_by B within 5 mins
> # Time Characteristic - EventTime
> # StateBackend - HashMapStateBackend
> On my local machine, I started this pipeline and started sending events at
> the rate of 10 events per second (only A) and as expected after 5 mins, CEP
> started sending pattern matched output with the same rate. But the issue was
> that after every 2 mins (checkpoint interval), checkpoint size kept on
> increasing. Expectation was that after 5 mins (2-3 checkpoints), checkpoint
> size will remain constant since any window of 5 mins will consist of the same
> number of unique keys (older ones will get matched or timed out hence removed
> from state). But as you can see below attached images, checkpoint size kept
> on increasing till 40 checkpoints (around 1.5hrs).
> P.S. - After 3 checkpoints (6 mins), the checkpoint size was around 1.78MB.
> Hence assumption is that ideal checkpoint size for a 5 min window should be
> less than 1.78MB.
> As you can see after 39 checkpoints, I triggered a savepoint for this
> pipeline. After that I used a savepoint reader to investigate what all is
> getting stored in CEP states. Below code investigates NFAState of CEPOperator
> for potential memory leak.
> {code:java}
> import lombok.AllArgsConstructor;
> import lombok.Data;
> import lombok.NoArgsConstructor;
> import org.apache.flink.api.common.state.ValueState;
> import org.apache.flink.api.common.state.ValueStateDescriptor;
> import org.apache.flink.cep.nfa.NFAState;
> import org.apache.flink.cep.nfa.NFAStateSerializer;
> import org.apache.flink.configuration.Configuration;
> import org.apache.flink.runtime.state.filesystem.FsStateBackend;
> import org.apache.flink.state.api.OperatorIdentifier;
> import org.apache.flink.state.api.SavepointReader;
> import org.apache.flink.state.api.functions.KeyedStateReaderFunction;
> import org.apache.flink.streaming.api.datastream.DataStream;
> import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
> import org.apache.flink.util.Collector;
> import org.junit.jupiter.api.Test;
> import java.io.Serializable;
> import java.util.Objects;
> public class NFAStateReaderTest {
> private static final String NFA_STATE_NAME = "nfaStateName";
> @Test
> public void testNfaStateReader() throws Exception {
> StreamExecutionEnvironment environment =
> StreamExecutionEnvironment.getExecutionEnvironment();
> SavepointReader savepointReader =
> SavepointReader.read(environment,
> "file:///opt/flink/savepoints/savepoint-093404-9bc0a38654df", new
> FsStateBackend("file:///abc"));
> DataStream stream =
> savepointReader.readKeyedState(OperatorIdentifier.forUid("select_pattern_events"),
> new NFAStateReaderTest.NFAStateReaderFunction());
> stream.print();
> environment.execute();
> }
> static class NFAStateRea
[jira] [Created] (FLINK-34904) [Feature] submit Flink CDC pipeline job to yarn application cluster.
ZhengYu Chen created FLINK-34904: Summary: [Feature] submit Flink CDC pipeline job to yarn application cluster. Key: FLINK-34904 URL: https://issues.apache.org/jira/browse/FLINK-34904 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: 3.1.0 Reporter: ZhengYu Chen Fix For: 3.1.0 support flink cdc cli submit pipeline job to yarn application cluster.discuss in FLINK-34853 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34904) [Feature] submit Flink CDC pipeline job to yarn application cluster.
[ https://issues.apache.org/jira/browse/FLINK-34904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829434#comment-17829434 ] ZhengYu Chen commented on FLINK-34904: -- cc [~czy006] > [Feature] submit Flink CDC pipeline job to yarn application cluster. > > > Key: FLINK-34904 > URL: https://issues.apache.org/jira/browse/FLINK-34904 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: 3.1.0 >Reporter: ZhengYu Chen >Priority: Minor > Fix For: 3.1.0 > > > support flink cdc cli submit pipeline job to yarn application cluster.discuss > in FLINK-34853 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-26088][Connectors/ElasticSearch] Add Elasticsearch 8.0 support [flink-connector-elasticsearch]
reswqa commented on PR #53: URL: https://github.com/apache/flink-connector-elasticsearch/pull/53#issuecomment-2011649582 > when is the next release planned? TBH, I'm not really sure. But I think we will probably release a series of connectors that supporting flink-1.19 in the near future. You can make a release request on the flink mailing list. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34892][ci] Fix python test failure due to config file change [flink-connector-aws]
dannycranmer merged PR #133: URL: https://github.com/apache/flink-connector-aws/pull/133 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34892) Nightly AWS connectors build fails on running python tests
[
https://issues.apache.org/jira/browse/FLINK-34892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829440#comment-17829440
]
Danny Cranmer commented on FLINK-34892:
---
Merged commit
[{{e8ba71e}}|https://github.com/apache/flink-connector-aws/commit/e8ba71ec3c27903c838701d536a8ae05bc5bb523]
into apache:main
> Nightly AWS connectors build fails on running python tests
> --
>
> Key: FLINK-34892
> URL: https://issues.apache.org/jira/browse/FLINK-34892
> Project: Flink
> Issue Type: Bug
> Components: Connectors / AWS
>Affects Versions: aws-connector-4.2.0
>Reporter: Aleksandr Pilipenko
>Priority: Major
> Labels: pull-request-available
>
> Build for externalized python connector code fails:
> [https://github.com/apache/flink-connector-aws/actions/runs/8351768294/job/22860710449]
> {code:java}
> 2024-03-20T00:14:35.5215863Z __
> FlinkKinesisTest.test_kinesis_streams_sink __
> 2024-03-20T00:14:35.5216781Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/testing/test_case_utils.py:149:
> in setUp
> 2024-03-20T00:14:35.5217584Z self.env =
> StreamExecutionEnvironment.get_execution_environment()
> 2024-03-20T00:14:35.5218901Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/datastream/stream_execution_environment.py:876:
> in get_execution_environment
> 2024-03-20T00:14:35.5219751Z gateway = get_gateway()
> 2024-03-20T00:14:35.5220635Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/java_gateway.py:64: in
> get_gateway
> 2024-03-20T00:14:35.5221378Z _gateway = launch_gateway()
> 2024-03-20T00:14:35.5222111Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/java_gateway.py:110:
> in launch_gateway
> 2024-03-20T00:14:35.5222956Z p = launch_gateway_server_process(env, args)
> 2024-03-20T00:14:35.5223854Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/pyflink_gateway_server.py:262:
> in launch_gateway_server_process
> 2024-03-20T00:14:35.5224649Z java_executable = find_java_executable()
> 2024-03-20T00:14:35.5225583Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/pyflink_gateway_server.py:75:
> in find_java_executable
> 2024-03-20T00:14:35.5226449Z java_home =
> read_from_config(KEY_ENV_JAVA_HOME, None, flink_conf_file)
> 2024-03-20T00:14:35.5227099Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> 2024-03-20T00:14:35.5227450Z
> 2024-03-20T00:14:35.5227774Z key = 'env.java.home', default_value = None
> 2024-03-20T00:14:35.5228925Z flink_conf_file =
> '/home/runner/work/flink-connector-aws/flink-connector-aws/flink-python/.tox/py310-cython/lib/python3.10/site-packages/pyflink/conf/flink-conf.yaml'
> 2024-03-20T00:14:35.5229778Z
> 2024-03-20T00:14:35.5230010Z def read_from_config(key, default_value,
> flink_conf_file):
> 2024-03-20T00:14:35.5230581Z value = default_value
> 2024-03-20T00:14:35.5231236Z # get the realpath of tainted path value
> to avoid CWE22 problem that constructs a path or URI
> 2024-03-20T00:14:35.5232195Z # using the tainted value and might
> allow an attacker to access, modify, or test the existence
> 2024-03-20T00:14:35.5232940Z # of critical or sensitive files.
> 2024-03-20T00:14:35.5233417Z > with
> open(os.path.realpath(flink_conf_file), "r") as f:
> 2024-03-20T00:14:35.5234874Z E FileNotFoundError: [Errno 2] No such
> file or directory:
> '/home/runner/work/flink-connector-aws/flink-connector-aws/flink-python/.tox/py310-cython/lib/python3.10/site-packages/pyflink/conf/flink-conf.yaml'
> 2024-03-20T00:14:35.5235954Z
> 2024-03-20T00:14:35.5236484Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/pyflink_gateway_server.py:58:
> FileNotFoundError {code}
> Failure started after the release of apache-flink python package for 1.19.0
> due to change of default config file provided within artifact.
>
>
> Issue comes from outdated copy of pyflink_gateway_server.py created as part
> of [https://github.com/apache/flink-connector-kafka/pull/69] (same change is
> duplicated in AWS connectors repository).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-34892) Nightly AWS connectors build fails on running python tests
[
https://issues.apache.org/jira/browse/FLINK-34892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Cranmer reassigned FLINK-34892:
-
Assignee: Aleksandr Pilipenko
> Nightly AWS connectors build fails on running python tests
> --
>
> Key: FLINK-34892
> URL: https://issues.apache.org/jira/browse/FLINK-34892
> Project: Flink
> Issue Type: Bug
> Components: Connectors / AWS
>Affects Versions: aws-connector-4.2.0
>Reporter: Aleksandr Pilipenko
>Assignee: Aleksandr Pilipenko
>Priority: Major
> Labels: pull-request-available
>
> Build for externalized python connector code fails:
> [https://github.com/apache/flink-connector-aws/actions/runs/8351768294/job/22860710449]
> {code:java}
> 2024-03-20T00:14:35.5215863Z __
> FlinkKinesisTest.test_kinesis_streams_sink __
> 2024-03-20T00:14:35.5216781Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/testing/test_case_utils.py:149:
> in setUp
> 2024-03-20T00:14:35.5217584Z self.env =
> StreamExecutionEnvironment.get_execution_environment()
> 2024-03-20T00:14:35.5218901Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/datastream/stream_execution_environment.py:876:
> in get_execution_environment
> 2024-03-20T00:14:35.5219751Z gateway = get_gateway()
> 2024-03-20T00:14:35.5220635Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/java_gateway.py:64: in
> get_gateway
> 2024-03-20T00:14:35.5221378Z _gateway = launch_gateway()
> 2024-03-20T00:14:35.5222111Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/java_gateway.py:110:
> in launch_gateway
> 2024-03-20T00:14:35.5222956Z p = launch_gateway_server_process(env, args)
> 2024-03-20T00:14:35.5223854Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/pyflink_gateway_server.py:262:
> in launch_gateway_server_process
> 2024-03-20T00:14:35.5224649Z java_executable = find_java_executable()
> 2024-03-20T00:14:35.5225583Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/pyflink_gateway_server.py:75:
> in find_java_executable
> 2024-03-20T00:14:35.5226449Z java_home =
> read_from_config(KEY_ENV_JAVA_HOME, None, flink_conf_file)
> 2024-03-20T00:14:35.5227099Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> 2024-03-20T00:14:35.5227450Z
> 2024-03-20T00:14:35.5227774Z key = 'env.java.home', default_value = None
> 2024-03-20T00:14:35.5228925Z flink_conf_file =
> '/home/runner/work/flink-connector-aws/flink-connector-aws/flink-python/.tox/py310-cython/lib/python3.10/site-packages/pyflink/conf/flink-conf.yaml'
> 2024-03-20T00:14:35.5229778Z
> 2024-03-20T00:14:35.5230010Z def read_from_config(key, default_value,
> flink_conf_file):
> 2024-03-20T00:14:35.5230581Z value = default_value
> 2024-03-20T00:14:35.5231236Z # get the realpath of tainted path value
> to avoid CWE22 problem that constructs a path or URI
> 2024-03-20T00:14:35.5232195Z # using the tainted value and might
> allow an attacker to access, modify, or test the existence
> 2024-03-20T00:14:35.5232940Z # of critical or sensitive files.
> 2024-03-20T00:14:35.5233417Z > with
> open(os.path.realpath(flink_conf_file), "r") as f:
> 2024-03-20T00:14:35.5234874Z E FileNotFoundError: [Errno 2] No such
> file or directory:
> '/home/runner/work/flink-connector-aws/flink-connector-aws/flink-python/.tox/py310-cython/lib/python3.10/site-packages/pyflink/conf/flink-conf.yaml'
> 2024-03-20T00:14:35.5235954Z
> 2024-03-20T00:14:35.5236484Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/pyflink_gateway_server.py:58:
> FileNotFoundError {code}
> Failure started after the release of apache-flink python package for 1.19.0
> due to change of default config file provided within artifact.
>
>
> Issue comes from outdated copy of pyflink_gateway_server.py created as part
> of [https://github.com/apache/flink-connector-kafka/pull/69] (same change is
> duplicated in AWS connectors repository).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-34892) Nightly AWS connectors build fails on running python tests
[
https://issues.apache.org/jira/browse/FLINK-34892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Cranmer resolved FLINK-34892.
---
Resolution: Fixed
> Nightly AWS connectors build fails on running python tests
> --
>
> Key: FLINK-34892
> URL: https://issues.apache.org/jira/browse/FLINK-34892
> Project: Flink
> Issue Type: Bug
> Components: Connectors / AWS
>Affects Versions: aws-connector-4.2.0
>Reporter: Aleksandr Pilipenko
>Assignee: Aleksandr Pilipenko
>Priority: Major
> Labels: pull-request-available
>
> Build for externalized python connector code fails:
> [https://github.com/apache/flink-connector-aws/actions/runs/8351768294/job/22860710449]
> {code:java}
> 2024-03-20T00:14:35.5215863Z __
> FlinkKinesisTest.test_kinesis_streams_sink __
> 2024-03-20T00:14:35.5216781Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/testing/test_case_utils.py:149:
> in setUp
> 2024-03-20T00:14:35.5217584Z self.env =
> StreamExecutionEnvironment.get_execution_environment()
> 2024-03-20T00:14:35.5218901Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/datastream/stream_execution_environment.py:876:
> in get_execution_environment
> 2024-03-20T00:14:35.5219751Z gateway = get_gateway()
> 2024-03-20T00:14:35.5220635Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/java_gateway.py:64: in
> get_gateway
> 2024-03-20T00:14:35.5221378Z _gateway = launch_gateway()
> 2024-03-20T00:14:35.5222111Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/java_gateway.py:110:
> in launch_gateway
> 2024-03-20T00:14:35.5222956Z p = launch_gateway_server_process(env, args)
> 2024-03-20T00:14:35.5223854Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/pyflink_gateway_server.py:262:
> in launch_gateway_server_process
> 2024-03-20T00:14:35.5224649Z java_executable = find_java_executable()
> 2024-03-20T00:14:35.5225583Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/pyflink_gateway_server.py:75:
> in find_java_executable
> 2024-03-20T00:14:35.5226449Z java_home =
> read_from_config(KEY_ENV_JAVA_HOME, None, flink_conf_file)
> 2024-03-20T00:14:35.5227099Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> 2024-03-20T00:14:35.5227450Z
> 2024-03-20T00:14:35.5227774Z key = 'env.java.home', default_value = None
> 2024-03-20T00:14:35.5228925Z flink_conf_file =
> '/home/runner/work/flink-connector-aws/flink-connector-aws/flink-python/.tox/py310-cython/lib/python3.10/site-packages/pyflink/conf/flink-conf.yaml'
> 2024-03-20T00:14:35.5229778Z
> 2024-03-20T00:14:35.5230010Z def read_from_config(key, default_value,
> flink_conf_file):
> 2024-03-20T00:14:35.5230581Z value = default_value
> 2024-03-20T00:14:35.5231236Z # get the realpath of tainted path value
> to avoid CWE22 problem that constructs a path or URI
> 2024-03-20T00:14:35.5232195Z # using the tainted value and might
> allow an attacker to access, modify, or test the existence
> 2024-03-20T00:14:35.5232940Z # of critical or sensitive files.
> 2024-03-20T00:14:35.5233417Z > with
> open(os.path.realpath(flink_conf_file), "r") as f:
> 2024-03-20T00:14:35.5234874Z E FileNotFoundError: [Errno 2] No such
> file or directory:
> '/home/runner/work/flink-connector-aws/flink-connector-aws/flink-python/.tox/py310-cython/lib/python3.10/site-packages/pyflink/conf/flink-conf.yaml'
> 2024-03-20T00:14:35.5235954Z
> 2024-03-20T00:14:35.5236484Z
> .tox/py310-cython/lib/python3.10/site-packages/pyflink/pyflink_gateway_server.py:58:
> FileNotFoundError {code}
> Failure started after the release of apache-flink python package for 1.19.0
> due to change of default config file provided within artifact.
>
>
> Issue comes from outdated copy of pyflink_gateway_server.py created as part
> of [https://github.com/apache/flink-connector-kafka/pull/69] (same change is
> duplicated in AWS connectors repository).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34905) The default length of CHAR/BINARY data type of Add column DDL
Qishang Zhong created FLINK-34905:
-
Summary: The default length of CHAR/BINARY data type of Add column
DDL
Key: FLINK-34905
URL: https://issues.apache.org/jira/browse/FLINK-34905
Project: Flink
Issue Type: Bug
Components: Flink CDC
Reporter: Qishang Zhong
I run the DDL in mysql
{code:java}
ALTER TABLE test.products ADD Column1 BINARY NULL;
ALTER TABLE test.products ADD Column2 CHAR NULL; {code}
Encountered the follow error:
{code:java}
Caused by: java.lang.IllegalArgumentException: Binary string length must be
between 1 and 2147483647 (both inclusive).
at
org.apache.flink.cdc.common.types.BinaryType.(BinaryType.java:53)
at
org.apache.flink.cdc.common.types.BinaryType.(BinaryType.java:61)
at org.apache.flink.cdc.common.types.DataTypes.BINARY(DataTypes.java:42)
at
org.apache.flink.cdc.connectors.mysql.utils.MySqlTypeUtils.convertFromColumn(MySqlTypeUtils.java:221)
at
org.apache.flink.cdc.connectors.mysql.utils.MySqlTypeUtils.fromDbzColumn(MySqlTypeUtils.java:111)
at
org.apache.flink.cdc.connectors.mysql.source.parser.CustomAlterTableParserListener.toCdcColumn(CustomAlterTableParserListener.java:256)
at
org.apache.flink.cdc.connectors.mysql.source.parser.CustomAlterTableParserListener.lambda$exitAlterByAddColumn$0(CustomAlterTableParserListener.java:126)
at
io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.runIfNotNull(MySqlAntlrDdlParser.java:358)
at
org.apache.flink.cdc.connectors.mysql.source.parser.CustomAlterTableParserListener.exitAlterByAddColumn(CustomAlterTableParserListener.java:98)
at
io.debezium.ddl.parser.mysql.generated.MySqlParser$AlterByAddColumnContext.exitRule(MySqlParser.java:15459)
at
io.debezium.antlr.ProxyParseTreeListenerUtil.delegateExitRule(ProxyParseTreeListenerUtil.java:64)
at
org.apache.flink.cdc.connectors.mysql.source.parser.CustomMySqlAntlrDdlParserListener.exitEveryRule(CustomMySqlAntlrDdlParserListener.java:124)
at
org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:48)
at
org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:30)
at
org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
at
org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
at
org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
at
org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
at
org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
at io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:87)
at
org.apache.flink.cdc.connectors.mysql.source.MySqlEventDeserializer.deserializeSchemaChangeRecord(MySqlEventDeserializer.java:88)
at
org.apache.flink.cdc.debezium.event.SourceRecordEventDeserializer.deserialize(SourceRecordEventDeserializer.java:52)
at
org.apache.flink.cdc.debezium.event.DebeziumEventDeserializationSchema.deserialize(DebeziumEventDeserializationSchema.java:93)
at
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitElement(MySqlRecordEmitter.java:119)
at
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.processElement(MySqlRecordEmitter.java:96)
at
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlPipelineRecordEmitter.processElement(MySqlPipelineRecordEmitter.java:120)
at
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitRecord(MySqlRecordEmitter.java:73)
at
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitRecord(MySqlRecordEmitter.java:46)
at
org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:160)
at
org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:419)
at
org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:562)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:858)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:807)
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953)
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:932)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746)
at org.apache.flink.runtime.taskmanager.Task
Re: [PR] [FLINK-34731][runtime] Remove SpeculativeScheduler and incorporate its features into AdaptiveBatchScheduler. [flink]
zhuzhurk commented on code in PR #24524:
URL: https://github.com/apache/flink/pull/24524#discussion_r1533349277
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java:
##
@@ -183,21 +193,59 @@ public AdaptiveBatchScheduler(
this.hybridPartitionDataConsumeConstraint =
hybridPartitionDataConsumeConstraint;
this.sourceParallelismFuturesByJobVertexId = new HashMap<>();
+
+//
+// Speculative execution handler
+//
Review Comment:
I would avoid the `// =` comments.
It's also better to introduce a `createSpeculativeExecutionHandler` method.
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultSpeculativeExecutionHandler.java:
##
@@ -7,62 +7,37 @@
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
- * http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
*
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
Review Comment:
Is the change to the license format expected?
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java:
##
@@ -183,21 +193,59 @@ public AdaptiveBatchScheduler(
this.hybridPartitionDataConsumeConstraint =
hybridPartitionDataConsumeConstraint;
this.sourceParallelismFuturesByJobVertexId = new HashMap<>();
+
+//
+// Speculative execution handler
+//
+if
(jobMasterConfiguration.get(BatchExecutionOptions.SPECULATIVE_ENABLED)) {
+speculativeExecutionHandler =
+new DefaultSpeculativeExecutionHandler(
+jobMasterConfiguration.get(
+
BatchExecutionOptions.SPECULATIVE_MAX_CONCURRENT_EXECUTIONS),
+jobMasterConfiguration.get(
+
BatchExecutionOptions.BLOCK_SLOW_NODE_DURATION),
+blocklistOperations,
+new
ExecutionTimeBasedSlowTaskDetector(jobMasterConfiguration),
+new SimpleCounter(),
+this::getExecutionVertex,
+() ->
getExecutionGraph().getRegisteredExecutions(),
+(newSpeculativeExecutions, verticesToDeploy) ->
+executionDeployer.allocateSlotsAndDeploy(
+newSpeculativeExecutions,
+
executionVertexVersioner.getExecutionVertexVersions(
+verticesToDeploy)),
+log);
+} else {
+speculativeExecutionHandler = new
DummySpeculativeExecutionHandler();
+}
}
@Override
protected void startSchedulingInternal() {
+speculativeExecutionHandler.registerMetrics(jobManagerJobMetricGroup);
+
tryComputeSourceParallelismThenRunAsync(
(Void value, Throwable throwable) -> {
if (getExecutionGraph().getState() == JobStatus.CREATED) {
initializeVerticesIfPossible();
super.startSchedulingInternal();
}
});
+
+speculativeExecutionHandler.startSlowTaskDetector(
Review Comment:
I think we can merge the two methods `startSlowTaskDetector` and
`registerMetrics` to one method `speculativeExecutionHandler.init()` .
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java:
##
@@ -183,21 +193,59 @@ public AdaptiveBatchScheduler(
this.hybridPartitionDataConsumeConstraint =
hybridPartitionDataConsumeConstraint;
this.sourceParallelismFuturesByJobVertexId = new HashMap<>();
+
+//
=
[jira] [Commented] (FLINK-34898) Cannot create named STRUCT with a single field
[
https://issues.apache.org/jira/browse/FLINK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829454#comment-17829454
]
Martijn Visser commented on FLINK-34898:
It does look like this is a user support ticket, not an actual bug problem.
Questions like these should be posted on the User mailing list, Slack or
Stackoverflow.
> Cannot create named STRUCT with a single field
> --
>
> Key: FLINK-34898
> URL: https://issues.apache.org/jira/browse/FLINK-34898
> Project: Flink
> Issue Type: Bug
>Reporter: Chloe He
>Priority: Major
> Attachments: image-2024-03-21-12-00-00-183.png
>
>
> I'm trying to create named structs using Flink SQL and I found a previous
> ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use
> of the following syntax:
> {code:java}
> SELECT CAST(('a', 1) as ROW) AS row1;
> {code}
> However, my named struct has a single field and effectively it should look
> something like `\{"a": 1}`. I can't seem to be able to find a way to
> construct this. I have experimented with a few different syntax and it either
> throws parsing error or casting error:
> {code:java}
> Cast function cannot convert value of type INTEGER to type
> RecordType(VARCHAR(2147483647) a) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34906) Don't start autoscaling when some tasks are not running
Rui Fan created FLINK-34906: --- Summary: Don't start autoscaling when some tasks are not running Key: FLINK-34906 URL: https://issues.apache.org/jira/browse/FLINK-34906 Project: Flink Issue Type: Improvement Components: Autoscaler Reporter: Rui Fan Assignee: Rui Fan Fix For: 1.9.0 Attachments: image-2024-03-21-17-40-23-523.png Currently, the autoscaler will scale a job when the JobStatus is RUNNING. But the JobStatus will be RUNNING once job starts schedule, so it doesn't mean all tasks are running. Especially, when the resource isn't enough or job recovers from large state. The autoscaler will throw exception and generate the AutoscalerError event when tasks are not ready, such as: !image-2024-03-21-17-40-23-523.png! Solution: we only scale job that all tasks are running(some of tasks may be finished). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34751) RestClusterClient APIs doesn't work with running Flink application on YARN
[
https://issues.apache.org/jira/browse/FLINK-34751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-34751:
---
Component/s: Deployment / YARN
> RestClusterClient APIs doesn't work with running Flink application on YARN
> --
>
> Key: FLINK-34751
> URL: https://issues.apache.org/jira/browse/FLINK-34751
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Reporter: Venkata krishnan Sowrirajan
>Priority: Major
>
> Apache YARN uses web proxy in Resource Manager to expose the endpoints
> available through the AM process (in this case RestServerEndpoint that run as
> part of AM). Note: this is in the context of running Flink cluster in YARN
> application mode.
> For eg: in the case of RestClusterClient#listJobs -
> {{Standalone listJobs}} makes the request as -
> {{{}https://:/v1/{}}}{{{}jobs{}}}{{{}/overview{}}}
> YARN the same request has to be proxified as -
> {{{}https://:/proxy//v1/{}}}{{{}jobs{}}}{{{}/overview?proxyapproved=true{}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34044] Copy dynamic table options before mapping deprecated configs [flink-connector-aws]
vahmed-hamdy commented on code in PR #132:
URL:
https://github.com/apache/flink-connector-aws/pull/132#discussion_r1533542650
##
flink-connector-aws/flink-connector-aws-kinesis-streams/src/main/java/org/apache/flink/connector/kinesis/table/util/KinesisStreamsConnectorOptionsUtils.java:
##
@@ -148,13 +149,13 @@ public static class KinesisProducerOptionsMapper {
public KinesisProducerOptionsMapper(
ReadableConfig tableOptions, Map
resolvedOptions) {
this.tableOptions = tableOptions;
-this.resolvedOptions = resolvedOptions;
+this.resolvedOptions = new HashMap<>(resolvedOptions);
}
@VisibleForTesting
public KinesisProducerOptionsMapper(Map allOptions) {
Review Comment:
The constructor is used differently, the one with `allOPtions` should
contain both `tableOptions` and `resolvedOptions`. you can see it is parsed
into both in this constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-34906) Don't start autoscaling when some tasks are not running
[ https://issues.apache.org/jira/browse/FLINK-34906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-34906: Description: Currently, the autoscaler will scale a job when the JobStatus is RUNNING. But the JobStatus will be RUNNING once job starts schedule, so it doesn't mean all tasks are running. Especially, when the resource isn't enough or job recovers from large state. The autoscaler will throw exception and generate the AutoscalerError event when tasks are not ready, such as: !image-2024-03-21-17-40-23-523.png! Also, we don't need to scale it when some tasks are not ready. Solution: we only scale job that all tasks are running(some of tasks may be finished). was: Currently, the autoscaler will scale a job when the JobStatus is RUNNING. But the JobStatus will be RUNNING once job starts schedule, so it doesn't mean all tasks are running. Especially, when the resource isn't enough or job recovers from large state. The autoscaler will throw exception and generate the AutoscalerError event when tasks are not ready, such as: !image-2024-03-21-17-40-23-523.png! Solution: we only scale job that all tasks are running(some of tasks may be finished). > Don't start autoscaling when some tasks are not running > --- > > Key: FLINK-34906 > URL: https://issues.apache.org/jira/browse/FLINK-34906 > Project: Flink > Issue Type: Improvement > Components: Autoscaler >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > Fix For: 1.9.0 > > Attachments: image-2024-03-21-17-40-23-523.png > > > Currently, the autoscaler will scale a job when the JobStatus is RUNNING. But > the JobStatus will be RUNNING once job starts schedule, so it doesn't mean > all tasks are running. Especially, when the resource isn't enough or job > recovers from large state. > The autoscaler will throw exception and generate the AutoscalerError event > when tasks are not ready, such as: > !image-2024-03-21-17-40-23-523.png! > Also, we don't need to scale it when some tasks are not ready. > Solution: we only scale job that all tasks are running(some of tasks may be > finished). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-34906] Only scale when all tasks are running [flink-kubernetes-operator]
1996fanrui opened a new pull request, #801: URL: https://github.com/apache/flink-kubernetes-operator/pull/801 ## What is the purpose of the change Currently, the autoscaler will scale a job when the JobStatus is RUNNING. But the JobStatus will be RUNNING once job starts schedule, so it doesn't mean all tasks are running. Especially, when the resource isn't enough or job recovers from large state. The autoscaler will throw exception and generate the AutoscalerError event when tasks are not ready. Also, we don't need to scale it when some tasks are not ready. ## Brief change log - [FLINK-34906] Only scale when all tasks are running - Solution: we only scale job that all tasks are running(some of tasks may be finished). We can know how many tasks are running from `JobDetailsInfo`:  ## Verifying this change Manually test is done, unit test is still writing. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changes to the `CustomResourceDescriptors`: no - Core observer or reconciler logic that is regularly executed: no ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34906) Don't start autoscaling when some tasks are not running
[ https://issues.apache.org/jira/browse/FLINK-34906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34906: --- Labels: pull-request-available (was: ) > Don't start autoscaling when some tasks are not running > --- > > Key: FLINK-34906 > URL: https://issues.apache.org/jira/browse/FLINK-34906 > Project: Flink > Issue Type: Improvement > Components: Autoscaler >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > Labels: pull-request-available > Fix For: 1.9.0 > > Attachments: image-2024-03-21-17-40-23-523.png > > > Currently, the autoscaler will scale a job when the JobStatus is RUNNING. But > the JobStatus will be RUNNING once job starts schedule, so it doesn't mean > all tasks are running. Especially, when the resource isn't enough or job > recovers from large state. > The autoscaler will throw exception and generate the AutoscalerError event > when tasks are not ready, such as: > !image-2024-03-21-17-40-23-523.png! > Also, we don't need to scale it when some tasks are not ready. > Solution: we only scale job that all tasks are running(some of tasks may be > finished). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34746) Switching to the Apache CDN for Dockerfile
[ https://issues.apache.org/jira/browse/FLINK-34746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-34746: -- Assignee: Martijn Visser > Switching to the Apache CDN for Dockerfile > -- > > Key: FLINK-34746 > URL: https://issues.apache.org/jira/browse/FLINK-34746 > Project: Flink > Issue Type: Improvement > Components: flink-docker >Reporter: lincoln lee >Assignee: Martijn Visser >Priority: Major > Fix For: 1.18.2, 1.20.0, 1.19.1 > > > During publishing the official image, we received some comments > for Switching to the Apache CDN > > See > https://github.com/docker-library/official-images/pull/16114 > https://github.com/docker-library/official-images/pull/16430 > > Reason for switching: [https://apache.org/history/mirror-history.html] (also > [https://www.apache.org/dyn/closer.cgi] and [https://www.apache.org/mirrors]) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34746) Switching to the Apache CDN for Dockerfile
[ https://issues.apache.org/jira/browse/FLINK-34746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34746: --- Labels: pull-request-available (was: ) > Switching to the Apache CDN for Dockerfile > -- > > Key: FLINK-34746 > URL: https://issues.apache.org/jira/browse/FLINK-34746 > Project: Flink > Issue Type: Improvement > Components: flink-docker >Reporter: lincoln lee >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: 1.18.2, 1.20.0, 1.19.1 > > > During publishing the official image, we received some comments > for Switching to the Apache CDN > > See > https://github.com/docker-library/official-images/pull/16114 > https://github.com/docker-library/official-images/pull/16430 > > Reason for switching: [https://apache.org/history/mirror-history.html] (also > [https://www.apache.org/dyn/closer.cgi] and [https://www.apache.org/mirrors]) -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34906] Only scale when all tasks are running [flink-kubernetes-operator]
gyfora commented on PR #801: URL: https://github.com/apache/flink-kubernetes-operator/pull/801#issuecomment-2011816958 This issue only affects the standalone autoscaler as the kubernetes operator has this logic already in place for setting the RUNNING state. Can we somehow deduplicate this logic? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34906) Don't start autoscaling when some tasks are not running
[ https://issues.apache.org/jira/browse/FLINK-34906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-34906: Fix Version/s: kubernetes-operator-1.9.0 (was: 1.9.0) > Don't start autoscaling when some tasks are not running > --- > > Key: FLINK-34906 > URL: https://issues.apache.org/jira/browse/FLINK-34906 > Project: Flink > Issue Type: Improvement > Components: Autoscaler >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.9.0 > > Attachments: image-2024-03-21-17-40-23-523.png > > > Currently, the autoscaler will scale a job when the JobStatus is RUNNING. But > the JobStatus will be RUNNING once job starts schedule, so it doesn't mean > all tasks are running. Especially, when the resource isn't enough or job > recovers from large state. > The autoscaler will throw exception and generate the AutoscalerError event > when tasks are not ready, such as: > !image-2024-03-21-17-40-23-523.png! > Also, we don't need to scale it when some tasks are not ready. > Solution: we only scale job that all tasks are running(some of tasks may be > finished). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34643) JobIDLoggingITCase failed
[
https://issues.apache.org/jira/browse/FLINK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829466#comment-17829466
]
Ryan Skraba commented on FLINK-34643:
-
*
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58455&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8349]
*
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58455&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7898]
> JobIDLoggingITCase failed
> -
>
> Key: FLINK-34643
> URL: https://issues.apache.org/jira/browse/FLINK-34643
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7897
> {code}
> Mar 09 01:24:23 01:24:23.498 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 4.209 s <<< FAILURE! -- in
> org.apache.flink.test.misc.JobIDLoggingITCase
> Mar 09 01:24:23 01:24:23.498 [ERROR]
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(ClusterClient)
> -- Time elapsed: 1.459 s <<< ERROR!
> Mar 09 01:24:23 java.lang.IllegalStateException: Too few log events recorded
> for org.apache.flink.runtime.jobmaster.JobMaster (12) - this must be a bug in
> the test code
> Mar 09 01:24:23 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.assertJobIDPresent(JobIDLoggingITCase.java:148)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(JobIDLoggingITCase.java:132)
> Mar 09 01:24:23 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 09 01:24:23 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Mar 09 01:24:23
> {code}
> The other test failures of this build were also caused by the same test:
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8349
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3&t=712ade8c-ca16-5b76-3acd-14df33bc1cb1&l=8209
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34907) jobRunningTs should be the timestamp that all tasks are running
Rui Fan created FLINK-34907: --- Summary: jobRunningTs should be the timestamp that all tasks are running Key: FLINK-34907 URL: https://issues.apache.org/jira/browse/FLINK-34907 Project: Flink Issue Type: Improvement Components: Autoscaler Reporter: Rui Fan Assignee: Rui Fan Fix For: kubernetes-operator-1.9.0 Currently, we consider the timestamp that JobStatus is changed to RUNNING as jobRunningTs. But the JobStatus will be RUNNING once job starts schedule, so it doesn't mean all tasks are running. It will let the isStabilizing or estimating restart time are not accurate. Solution: jobRunningTs should be the timestamp that all tasks are running. It can be got from SubtasksTimesHeaders rest api. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [hotfix][Connectors/AWS] Update Flink versions in CI [flink-connector-aws]
dannycranmer merged PR #134: URL: https://github.com/apache/flink-connector-aws/pull/134 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34906] Only scale when all tasks are running [flink-kubernetes-operator]
1996fanrui commented on PR #801: URL: https://github.com/apache/flink-kubernetes-operator/pull/801#issuecomment-2011833091 > This issue only affects the standalone autoscaler as the kubernetes operator has this logic already in place for setting the RUNNING state. Can we somehow deduplicate this logic? Thanks @gyfora provide this information, I didn't notice kubernetes operator has this logic before. Let me look into it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34907) jobRunningTs should be the timestamp that all tasks are running
[ https://issues.apache.org/jira/browse/FLINK-34907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829467#comment-17829467 ] Gyula Fora commented on FLINK-34907: similar to the other Jirayou opened this only seems to affect the standalone autoscaler logic > jobRunningTs should be the timestamp that all tasks are running > --- > > Key: FLINK-34907 > URL: https://issues.apache.org/jira/browse/FLINK-34907 > Project: Flink > Issue Type: Improvement > Components: Autoscaler >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > Fix For: kubernetes-operator-1.9.0 > > > Currently, we consider the timestamp that JobStatus is changed to RUNNING as > jobRunningTs. But the JobStatus will be RUNNING once job starts schedule, so > it doesn't mean all tasks are running. > It will let the isStabilizing or estimating restart time are not accurate. > Solution: jobRunningTs should be the timestamp that all tasks are running. > It can be got from SubtasksTimesHeaders rest api. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34891) RemoteStorageScannerTest causes exit 239
[
https://issues.apache.org/jira/browse/FLINK-34891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-34891:
--
Description:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58432&view=logs&j=f0ac5c25-1168-55a5-07ff-0e88223afed9&t=50bf7a25-bdc4-5e56-5478-c7b4511dde53&l=9121]
{code}
Mar 20 01:22:54 01:22:54.671 [ERROR] Error occurred in starting fork, check
output in log
Mar 20 01:22:54 01:22:54.671 [ERROR] Process Exit Code: 239
Mar 20 01:22:54 01:22:54.671 [ERROR] Crashed tests:
Mar 20 01:22:54 01:22:54.671 [ERROR]
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest
Mar 20 01:22:54 01:22:54.671 [ERROR]at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:643)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.lambda$runSuitesForkOnceMultiple$3(ForkStarter.java:350)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
java.base/java.lang.Thread.run(Thread.java:829)
Mar 20 01:22:54 01:22:54.671 [ERROR] -> [Help 1]{code}
{{Tje build artifact mvn-1.log the following FATAL error is found. It appears
that the LocalFileSystem being used by the test can return null for listStatus,
causing a NullPointerException in
{{RemoteStorageScanner#[scanMaxSegmentId|https://github.com/apache/flink/blob/6e138f1a4832bd4379fa6289f0445f1ede9b5a54/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java#L238]
{code:java}
01:16:09,428 [ForkJoinPool-39-worker-51] INFO
org.apache.flink.util.TestLoggerExtension [] -
Test
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest.testWatchSegmentIgnored[testWatchSegmentIgnored()]
is running.
01:16:10,142 [ForkJoinPool-39-worker-51] INFO
org.apache.flink.util.TestLoggerExtension [] -
Test
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest.testWatchSegmentIgnored[testWatchSegmentIgnored()]
successfully run.
01:16:10,142 [remote storage scanner-thread-1] ERROR
org.apache.flink.util.FatalExitExceptionHandler [] - FATAL: Thread
'remote storage scanner-thread-1' produced an uncaught exception. Stopping the
process...
java.lang.NullPointerException: null
at
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScanner.scanMaxSegmentId(RemoteStorageScanner.java:247)
~[classes/:?]
at
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScanner.run(RemoteStorageScanner.java:200)
[classes/:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
01:16:10,144 [ForkJoinPool-39-worker-51] INFO
org.apache.flink.util.TestLoggerExtension [] -
Test
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest.testScanStrategy[testScanStrategy()]
is running.
01:16:10,145 [ForkJoinPool-39-worker-51] INFO
org.apache.flink.util.TestLoggerExtension [] -
Test
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest.testScanStrategy[testScanStrategy()]
successfully run.
{code}
wa
[jira] [Updated] (FLINK-34891) RemoteStorageScannerTest causes exit 239
[
https://issues.apache.org/jira/browse/FLINK-34891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-34891:
--
Description:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58432&view=logs&j=f0ac5c25-1168-55a5-07ff-0e88223afed9&t=50bf7a25-bdc4-5e56-5478-c7b4511dde53&l=9121]
{code}
Mar 20 01:22:54 01:22:54.671 [ERROR] Error occurred in starting fork, check
output in log
Mar 20 01:22:54 01:22:54.671 [ERROR] Process Exit Code: 239
Mar 20 01:22:54 01:22:54.671 [ERROR] Crashed tests:
Mar 20 01:22:54 01:22:54.671 [ERROR]
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest
Mar 20 01:22:54 01:22:54.671 [ERROR]at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:643)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.lambda$runSuitesForkOnceMultiple$3(ForkStarter.java:350)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
java.base/java.lang.Thread.run(Thread.java:829)
Mar 20 01:22:54 01:22:54.671 [ERROR] -> [Help 1]{code}
The build artifact mvn-1.log the following FATAL error is found. It appears
that the LocalFileSystem being used by the test can return null for listStatus,
causing a NullPointerException in
{{RemoteStorageScanner#[scanMaxSegmentId|https://github.com/apache/flink/blob/6e138f1a4832bd4379fa6289f0445f1ede9b5a54/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java#L238]
{code:java}
01:16:09,428 [ForkJoinPool-39-worker-51] INFO
org.apache.flink.util.TestLoggerExtension [] -
Test
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest.testWatchSegmentIgnored[testWatchSegmentIgnored()]
is running.
01:16:10,142 [ForkJoinPool-39-worker-51] INFO
org.apache.flink.util.TestLoggerExtension [] -
Test
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest.testWatchSegmentIgnored[testWatchSegmentIgnored()]
successfully run.
01:16:10,142 [remote storage scanner-thread-1] ERROR
org.apache.flink.util.FatalExitExceptionHandler [] - FATAL: Thread
'remote storage scanner-thread-1' produced an uncaught exception. Stopping the
process...
java.lang.NullPointerException: null
at
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScanner.scanMaxSegmentId(RemoteStorageScanner.java:247)
~[classes/:?]
at
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScanner.run(RemoteStorageScanner.java:200)
[classes/:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
01:16:10,144 [ForkJoinPool-39-worker-51] INFO
org.apache.flink.util.TestLoggerExtension [] -
Test
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest.testScanStrategy[testScanStrategy()]
is running.
01:16:10,145 [ForkJoinPool-39-worker-51] INFO
org.apache.flink.util.TestLoggerExtension [] -
Test
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest.testScanStrategy[testScanStrategy()]
successfully run.
{code}
was:
[jira] [Updated] (FLINK-34891) RemoteStorageScannerTest causes exit 239
[
https://issues.apache.org/jira/browse/FLINK-34891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-34891:
--
Description:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58432&view=logs&j=f0ac5c25-1168-55a5-07ff-0e88223afed9&t=50bf7a25-bdc4-5e56-5478-c7b4511dde53&l=9121]
{code}
Mar 20 01:22:54 01:22:54.671 [ERROR] Error occurred in starting fork, check
output in log
Mar 20 01:22:54 01:22:54.671 [ERROR] Process Exit Code: 239
Mar 20 01:22:54 01:22:54.671 [ERROR] Crashed tests:
Mar 20 01:22:54 01:22:54.671 [ERROR]
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest
Mar 20 01:22:54 01:22:54.671 [ERROR]at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:643)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.lambda$runSuitesForkOnceMultiple$3(ForkStarter.java:350)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Mar 20 01:22:54 01:22:54.671 [ERROR]at
java.base/java.lang.Thread.run(Thread.java:829)
Mar 20 01:22:54 01:22:54.671 [ERROR] -> [Help 1]{code}
The build artifact {{mvn-1.log}} the following {{FATAL}} error is found. It
appears that the {{LocalFileSystem}} being used by the test can return null for
listStatus, causing a NullPointerException in
{{RemoteStorageScanner#[scanMaxSegmentId|https://github.com/apache/flink/blob/6e138f1a4832bd4379fa6289f0445f1ede9b5a54/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java#L238]}}
{code:java}
01:16:09,428 [ForkJoinPool-39-worker-51] INFO
org.apache.flink.util.TestLoggerExtension [] -
Test
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest.testWatchSegmentIgnored[testWatchSegmentIgnored()]
is running.
01:16:10,142 [ForkJoinPool-39-worker-51] INFO
org.apache.flink.util.TestLoggerExtension [] -
Test
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest.testWatchSegmentIgnored[testWatchSegmentIgnored()]
successfully run.
01:16:10,142 [remote storage scanner-thread-1] ERROR
org.apache.flink.util.FatalExitExceptionHandler [] - FATAL: Thread
'remote storage scanner-thread-1' produced an uncaught exception. Stopping the
process...
java.lang.NullPointerException: null
at
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScanner.scanMaxSegmentId(RemoteStorageScanner.java:247)
~[classes/:?]
at
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScanner.run(RemoteStorageScanner.java:200)
[classes/:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
01:16:10,144 [ForkJoinPool-39-worker-51] INFO
org.apache.flink.util.TestLoggerExtension [] -
Test
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest.testScanStrategy[testScanStrategy()]
is running.
01:16:10,145 [ForkJoinPool-39-worker-51] INFO
org.apache.flink.util.TestLoggerExtension [] -
Test
org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScannerTest.testScanStrategy[testScanStrategy()]
successfully run.
{cod
[jira] [Commented] (FLINK-34907) jobRunningTs should be the timestamp that all tasks are running
[ https://issues.apache.org/jira/browse/FLINK-34907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829473#comment-17829473 ] Rui Fan commented on FLINK-34907: - I see the jobRunningTs is generated from JobDetailsInfo instead of kubernetes operator. The JobDetailsInfo is got via rest api in autoscaler module, so it may be a common issue. Feel free to correct me if my understanding is wrong, thanks a lot. > jobRunningTs should be the timestamp that all tasks are running > --- > > Key: FLINK-34907 > URL: https://issues.apache.org/jira/browse/FLINK-34907 > Project: Flink > Issue Type: Improvement > Components: Autoscaler >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > Fix For: kubernetes-operator-1.9.0 > > > Currently, we consider the timestamp that JobStatus is changed to RUNNING as > jobRunningTs. But the JobStatus will be RUNNING once job starts schedule, so > it doesn't mean all tasks are running. > It will let the isStabilizing or estimating restart time are not accurate. > Solution: jobRunningTs should be the timestamp that all tasks are running. > It can be got from SubtasksTimesHeaders rest api. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-34660) AutoRescalingITCase#testCheckpointRescalingInKeyedState AssertionError
[
https://issues.apache.org/jira/browse/FLINK-34660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl resolved FLINK-34660.
---
Resolution: Fixed
* master
**
[e79df3544a59ec7bcd97c9946eb54b7bc3ecf009|https://github.com/apache/flink/commit/e79df3544a59ec7bcd97c9946eb54b7bc3ecf009]
**
[aa7158934bbe17fa13945dddc32d14e2aa613ae5|https://github.com/apache/flink/commit/aa7158934bbe17fa13945dddc32d14e2aa613ae5]
> AutoRescalingITCase#testCheckpointRescalingInKeyedState AssertionError
> --
>
> Key: FLINK-34660
> URL: https://issues.apache.org/jira/browse/FLINK-34660
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Reporter: Hangxiang Yu
>Assignee: Hangxiang Yu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58249&view=ms.vss-test-web.build-test-results-tab&runId=4036370&resultId=100718&paneView=debug]
>
> {code:java}
> expected:<[(0,8000), (0,32000), (0,48000), (0,72000), (1,78000), (1,3),
> (1,54000), (0,2000), (0,1), (0,5), (0,66000), (0,74000), (0,82000),
> (1,8), (1,0), (1,16000), (1,24000), (1,4), (1,56000), (1,64000),
> (0,12000), (0,28000), (0,52000), (0,6), (0,68000), (0,76000), (1,18000),
> (1,26000), (1,34000), (1,42000), (1,58000), (0,6000), (0,14000), (0,22000),
> (0,38000), (0,46000), (0,62000), (0,7), (1,4000), (1,2), (1,36000),
> (1,44000)]> but was:<[(0,8000), (0,32000), (0,48000), (0,72000), (1,78000),
> (1,3), (1,54000), (0,2000), (0,1), (0,5), (0,66000), (0,74000),
> (0,82000), (0,23000), (0,31000), (1,8), (1,0), (1,16000), (1,24000),
> (1,4), (1,56000), (1,64000), (0,12000), (0,28000), (0,52000), (0,6),
> (0,68000), (0,76000), (1,18000), (1,26000), (1,34000), (1,42000), (1,58000),
> (0,6000), (0,14000), (0,22000), (0,19000), (0,35000), (1,4000), (1,2),
> (1,36000), (1,44000)]> {code}
>
> This maybe related to FLINK-34624 as we could see from the log:
> {code:java}
> 03:31:02,073 [ main] INFO
> org.apache.flink.runtime.testutils.PseudoRandomValueSelector [] - Randomly
> selected true for state.changelog.enabled
> 03:31:02,163 [jobmanager-io-thread-2] INFO
> org.apache.flink.state.changelog.AbstractChangelogStateBackend [] -
> ChangelogStateBackend is used, delegating EmbeddedRocksDBStateBackend. {code}
> FLINK-34624 disables changelog since it doesn't support local rescaling
> currently.
> Even if disabling changelog for AutoRescalingITCase manually,
> randomization may still be applied to it.
> We should apply randomization only when it's not pre-defined.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34227) Job doesn't disconnect from ResourceManager
[
https://issues.apache.org/jira/browse/FLINK-34227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829479#comment-17829479
]
Matthias Pohl commented on FLINK-34227:
---
https://github.com/apache/flink/actions/runs/8290287446/job/22688312326#step:10:14956
> Job doesn't disconnect from ResourceManager
> ---
>
> Key: FLINK-34227
> URL: https://issues.apache.org/jira/browse/FLINK-34227
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.19.0, 1.18.1
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Critical
> Labels: github-actions, pull-request-available, test-stability
> Attachments: FLINK-34227.7e7d69daebb438b8d03b7392c9c55115.log,
> FLINK-34227.log
>
>
> https://github.com/XComp/flink/actions/runs/7634987973/job/20800205972#step:10:14557
> {code}
> [...]
> "main" #1 prio=5 os_prio=0 tid=0x7f4b7000 nid=0x24ec0 waiting on
> condition [0x7fccce1eb000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xbdd52618> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2131)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2099)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2077)
> at
> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:876)
> at
> org.apache.flink.table.planner.runtime.stream.sql.WindowDistinctAggregateITCase.testHopWindow_Cube(WindowDistinctAggregateITCase.scala:550)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34404) GroupWindowAggregateProcTimeRestoreTest#testRestore times out
[
https://issues.apache.org/jira/browse/FLINK-34404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829481#comment-17829481
]
Matthias Pohl commented on FLINK-34404:
---
https://github.com/apache/flink/actions/runs/8290287716/job/22688320933#step:10:11684
{code}
cat mvn-*.log | grep testRestore | cut -d' ' -f2 | sort | uniq -c | sort -n |
head -3
1
org.apache.flink.table.planner.plan.nodes.exec.stream.WindowTableFunctionProcTimeRestoreTest.testRestore[[1]
2
org.apache.flink.table.planner.plan.nodes.exec.stream.AsyncCalcRestoreTest.testRestore[[1]
2
org.apache.flink.table.planner.plan.nodes.exec.stream.AsyncCalcRestoreTest.testRestore[[2]
{code}
> GroupWindowAggregateProcTimeRestoreTest#testRestore times out
> -
>
> Key: FLINK-34404
> URL: https://issues.apache.org/jira/browse/FLINK-34404
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0, 1.20.0
>Reporter: Matthias Pohl
>Assignee: Alan Sheinberg
>Priority: Critical
> Labels: test-stability
> Attachments: FLINK-34404.failure.log, FLINK-34404.success.log
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=57357&view=logs&j=32715a4c-21b8-59a3-4171-744e5ab107eb&t=ff64056b-5320-5afe-c22c-6fa339e59586&l=11603
> {code}
> Feb 07 02:17:40 "ForkJoinPool-74-worker-1" #382 daemon prio=5 os_prio=0
> cpu=282.22ms elapsed=961.78s tid=0x7f880a485c00 nid=0x6745 waiting on
> condition [0x7f878a6f9000]
> Feb 07 02:17:40java.lang.Thread.State: WAITING (parking)
> Feb 07 02:17:40 at
> jdk.internal.misc.Unsafe.park(java.base@17.0.7/Native Method)
> Feb 07 02:17:40 - parking to wait for <0xff73d060> (a
> java.util.concurrent.CompletableFuture$Signaller)
> Feb 07 02:17:40 at
> java.util.concurrent.locks.LockSupport.park(java.base@17.0.7/LockSupport.java:211)
> Feb 07 02:17:40 at
> java.util.concurrent.CompletableFuture$Signaller.block(java.base@17.0.7/CompletableFuture.java:1864)
> Feb 07 02:17:40 at
> java.util.concurrent.ForkJoinPool.compensatedBlock(java.base@17.0.7/ForkJoinPool.java:3449)
> Feb 07 02:17:40 at
> java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.7/ForkJoinPool.java:3432)
> Feb 07 02:17:40 at
> java.util.concurrent.CompletableFuture.waitingGet(java.base@17.0.7/CompletableFuture.java:1898)
> Feb 07 02:17:40 at
> java.util.concurrent.CompletableFuture.get(java.base@17.0.7/CompletableFuture.java:2072)
> Feb 07 02:17:40 at
> org.apache.flink.table.planner.plan.nodes.exec.testutils.RestoreTestBase.testRestore(RestoreTestBase.java:292)
> Feb 07 02:17:40 at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.7/Native
> Method)
> Feb 07 02:17:40 at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.7/NativeMethodAccessorImpl.java:77)
> Feb 07 02:17:40 at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.7/DelegatingMethodAccessorImpl.java:43)
> Feb 07 02:17:40 at
> java.lang.reflect.Method.invoke(java.base@17.0.7/Method.java:568)
> Feb 07 02:17:40 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34660) AutoRescalingITCase#testCheckpointRescalingInKeyedState AssertionError
[
https://issues.apache.org/jira/browse/FLINK-34660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829482#comment-17829482
]
Matthias Pohl commented on FLINK-34660:
---
These test failures didn't include the fix from above, yet:
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688337407#step:10:8913
> AutoRescalingITCase#testCheckpointRescalingInKeyedState AssertionError
> --
>
> Key: FLINK-34660
> URL: https://issues.apache.org/jira/browse/FLINK-34660
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Reporter: Hangxiang Yu
>Assignee: Hangxiang Yu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58249&view=ms.vss-test-web.build-test-results-tab&runId=4036370&resultId=100718&paneView=debug]
>
> {code:java}
> expected:<[(0,8000), (0,32000), (0,48000), (0,72000), (1,78000), (1,3),
> (1,54000), (0,2000), (0,1), (0,5), (0,66000), (0,74000), (0,82000),
> (1,8), (1,0), (1,16000), (1,24000), (1,4), (1,56000), (1,64000),
> (0,12000), (0,28000), (0,52000), (0,6), (0,68000), (0,76000), (1,18000),
> (1,26000), (1,34000), (1,42000), (1,58000), (0,6000), (0,14000), (0,22000),
> (0,38000), (0,46000), (0,62000), (0,7), (1,4000), (1,2), (1,36000),
> (1,44000)]> but was:<[(0,8000), (0,32000), (0,48000), (0,72000), (1,78000),
> (1,3), (1,54000), (0,2000), (0,1), (0,5), (0,66000), (0,74000),
> (0,82000), (0,23000), (0,31000), (1,8), (1,0), (1,16000), (1,24000),
> (1,4), (1,56000), (1,64000), (0,12000), (0,28000), (0,52000), (0,6),
> (0,68000), (0,76000), (1,18000), (1,26000), (1,34000), (1,42000), (1,58000),
> (0,6000), (0,14000), (0,22000), (0,19000), (0,35000), (1,4000), (1,2),
> (1,36000), (1,44000)]> {code}
>
> This maybe related to FLINK-34624 as we could see from the log:
> {code:java}
> 03:31:02,073 [ main] INFO
> org.apache.flink.runtime.testutils.PseudoRandomValueSelector [] - Randomly
> selected true for state.changelog.enabled
> 03:31:02,163 [jobmanager-io-thread-2] INFO
> org.apache.flink.state.changelog.AbstractChangelogStateBackend [] -
> ChangelogStateBackend is used, delegating EmbeddedRocksDBStateBackend. {code}
> FLINK-34624 disables changelog since it doesn't support local rescaling
> currently.
> Even if disabling changelog for AutoRescalingITCase manually,
> randomization may still be applied to it.
> We should apply randomization only when it's not pre-defined.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28440) EventTimeWindowCheckpointingITCase failed with restore
[
https://issues.apache.org/jira/browse/FLINK-28440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829483#comment-17829483
]
Matthias Pohl commented on FLINK-28440:
---
https://github.com/apache/flink/actions/runs/8290287716/job/22688395896#step:10:8229
> EventTimeWindowCheckpointingITCase failed with restore
> --
>
> Key: FLINK-28440
> URL: https://issues.apache.org/jira/browse/FLINK-28440
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / State Backends
>Affects Versions: 1.16.0, 1.17.0, 1.18.0, 1.19.0
>Reporter: Huang Xingbo
>Assignee: Yanfei Lei
>Priority: Critical
> Labels: auto-deprioritized-critical, pull-request-available,
> stale-assigned, test-stability
> Fix For: 1.20.0
>
> Attachments: image-2023-02-01-00-51-54-506.png,
> image-2023-02-01-01-10-01-521.png, image-2023-02-01-01-19-12-182.png,
> image-2023-02-01-16-47-23-756.png, image-2023-02-01-16-57-43-889.png,
> image-2023-02-02-10-52-56-599.png, image-2023-02-03-10-09-07-586.png,
> image-2023-02-03-12-03-16-155.png, image-2023-02-03-12-03-56-614.png
>
>
> {code:java}
> Caused by: java.lang.Exception: Exception while creating
> StreamOperatorStateContext.
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:256)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268)
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:722)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:698)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:665)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:904)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for WindowOperator_0a448493b4782967b150582570326227_(2/4) from
> any of the 1 provided restore options.
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:353)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:165)
> ... 11 more
> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException:
> /tmp/junit1835099326935900400/junit1113650082510421526/52ee65b7-033f-4429-8ddd-adbe85e27ced
> (No such file or directory)
> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
> at
> org.apache.flink.runtime.state.changelog.StateChangelogHandleStreamHandleReader$1.advance(StateChangelogHandleStreamHandleReader.java:87)
> at
> org.apache.flink.runtime.state.changelog.StateChangelogHandleStreamHandleReader$1.hasNext(StateChangelogHandleStreamHandleReader.java:69)
> at
> org.apache.flink.state.changelog.restore.ChangelogBackendRestoreOperation.readBackendHandle(ChangelogBackendRestoreOperation.java:96)
> at
> org.apache.flink.state.changelog.restore.ChangelogBackendRestoreOperation.restore(ChangelogBackendRestoreOperation.java:75)
> at
> org.apache.flink.state.changelog.ChangelogStateBackend.restore(ChangelogStateBackend.java:92)
> at
> org.apache.flink.state.changelog.AbstractChangelogStateBackend.createKeyedStateBackend(AbstractChangelogStateBackend.java:136)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:336)
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
> ... 13 more
> Caused by: java.io.FileNotFoundException:
> /tmp/junit1835099326935900400/junit1113650082510421526/52ee65b7-03
[jira] [Commented] (FLINK-34718) KeyedPartitionWindowedStream and NonPartitionWindowedStream IllegalStateException in AZP
[
https://issues.apache.org/jira/browse/FLINK-34718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829484#comment-17829484
]
Matthias Pohl commented on FLINK-34718:
---
before the fix was committed to master:
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9329
> KeyedPartitionWindowedStream and NonPartitionWindowedStream
> IllegalStateException in AZP
>
>
> Key: FLINK-34718
> URL: https://issues.apache.org/jira/browse/FLINK-34718
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Assignee: Ryan Skraba
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58320&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=9646]
> 18 of the KeyedPartitionWindowedStreamITCase and
> NonKeyedPartitionWindowedStreamITCase unit tests introduced in FLINK-34543
> are failing in the adaptive scheduler profile, with errors similar to:
> {code:java}
> Mar 15 01:54:12 Caused by: java.lang.IllegalStateException: The adaptive
> scheduler supports pipelined data exchanges (violated by MapPartition
> (org.apache.flink.streaming.runtime.tasks.OneInputStreamTask) ->
> ddb598ad156ed281023ba4eebbe487e3).
> Mar 15 01:54:12 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.assertPreconditions(AdaptiveScheduler.java:438)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.(AdaptiveScheduler.java:356)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerFactory.createInstance(AdaptiveSchedulerFactory.java:124)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:121)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:384)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:361)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
> Mar 15 01:54:12 at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
> Mar 15 01:54:12 at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> Mar 15 01:54:12 ... 4 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34643) JobIDLoggingITCase failed
[
https://issues.apache.org/jira/browse/FLINK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829485#comment-17829485
]
Matthias Pohl commented on FLINK-34643:
---
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
> JobIDLoggingITCase failed
> -
>
> Key: FLINK-34643
> URL: https://issues.apache.org/jira/browse/FLINK-34643
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7897
> {code}
> Mar 09 01:24:23 01:24:23.498 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 4.209 s <<< FAILURE! -- in
> org.apache.flink.test.misc.JobIDLoggingITCase
> Mar 09 01:24:23 01:24:23.498 [ERROR]
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(ClusterClient)
> -- Time elapsed: 1.459 s <<< ERROR!
> Mar 09 01:24:23 java.lang.IllegalStateException: Too few log events recorded
> for org.apache.flink.runtime.jobmaster.JobMaster (12) - this must be a bug in
> the test code
> Mar 09 01:24:23 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.assertJobIDPresent(JobIDLoggingITCase.java:148)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(JobIDLoggingITCase.java:132)
> Mar 09 01:24:23 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 09 01:24:23 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Mar 09 01:24:23
> {code}
> The other test failures of this build were also caused by the same test:
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8349
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3&t=712ade8c-ca16-5b76-3acd-14df33bc1cb1&l=8209
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-26974) Python EmbeddedThreadDependencyTests.test_add_python_file failed on azure
[
https://issues.apache.org/jira/browse/FLINK-26974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829486#comment-17829486
]
Matthias Pohl commented on FLINK-26974:
---
https://github.com/apache/flink/actions/runs/8293763348/job/22697880009#step:10:24188
> Python EmbeddedThreadDependencyTests.test_add_python_file failed on azure
> -
>
> Key: FLINK-26974
> URL: https://issues.apache.org/jira/browse/FLINK-26974
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.15.0, 1.16.0, 1.17.0, 1.19.0
>Reporter: Yun Gao
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: auto-deprioritized-major, stale-assigned, test-stability
>
> {code:java}
> Mar 31 10:49:17 === FAILURES
> ===
> Mar 31 10:49:17 __
> EmbeddedThreadDependencyTests.test_add_python_file __
> Mar 31 10:49:17
> Mar 31 10:49:17 self =
> testMethod=test_add_python_file>
> Mar 31 10:49:17
> Mar 31 10:49:17 def test_add_python_file(self):
> Mar 31 10:49:17 python_file_dir = os.path.join(self.tempdir,
> "python_file_dir_" + str(uuid.uuid4()))
> Mar 31 10:49:17 os.mkdir(python_file_dir)
> Mar 31 10:49:17 python_file_path = os.path.join(python_file_dir,
> "test_dependency_manage_lib.py")
> Mar 31 10:49:17 with open(python_file_path, 'w') as f:
> Mar 31 10:49:17 f.write("def add_two(a):\nraise
> Exception('This function should not be called!')")
> Mar 31 10:49:17 self.t_env.add_python_file(python_file_path)
> Mar 31 10:49:17
> Mar 31 10:49:17 python_file_dir_with_higher_priority = os.path.join(
> Mar 31 10:49:17 self.tempdir, "python_file_dir_" +
> str(uuid.uuid4()))
> Mar 31 10:49:17 os.mkdir(python_file_dir_with_higher_priority)
> Mar 31 10:49:17 python_file_path_higher_priority =
> os.path.join(python_file_dir_with_higher_priority,
> Mar 31 10:49:17
> "test_dependency_manage_lib.py")
> Mar 31 10:49:17 with open(python_file_path_higher_priority, 'w') as f:
> Mar 31 10:49:17 f.write("def add_two(a):\nreturn a + 2")
> Mar 31 10:49:17
> self.t_env.add_python_file(python_file_path_higher_priority)
> Mar 31 10:49:17
> Mar 31 10:49:17 def plus_two(i):
> Mar 31 10:49:17 from test_dependency_manage_lib import add_two
> Mar 31 10:49:17 return add_two(i)
> Mar 31 10:49:17
> Mar 31 10:49:17 self.t_env.create_temporary_system_function(
> Mar 31 10:49:17 "add_two", udf(plus_two, DataTypes.BIGINT(),
> DataTypes.BIGINT()))
> Mar 31 10:49:17 table_sink = source_sink_utils.TestAppendSink(
> Mar 31 10:49:17 ['a', 'b'], [DataTypes.BIGINT(),
> DataTypes.BIGINT()])
> Mar 31 10:49:17 self.t_env.register_table_sink("Results", table_sink)
> Mar 31 10:49:17 t = self.t_env.from_elements([(1, 2), (2, 5), (3,
> 1)], ['a', 'b'])
> Mar 31 10:49:17 > t.select(expr.call("add_two", t.a),
> t.a).execute_insert("Results").wait()
> Mar 31 10:49:17
> Mar 31 10:49:17 pyflink/table/tests/test_dependency.py:63:
> Mar 31 10:49:17 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _
> Mar 31 10:49:17 pyflink/table/table_result.py:76: in wait
> Mar 31 10:49:17 get_method(self._j_table_result, "await")()
> Mar 31 10:49:17
> .tox/py38-cython/lib/python3.8/site-packages/py4j/java_gateway.py:1321: in
> __call__
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=34001&view=logs&j=821b528f-1eed-5598-a3b4-7f748b13f261&t=6bb545dd-772d-5d8c-f258-f5085fba3295&l=27239
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34908) mysql pipeline to doris and starrocks will lost precision for timestamp
Xin Gong created FLINK-34908: Summary: mysql pipeline to doris and starrocks will lost precision for timestamp Key: FLINK-34908 URL: https://issues.apache.org/jira/browse/FLINK-34908 Project: Flink Issue Type: Improvement Components: Flink CDC Reporter: Xin Gong Fix For: cdc-3.1.0 flink cdc pipeline will decide timestamp zone by config of pipeline. I found mysql2doris and mysql2starracks will specific datetime format -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we can don't specific datetime format, just return LocalDateTime object. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34908) mysql pipeline to doris and starrocks will lost precision for timestamp
[ https://issues.apache.org/jira/browse/FLINK-34908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Gong updated FLINK-34908: - Description: flink cdc pipeline will decide timestamp zone by config of pipeline. I found mysql2doris and mysql2starracks will specific datetime format -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we don't specific datetime format, just return LocalDateTime object. was: flink cdc pipeline will decide timestamp zone by config of pipeline. I found mysql2doris and mysql2starracks will specific datetime format -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we can don't specific datetime format, just return LocalDateTime object. > mysql pipeline to doris and starrocks will lost precision for timestamp > --- > > Key: FLINK-34908 > URL: https://issues.apache.org/jira/browse/FLINK-34908 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Xin Gong >Priority: Minor > Fix For: cdc-3.1.0 > > > flink cdc pipeline will decide timestamp zone by config of pipeline. I found > mysql2doris and mysql2starracks will specific datetime format > -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we don't > specific datetime format, just return LocalDateTime object. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
[
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829487#comment-17829487
]
Matthias Pohl commented on FLINK-31472:
---
https://github.com/apache/flink/actions/runs/8297049625/job/22707710859#step:10:10483
> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
>
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1, 1.18.0, 1.19.0, 1.20.0
>Reporter: Ran Tao
>Assignee: Ahmed Hamdy
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> when run mvn clean test, this case failed occasionally.
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955
> s <<< FAILURE! - in
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> Time elapsed: 0.492 s <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be
> called from inside the mailbox thread!
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
> at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
> at
> org.junit.platform.launcher.core.EngineExecutionOrches
[jira] [Comment Edited] (FLINK-34660) AutoRescalingITCase#testCheckpointRescalingInKeyedState AssertionError
[
https://issues.apache.org/jira/browse/FLINK-34660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829482#comment-17829482
]
Matthias Pohl edited comment on FLINK-34660 at 3/21/24 10:51 AM:
-
These test failures didn't include the fix from above, yet:
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688337407#step:10:8913
*
https://github.com/apache/flink/actions/runs/8297049625/job/22707711131#step:10:8391
was (Author: mapohl):
These test failures didn't include the fix from above, yet:
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688337407#step:10:8913
> AutoRescalingITCase#testCheckpointRescalingInKeyedState AssertionError
> --
>
> Key: FLINK-34660
> URL: https://issues.apache.org/jira/browse/FLINK-34660
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Reporter: Hangxiang Yu
>Assignee: Hangxiang Yu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58249&view=ms.vss-test-web.build-test-results-tab&runId=4036370&resultId=100718&paneView=debug]
>
> {code:java}
> expected:<[(0,8000), (0,32000), (0,48000), (0,72000), (1,78000), (1,3),
> (1,54000), (0,2000), (0,1), (0,5), (0,66000), (0,74000), (0,82000),
> (1,8), (1,0), (1,16000), (1,24000), (1,4), (1,56000), (1,64000),
> (0,12000), (0,28000), (0,52000), (0,6), (0,68000), (0,76000), (1,18000),
> (1,26000), (1,34000), (1,42000), (1,58000), (0,6000), (0,14000), (0,22000),
> (0,38000), (0,46000), (0,62000), (0,7), (1,4000), (1,2), (1,36000),
> (1,44000)]> but was:<[(0,8000), (0,32000), (0,48000), (0,72000), (1,78000),
> (1,3), (1,54000), (0,2000), (0,1), (0,5), (0,66000), (0,74000),
> (0,82000), (0,23000), (0,31000), (1,8), (1,0), (1,16000), (1,24000),
> (1,4), (1,56000), (1,64000), (0,12000), (0,28000), (0,52000), (0,6),
> (0,68000), (0,76000), (1,18000), (1,26000), (1,34000), (1,42000), (1,58000),
> (0,6000), (0,14000), (0,22000), (0,19000), (0,35000), (1,4000), (1,2),
> (1,36000), (1,44000)]> {code}
>
> This maybe related to FLINK-34624 as we could see from the log:
> {code:java}
> 03:31:02,073 [ main] INFO
> org.apache.flink.runtime.testutils.PseudoRandomValueSelector [] - Randomly
> selected true for state.changelog.enabled
> 03:31:02,163 [jobmanager-io-thread-2] INFO
> org.apache.flink.state.changelog.AbstractChangelogStateBackend [] -
> ChangelogStateBackend is used, delegating EmbeddedRocksDBStateBackend. {code}
> FLINK-34624 disables changelog since it doesn't support local rescaling
> currently.
> Even if disabling changelog for AutoRescalingITCase manually,
> randomization may still be applied to it.
> We should apply randomization only when it's not pre-defined.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34908) mysql pipeline to doris and starrocks will lost precision for timestamp
[ https://issues.apache.org/jira/browse/FLINK-34908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Gong updated FLINK-34908: - Description: flink cdc pipeline will decide timestamp zone by config of pipeline. I found mysql2doris and mysql2starracks will specific datetime format -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we don't set fixed datetime format, just return LocalDateTime object. was: flink cdc pipeline will decide timestamp zone by config of pipeline. I found mysql2doris and mysql2starracks will specific datetime format -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we don't specific datetime format, just return LocalDateTime object. > mysql pipeline to doris and starrocks will lost precision for timestamp > --- > > Key: FLINK-34908 > URL: https://issues.apache.org/jira/browse/FLINK-34908 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Xin Gong >Priority: Minor > Fix For: cdc-3.1.0 > > > flink cdc pipeline will decide timestamp zone by config of pipeline. I found > mysql2doris and mysql2starracks will specific datetime format > -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we don't > set fixed datetime format, just return LocalDateTime object. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33186) CheckpointAfterAllTasksFinishedITCase.testRestoreAfterSomeTasksFinished fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-33186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829488#comment-17829488
]
Matthias Pohl commented on FLINK-33186:
---
https://github.com/apache/flink/actions/runs/8304570591/job/22730524813#step:10:7494
> CheckpointAfterAllTasksFinishedITCase.testRestoreAfterSomeTasksFinished
> fails on AZP
> -
>
> Key: FLINK-33186
> URL: https://issues.apache.org/jira/browse/FLINK-33186
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.19.0, 1.18.1
>Reporter: Sergey Nuyanzin
>Assignee: Jiang Xin
>Priority: Critical
> Labels: test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53509&view=logs&j=baf26b34-3c6a-54e8-f93f-cf269b32f802&t=8c9d126d-57d2-5a9e-a8c8-ff53f7b35cd9&l=8762
> fails as
> {noformat}
> Sep 28 01:23:43 Caused by:
> org.apache.flink.runtime.checkpoint.CheckpointException: Task local
> checkpoint failure.
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.PendingCheckpoint.abort(PendingCheckpoint.java:550)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2248)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2235)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:817)
> Sep 28 01:23:43 at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> Sep 28 01:23:43 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Sep 28 01:23:43 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> Sep 28 01:23:43 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> Sep 28 01:23:43 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> Sep 28 01:23:43 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> Sep 28 01:23:43 at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34908) mysql pipeline to doris and starrocks will lost precision for timestamp
[ https://issues.apache.org/jira/browse/FLINK-34908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Gong updated FLINK-34908: - Description: flink cdc pipeline will decide timestamp zone by config of pipeline. I found mysql2doris and mysql2starracks will specific datetime format -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we should't set fixed datetime format, just return LocalDateTime object. was: flink cdc pipeline will decide timestamp zone by config of pipeline. I found mysql2doris and mysql2starracks will specific datetime format -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we don't set fixed datetime format, just return LocalDateTime object. > mysql pipeline to doris and starrocks will lost precision for timestamp > --- > > Key: FLINK-34908 > URL: https://issues.apache.org/jira/browse/FLINK-34908 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Xin Gong >Priority: Minor > Fix For: cdc-3.1.0 > > > flink cdc pipeline will decide timestamp zone by config of pipeline. I found > mysql2doris and mysql2starracks will specific datetime format > -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we > should't set fixed datetime format, just return LocalDateTime object. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-26515) RetryingExecutorTest. testDiscardOnTimeout failed on azure
[
https://issues.apache.org/jira/browse/FLINK-26515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829490#comment-17829490
]
Matthias Pohl commented on FLINK-26515:
---
1.18:
https://github.com/apache/flink/actions/runs/8304570797/job/22730570668#step:10:10504
> RetryingExecutorTest. testDiscardOnTimeout failed on azure
> --
>
> Key: FLINK-26515
> URL: https://issues.apache.org/jira/browse/FLINK-26515
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.14.3, 1.17.0, 1.16.1, 1.18.0, 1.19.0
>Reporter: Yun Gao
>Priority: Critical
> Labels: auto-deprioritized-major, pull-request-available,
> test-stability
>
> {code:java}
> Mar 06 01:20:29 [ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 1.941 s <<< FAILURE! - in
> org.apache.flink.changelog.fs.RetryingExecutorTest
> Mar 06 01:20:29 [ERROR] testTimeout Time elapsed: 1.934 s <<< FAILURE!
> Mar 06 01:20:29 java.lang.AssertionError: expected:<500.0> but
> was:<1922.869766>
> Mar 06 01:20:29 at org.junit.Assert.fail(Assert.java:89)
> Mar 06 01:20:29 at org.junit.Assert.failNotEquals(Assert.java:835)
> Mar 06 01:20:29 at org.junit.Assert.assertEquals(Assert.java:555)
> Mar 06 01:20:29 at org.junit.Assert.assertEquals(Assert.java:685)
> Mar 06 01:20:29 at
> org.apache.flink.changelog.fs.RetryingExecutorTest.testTimeout(RetryingExecutorTest.java:145)
> Mar 06 01:20:29 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Mar 06 01:20:29 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Mar 06 01:20:29 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Mar 06 01:20:29 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 06 01:20:29 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Mar 06 01:20:29 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Mar 06 01:20:29 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Mar 06 01:20:29 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Mar 06 01:20:29 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Mar 06 01:20:29 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Mar 06 01:20:29 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Mar 06 01:20:29 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Mar 06 01:20:29 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Mar 06 01:20:29 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Mar 06 01:20:29 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Mar 06 01:20:29 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Mar 06 01:20:29 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Mar 06 01:20:29 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Mar 06 01:20:29 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Mar 06 01:20:29 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Mar 06 01:20:29 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> Mar 06 01:20:29 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> Mar 06 01:20:29 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> Mar 06 01:20:29 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Mar 06 01:20:29 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Mar 06 01:20:29 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> Mar 06 01:20:29 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> Mar 06 01:20:29 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> Mar 06 01:20:29 at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32569&view=logs&j=f450c1a5-64b1-5955-e215-49cb1ad5ec88&t=cc452273-9efa-565d-9db8-ef62a38a0c10&l=22554
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34719) StreamRecordTest#testWithTimestamp fails on Azure
[
https://issues.apache.org/jira/browse/FLINK-34719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829492#comment-17829492
]
Matthias Pohl commented on FLINK-34719:
---
These build failures didn't include the fix, yet:
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730546057#step:10:10174
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730544007#step:10:10045
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531437#step:10:10077
> StreamRecordTest#testWithTimestamp fails on Azure
> -
>
> Key: FLINK-34719
> URL: https://issues.apache.org/jira/browse/FLINK-34719
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Assignee: Jiabao Sun
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> The ClassCastException *message* expected in
> StreamRecordTest#testWithTimestamp as well as
> StreamRecordTest#testWithNoTimestamp fails on JDK 11, 17, and 21
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352&view=logs&j=f0ac5c25-1168-55a5-07ff-0e88223afed9&t=50bf7a25-bdc4-5e56-5478-c7b4511dde53&l=10341]
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352&view=logs&j=675bf62c-8558-587e-2555-dcad13acefb5&t=5878eed3-cc1e-5b12-1ed0-9e7139ce0992&l=9828]
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352&view=logs&j=d06b80b4-9e88-5d40-12a2-18072cf60528&t=609ecd5a-3f6e-5d0c-2239-2096b155a4d0&l=9833]
> {code:java}
> Expecting throwable message:
> Mar 16 01:35:07 "class
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord cannot be cast
> to class org.apache.flink.streaming.api.watermark.Watermark
> (org.apache.flink.streaming.runtime.streamrecord.StreamRecord and
> org.apache.flink.streaming.api.watermark.Watermark are in unnamed module of
> loader 'app')"
> Mar 16 01:35:07 to contain:
> Mar 16 01:35:07 "cannot be cast to
> org.apache.flink.streaming.api.watermark.Watermark"
> Mar 16 01:35:07 but did not.
> Mar 16 01:35:07
> Mar 16 01:35:07 Throwable that failed the check:
> Mar 16 01:35:07
> Mar 16 01:35:07 java.lang.ClassCastException: class
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord cannot be cast
> to class org.apache.flink.streaming.api.watermark.Watermark
> (org.apache.flink.streaming.runtime.streamrecord.StreamRecord and
> org.apache.flink.streaming.api.watermark.Watermark are in unnamed module of
> loader 'app')
> Mar 16 01:35:07 at
> org.apache.flink.streaming.runtime.streamrecord.StreamElement.asWatermark(StreamElement.java:92)
> Mar 16 01:35:07 at
> org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
> Mar 16 01:35:07 at
> org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:892)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
[
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829487#comment-17829487
]
Matthias Pohl edited comment on FLINK-31472 at 3/21/24 10:56 AM:
-
*
https://github.com/apache/flink/actions/runs/8297049625/job/22707710859#step:10:10483
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730583765#step:10:10570
was (Author: mapohl):
https://github.com/apache/flink/actions/runs/8297049625/job/22707710859#step:10:10483
> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
>
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1, 1.18.0, 1.19.0, 1.20.0
>Reporter: Ran Tao
>Assignee: Ahmed Hamdy
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> when run mvn clean test, this case failed occasionally.
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955
> s <<< FAILURE! - in
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> Time elapsed: 0.492 s <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be
> called from inside the mailbox thread!
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
> at
> org.junit.platform.launcher.core.EngineExecutio
[jira] [Comment Edited] (FLINK-34643) JobIDLoggingITCase failed
[
https://issues.apache.org/jira/browse/FLINK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829485#comment-17829485
]
Matthias Pohl edited comment on FLINK-34643 at 3/21/24 10:57 AM:
-
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
was (Author: mapohl):
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
> JobIDLoggingITCase failed
> -
>
> Key: FLINK-34643
> URL: https://issues.apache.org/jira/browse/FLINK-34643
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7897
> {code}
> Mar 09 01:24:23 01:24:23.498 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 4.209 s <<< FAILURE! -- in
> org.apache.flink.test.misc.JobIDLoggingITCase
> Mar 09 01:24:23 01:24:23.498 [ERROR]
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(ClusterClient)
> -- Time elapsed: 1.459 s <<< ERROR!
> Mar 09 01:24:23 java.lang.IllegalStateException: Too few log events recorded
> for org.apache.flink.runtime.jobmaster.JobMaster (12) - this must be a bug in
> the test code
> Mar 09 01:24:23 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.assertJobIDPresent(JobIDLoggingITCase.java:148)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(JobIDLoggingITCase.java:132)
> Mar 09 01:24:23 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 09 01:24:23 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Mar 09 01:24:23
> {code}
> The other test failures of this build were also caused by the same test:
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8349
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3&t=712ade8c-ca16-5b76-3acd-14df33bc1cb1&l=8209
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34718) KeyedPartitionWindowedStream and NonPartitionWindowedStream IllegalStateException in AZP
[
https://issues.apache.org/jira/browse/FLINK-34718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829484#comment-17829484
]
Matthias Pohl edited comment on FLINK-34718 at 3/21/24 10:57 AM:
-
before the fix was committed to master:
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9329
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:8057
was (Author: mapohl):
before the fix was committed to master:
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9329
> KeyedPartitionWindowedStream and NonPartitionWindowedStream
> IllegalStateException in AZP
>
>
> Key: FLINK-34718
> URL: https://issues.apache.org/jira/browse/FLINK-34718
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Assignee: Ryan Skraba
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58320&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=9646]
> 18 of the KeyedPartitionWindowedStreamITCase and
> NonKeyedPartitionWindowedStreamITCase unit tests introduced in FLINK-34543
> are failing in the adaptive scheduler profile, with errors similar to:
> {code:java}
> Mar 15 01:54:12 Caused by: java.lang.IllegalStateException: The adaptive
> scheduler supports pipelined data exchanges (violated by MapPartition
> (org.apache.flink.streaming.runtime.tasks.OneInputStreamTask) ->
> ddb598ad156ed281023ba4eebbe487e3).
> Mar 15 01:54:12 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.assertPreconditions(AdaptiveScheduler.java:438)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.(AdaptiveScheduler.java:356)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerFactory.createInstance(AdaptiveSchedulerFactory.java:124)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:121)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:384)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:361)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
> Mar 15 01:54:12 at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
> Mar 15 01:54:12 at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> Mar 15 01:54:12 ... 4 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34719) StreamRecordTest#testWithTimestamp fails on Azure
[
https://issues.apache.org/jira/browse/FLINK-34719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829492#comment-17829492
]
Matthias Pohl edited comment on FLINK-34719 at 3/21/24 10:59 AM:
-
These build failures didn't include the fix, yet:
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730546057#step:10:10174
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730544007#step:10:10045
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531437#step:10:10077
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747313654#step:10:10176
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747387684#step:10:10044
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747314118#step:10:10073
was (Author: mapohl):
These build failures didn't include the fix, yet:
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730546057#step:10:10174
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730544007#step:10:10045
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531437#step:10:10077
> StreamRecordTest#testWithTimestamp fails on Azure
> -
>
> Key: FLINK-34719
> URL: https://issues.apache.org/jira/browse/FLINK-34719
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Assignee: Jiabao Sun
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> The ClassCastException *message* expected in
> StreamRecordTest#testWithTimestamp as well as
> StreamRecordTest#testWithNoTimestamp fails on JDK 11, 17, and 21
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352&view=logs&j=f0ac5c25-1168-55a5-07ff-0e88223afed9&t=50bf7a25-bdc4-5e56-5478-c7b4511dde53&l=10341]
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352&view=logs&j=675bf62c-8558-587e-2555-dcad13acefb5&t=5878eed3-cc1e-5b12-1ed0-9e7139ce0992&l=9828]
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352&view=logs&j=d06b80b4-9e88-5d40-12a2-18072cf60528&t=609ecd5a-3f6e-5d0c-2239-2096b155a4d0&l=9833]
> {code:java}
> Expecting throwable message:
> Mar 16 01:35:07 "class
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord cannot be cast
> to class org.apache.flink.streaming.api.watermark.Watermark
> (org.apache.flink.streaming.runtime.streamrecord.StreamRecord and
> org.apache.flink.streaming.api.watermark.Watermark are in unnamed module of
> loader 'app')"
> Mar 16 01:35:07 to contain:
> Mar 16 01:35:07 "cannot be cast to
> org.apache.flink.streaming.api.watermark.Watermark"
> Mar 16 01:35:07 but did not.
> Mar 16 01:35:07
> Mar 16 01:35:07 Throwable that failed the check:
> Mar 16 01:35:07
> Mar 16 01:35:07 java.lang.ClassCastException: class
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord cannot be cast
> to class org.apache.flink.streaming.api.watermark.Watermark
> (org.apache.flink.streaming.runtime.streamrecord.StreamRecord and
> org.apache.flink.streaming.api.watermark.Watermark are in unnamed module of
> loader 'app')
> Mar 16 01:35:07 at
> org.apache.flink.streaming.runtime.streamrecord.StreamElement.asWatermark(StreamElement.java:92)
> Mar 16 01:35:07 at
> org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
> Mar 16 01:35:07 at
> org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:892)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34643) JobIDLoggingITCase failed
[
https://issues.apache.org/jira/browse/FLINK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829485#comment-17829485
]
Matthias Pohl edited comment on FLINK-34643 at 3/21/24 11:00 AM:
-
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
was (Author: mapohl):
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
> JobIDLoggingITCase failed
> -
>
> Key: FLINK-34643
> URL: https://issues.apache.org/jira/browse/FLINK-34643
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7897
> {code}
> Mar 09 01:24:23 01:24:23.498 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 4.209 s <<< FAILURE! -- in
> org.apache.flink.test.misc.JobIDLoggingITCase
> Mar 09 01:24:23 01:24:23.498 [ERROR]
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(ClusterClient)
> -- Time elapsed: 1.459 s <<< ERROR!
> Mar 09 01:24:23 java.lang.IllegalStateException: Too few log events recorded
> for org.apache.flink.runtime.jobmaster.JobMaster (12) - this must be a bug in
> the test code
> Mar 09 01:24:23 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.assertJobIDPresent(JobIDLoggingITCase.java:148)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(JobIDLoggingITCase.java:132)
> Mar 09 01:24:23 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 09 01:24:23 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Mar 09 01:24:23
> {code}
> The other test failures of this build were also caused by the same test:
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8349
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3&t=712ade8c-ca16-5b76-3acd-14df33bc1cb1&l=8209
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34718) KeyedPartitionWindowedStream and NonPartitionWindowedStream IllegalStateException in AZP
[
https://issues.apache.org/jira/browse/FLINK-34718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829484#comment-17829484
]
Matthias Pohl edited comment on FLINK-34718 at 3/21/24 11:00 AM:
-
before the fix was committed to master:
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9329
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:8057
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:9345
was (Author: mapohl):
before the fix was committed to master:
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9329
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:8057
> KeyedPartitionWindowedStream and NonPartitionWindowedStream
> IllegalStateException in AZP
>
>
> Key: FLINK-34718
> URL: https://issues.apache.org/jira/browse/FLINK-34718
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Assignee: Ryan Skraba
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58320&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=9646]
> 18 of the KeyedPartitionWindowedStreamITCase and
> NonKeyedPartitionWindowedStreamITCase unit tests introduced in FLINK-34543
> are failing in the adaptive scheduler profile, with errors similar to:
> {code:java}
> Mar 15 01:54:12 Caused by: java.lang.IllegalStateException: The adaptive
> scheduler supports pipelined data exchanges (violated by MapPartition
> (org.apache.flink.streaming.runtime.tasks.OneInputStreamTask) ->
> ddb598ad156ed281023ba4eebbe487e3).
> Mar 15 01:54:12 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.assertPreconditions(AdaptiveScheduler.java:438)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.(AdaptiveScheduler.java:356)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerFactory.createInstance(AdaptiveSchedulerFactory.java:124)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:121)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:384)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:361)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
> Mar 15 01:54:12 at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
> Mar 15 01:54:12 at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> Mar 15 01:54:12 ... 4 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
[
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829487#comment-17829487
]
Matthias Pohl edited comment on FLINK-31472 at 3/21/24 11:01 AM:
-
*
https://github.com/apache/flink/actions/runs/8297049625/job/22707710859#step:10:10483
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730583765#step:10:10570
*
https://github.com/apache/flink/actions/runs/8312246681/job/22747082696#step:10:10710
was (Author: mapohl):
*
https://github.com/apache/flink/actions/runs/8297049625/job/22707710859#step:10:10483
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730583765#step:10:10570
> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
>
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1, 1.18.0, 1.19.0, 1.20.0
>Reporter: Ran Tao
>Assignee: Ahmed Hamdy
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> when run mvn clean test, this case failed occasionally.
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955
> s <<< FAILURE! - in
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> Time elapsed: 0.492 s <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be
> called from inside the mailbox thread!
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(Vin
[jira] [Comment Edited] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
[
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829487#comment-17829487
]
Matthias Pohl edited comment on FLINK-31472 at 3/21/24 11:01 AM:
-
*
https://github.com/apache/flink/actions/runs/8297049625/job/22707710859#step:10:10483
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730583765#step:10:10570
*
https://github.com/apache/flink/actions/runs/8312246681/job/22747082696#step:10:10710
*
https://github.com/apache/flink/actions/runs/8320242265/job/22764905648#step:10:10478
was (Author: mapohl):
*
https://github.com/apache/flink/actions/runs/8297049625/job/22707710859#step:10:10483
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730583765#step:10:10570
*
https://github.com/apache/flink/actions/runs/8312246681/job/22747082696#step:10:10710
> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
>
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1, 1.18.0, 1.19.0, 1.20.0
>Reporter: Ran Tao
>Assignee: Ahmed Hamdy
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> when run mvn clean test, this case failed occasionally.
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955
> s <<< FAILURE! - in
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> Time elapsed: 0.492 s <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be
> called from inside the mailbox thread!
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
> at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
>
[jira] [Updated] (FLINK-34908) [Feature][Pipeline] Mysql pipeline to doris and starrocks will lost precision for timestamp
[ https://issues.apache.org/jira/browse/FLINK-34908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Gong updated FLINK-34908: - Summary: [Feature][Pipeline] Mysql pipeline to doris and starrocks will lost precision for timestamp (was: mysql pipeline to doris and starrocks will lost precision for timestamp) > [Feature][Pipeline] Mysql pipeline to doris and starrocks will lost precision > for timestamp > --- > > Key: FLINK-34908 > URL: https://issues.apache.org/jira/browse/FLINK-34908 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Xin Gong >Priority: Minor > Fix For: cdc-3.1.0 > > > flink cdc pipeline will decide timestamp zone by config of pipeline. I found > mysql2doris and mysql2starracks will specific datetime format > -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we > should't set fixed datetime format, just return LocalDateTime object. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34719) StreamRecordTest#testWithTimestamp fails on Azure
[
https://issues.apache.org/jira/browse/FLINK-34719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829492#comment-17829492
]
Matthias Pohl edited comment on FLINK-34719 at 3/21/24 11:02 AM:
-
These build failures didn't include the fix, yet:
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730546057#step:10:10174
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730544007#step:10:10045
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531437#step:10:10077
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747313654#step:10:10176
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747387684#step:10:10044
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747314118#step:10:10073
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764923155#step:10:10167
was (Author: mapohl):
These build failures didn't include the fix, yet:
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730546057#step:10:10174
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730544007#step:10:10045
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531437#step:10:10077
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747313654#step:10:10176
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747387684#step:10:10044
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747314118#step:10:10073
> StreamRecordTest#testWithTimestamp fails on Azure
> -
>
> Key: FLINK-34719
> URL: https://issues.apache.org/jira/browse/FLINK-34719
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Assignee: Jiabao Sun
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> The ClassCastException *message* expected in
> StreamRecordTest#testWithTimestamp as well as
> StreamRecordTest#testWithNoTimestamp fails on JDK 11, 17, and 21
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352&view=logs&j=f0ac5c25-1168-55a5-07ff-0e88223afed9&t=50bf7a25-bdc4-5e56-5478-c7b4511dde53&l=10341]
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352&view=logs&j=675bf62c-8558-587e-2555-dcad13acefb5&t=5878eed3-cc1e-5b12-1ed0-9e7139ce0992&l=9828]
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352&view=logs&j=d06b80b4-9e88-5d40-12a2-18072cf60528&t=609ecd5a-3f6e-5d0c-2239-2096b155a4d0&l=9833]
> {code:java}
> Expecting throwable message:
> Mar 16 01:35:07 "class
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord cannot be cast
> to class org.apache.flink.streaming.api.watermark.Watermark
> (org.apache.flink.streaming.runtime.streamrecord.StreamRecord and
> org.apache.flink.streaming.api.watermark.Watermark are in unnamed module of
> loader 'app')"
> Mar 16 01:35:07 to contain:
> Mar 16 01:35:07 "cannot be cast to
> org.apache.flink.streaming.api.watermark.Watermark"
> Mar 16 01:35:07 but did not.
> Mar 16 01:35:07
> Mar 16 01:35:07 Throwable that failed the check:
> Mar 16 01:35:07
> Mar 16 01:35:07 java.lang.ClassCastException: class
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord cannot be cast
> to class org.apache.flink.streaming.api.watermark.Watermark
> (org.apache.flink.streaming.runtime.streamrecord.StreamRecord and
> org.apache.flink.streaming.api.watermark.Watermark are in unnamed module of
> loader 'app')
> Mar 16 01:35:07 at
> org.apache.flink.streaming.runtime.streamrecord.StreamElement.asWatermark(StreamElement.java:92)
> Mar 16 01:35:07 at
> org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
> Mar 16 01:35:07 at
> org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:892)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34719) StreamRecordTest#testWithTimestamp fails on Azure
[
https://issues.apache.org/jira/browse/FLINK-34719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829492#comment-17829492
]
Matthias Pohl edited comment on FLINK-34719 at 3/21/24 11:03 AM:
-
These build failures didn't include the fix, yet:
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730546057#step:10:10174
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730544007#step:10:10045
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531437#step:10:10077
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747313654#step:10:10176
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747387684#step:10:10044
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747314118#step:10:10073
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764923155#step:10:10167
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764925256#step:10:10045
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764919890#step:10:10076
was (Author: mapohl):
These build failures didn't include the fix, yet:
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730546057#step:10:10174
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730544007#step:10:10045
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531437#step:10:10077
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747313654#step:10:10176
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747387684#step:10:10044
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747314118#step:10:10073
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764923155#step:10:10167
> StreamRecordTest#testWithTimestamp fails on Azure
> -
>
> Key: FLINK-34719
> URL: https://issues.apache.org/jira/browse/FLINK-34719
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Assignee: Jiabao Sun
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> The ClassCastException *message* expected in
> StreamRecordTest#testWithTimestamp as well as
> StreamRecordTest#testWithNoTimestamp fails on JDK 11, 17, and 21
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352&view=logs&j=f0ac5c25-1168-55a5-07ff-0e88223afed9&t=50bf7a25-bdc4-5e56-5478-c7b4511dde53&l=10341]
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352&view=logs&j=675bf62c-8558-587e-2555-dcad13acefb5&t=5878eed3-cc1e-5b12-1ed0-9e7139ce0992&l=9828]
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352&view=logs&j=d06b80b4-9e88-5d40-12a2-18072cf60528&t=609ecd5a-3f6e-5d0c-2239-2096b155a4d0&l=9833]
> {code:java}
> Expecting throwable message:
> Mar 16 01:35:07 "class
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord cannot be cast
> to class org.apache.flink.streaming.api.watermark.Watermark
> (org.apache.flink.streaming.runtime.streamrecord.StreamRecord and
> org.apache.flink.streaming.api.watermark.Watermark are in unnamed module of
> loader 'app')"
> Mar 16 01:35:07 to contain:
> Mar 16 01:35:07 "cannot be cast to
> org.apache.flink.streaming.api.watermark.Watermark"
> Mar 16 01:35:07 but did not.
> Mar 16 01:35:07
> Mar 16 01:35:07 Throwable that failed the check:
> Mar 16 01:35:07
> Mar 16 01:35:07 java.lang.ClassCastException: class
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord cannot be cast
> to class org.apache.flink.streaming.api.watermark.Watermark
> (org.apache.flink.streaming.runtime.streamrecord.StreamRecord and
> org.apache.flink.streaming.api.watermark.Watermark are in unnamed module of
> loader 'app')
> Mar 16 01:35:07 at
> org.apache.flink.streaming.runtime.streamrecord.StreamElement.asWatermark(StreamElement.java:92)
> Mar 16 01:35:07 at
> org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
> Mar 16 01:35:07 at
> org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:892)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34643) JobIDLoggingITCase failed
[
https://issues.apache.org/jira/browse/FLINK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829485#comment-17829485
]
Matthias Pohl edited comment on FLINK-34643 at 3/21/24 11:04 AM:
-
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764925776#step:10:8913
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764920830#step:10:8727
was (Author: mapohl):
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764925776#step:10:8913
> JobIDLoggingITCase failed
> -
>
> Key: FLINK-34643
> URL: https://issues.apache.org/jira/browse/FLINK-34643
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7897
> {code}
> Mar 09 01:24:23 01:24:23.498 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 4.209 s <<< FAILURE! -- in
> org.apache.flink.test.misc.JobIDLoggingITCase
> Mar 09 01:24:23 01:24:23.498 [ERROR]
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(ClusterClient)
> -- Time elapsed: 1.459 s <<< ERROR!
> Mar 09 01:24:23 java.lang.IllegalStateException: Too few log events recorded
> for org.apache.flink.runtime.jobmaster.JobMaster (12) - this must be a bug in
> the test code
> Mar 09 01:24:23 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.assertJobIDPresent(JobIDLoggingITCase.java:148)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(JobIDLoggingITCase.java:132)
> Mar 09 01:24:23 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 09 01:24:23 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Mar 09 01:24:23
> {code}
> The other test failures of this build were also caused by the same test:
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8349
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3&t=712ade8c-ca16-5b76-3acd-14df33bc1cb1&l=8209
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34643) JobIDLoggingITCase failed
[
https://issues.apache.org/jira/browse/FLINK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829485#comment-17829485
]
Matthias Pohl edited comment on FLINK-34643 at 3/21/24 11:04 AM:
-
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764925776#step:10:8913
was (Author: mapohl):
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
> JobIDLoggingITCase failed
> -
>
> Key: FLINK-34643
> URL: https://issues.apache.org/jira/browse/FLINK-34643
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7897
> {code}
> Mar 09 01:24:23 01:24:23.498 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 4.209 s <<< FAILURE! -- in
> org.apache.flink.test.misc.JobIDLoggingITCase
> Mar 09 01:24:23 01:24:23.498 [ERROR]
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(ClusterClient)
> -- Time elapsed: 1.459 s <<< ERROR!
> Mar 09 01:24:23 java.lang.IllegalStateException: Too few log events recorded
> for org.apache.flink.runtime.jobmaster.JobMaster (12) - this must be a bug in
> the test code
> Mar 09 01:24:23 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.assertJobIDPresent(JobIDLoggingITCase.java:148)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(JobIDLoggingITCase.java:132)
> Mar 09 01:24:23 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 09 01:24:23 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Mar 09 01:24:23
> {code}
> The other test failures of this build were also caused by the same test:
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8349
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3&t=712ade8c-ca16-5b76-3acd-14df33bc1cb1&l=8209
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34643) JobIDLoggingITCase failed
[
https://issues.apache.org/jira/browse/FLINK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829485#comment-17829485
]
Matthias Pohl edited comment on FLINK-34643 at 3/21/24 11:05 AM:
-
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764925776#step:10:8913
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764920830#step:10:8727
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764903331#step:10:9336
was (Author: mapohl):
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764925776#step:10:8913
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764920830#step:10:8727
> JobIDLoggingITCase failed
> -
>
> Key: FLINK-34643
> URL: https://issues.apache.org/jira/browse/FLINK-34643
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7897
> {code}
> Mar 09 01:24:23 01:24:23.498 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 4.209 s <<< FAILURE! -- in
> org.apache.flink.test.misc.JobIDLoggingITCase
> Mar 09 01:24:23 01:24:23.498 [ERROR]
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(ClusterClient)
> -- Time elapsed: 1.459 s <<< ERROR!
> Mar 09 01:24:23 java.lang.IllegalStateException: Too few log events recorded
> for org.apache.flink.runtime.jobmaster.JobMaster (12) - this must be a bug in
> the test code
> Mar 09 01:24:23 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.assertJobIDPresent(JobIDLoggingITCase.java:148)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(JobIDLoggingITCase.java:132)
> Mar 09 01:24:23 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 09 01:24:23 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Mar 09 01:24:23
> {code}
> The other test failures of this build were also caused by the same test:
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8349
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3&t=712ade8c-ca16-5b76-3acd-14df33bc1cb1&l=8209
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34908) [Feature][Pipeline] Mysql pipeline to doris and starrocks will lost precision for timestamp
[ https://issues.apache.org/jira/browse/FLINK-34908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829495#comment-17829495 ] Xin Gong commented on FLINK-34908: -- I will submit a PR. cc [~Leonard] > [Feature][Pipeline] Mysql pipeline to doris and starrocks will lost precision > for timestamp > --- > > Key: FLINK-34908 > URL: https://issues.apache.org/jira/browse/FLINK-34908 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Xin Gong >Priority: Minor > Fix For: cdc-3.1.0 > > > flink cdc pipeline will decide timestamp zone by config of pipeline. I found > mysql2doris and mysql2starracks will specific datetime format > -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we > should't set fixed datetime format, just return LocalDateTime object. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33186) CheckpointAfterAllTasksFinishedITCase.testRestoreAfterSomeTasksFinished fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-33186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829496#comment-17829496
]
Matthias Pohl commented on FLINK-33186:
---
https://github.com/apache/flink/actions/runs/8320416262/job/22765302151#step:10:7608
> CheckpointAfterAllTasksFinishedITCase.testRestoreAfterSomeTasksFinished
> fails on AZP
> -
>
> Key: FLINK-33186
> URL: https://issues.apache.org/jira/browse/FLINK-33186
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.19.0, 1.18.1
>Reporter: Sergey Nuyanzin
>Assignee: Jiang Xin
>Priority: Critical
> Labels: test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53509&view=logs&j=baf26b34-3c6a-54e8-f93f-cf269b32f802&t=8c9d126d-57d2-5a9e-a8c8-ff53f7b35cd9&l=8762
> fails as
> {noformat}
> Sep 28 01:23:43 Caused by:
> org.apache.flink.runtime.checkpoint.CheckpointException: Task local
> checkpoint failure.
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.PendingCheckpoint.abort(PendingCheckpoint.java:550)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2248)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2235)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:817)
> Sep 28 01:23:43 at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> Sep 28 01:23:43 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Sep 28 01:23:43 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> Sep 28 01:23:43 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> Sep 28 01:23:43 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> Sep 28 01:23:43 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> Sep 28 01:23:43 at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34643) JobIDLoggingITCase failed
[
https://issues.apache.org/jira/browse/FLINK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829485#comment-17829485
]
Matthias Pohl edited comment on FLINK-34643 at 3/21/24 11:06 AM:
-
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764925776#step:10:8913
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764920830#step:10:8727
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764903331#step:10:9336
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813901357#step:10:8952
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813876201#step:10:9327
was (Author: mapohl):
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764925776#step:10:8913
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764920830#step:10:8727
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764903331#step:10:9336
> JobIDLoggingITCase failed
> -
>
> Key: FLINK-34643
> URL: https://issues.apache.org/jira/browse/FLINK-34643
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7897
> {code}
> Mar 09 01:24:23 01:24:23.498 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 4.209 s <<< FAILURE! -- in
> org.apache.flink.test.misc.JobIDLoggingITCase
> Mar 09 01:24:23 01:24:23.498 [ERROR]
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(ClusterClient)
> -- Time elapsed: 1.459 s <<< ERROR!
> Mar 09 01:24:23 java.lang.IllegalStateException: Too few log events recorded
> for org.apache.flink.runtime.jobmaster.JobMaster (12) - this must be a bug in
> the test code
> Mar 09 01:24:23 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.assertJobIDPresent(JobIDLoggingITCase.java:148)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(JobIDLoggingITCase.java:132)
> Mar 09 01:24:23 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 09 01:24:23 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Mar 09 01:24:23
> {code}
> The other test failures of this build were also caused by the same test:
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8349
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3&t=712ade8c-ca16-5b76-3acd-14df33bc1cb1&l=8209
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34718) KeyedPartitionWindowedStream and NonPartitionWindowedStream IllegalStateException in AZP
[
https://issues.apache.org/jira/browse/FLINK-34718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829484#comment-17829484
]
Matthias Pohl edited comment on FLINK-34718 at 3/21/24 11:07 AM:
-
before the fix was committed to master:
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9329
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:8057
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:9345
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813876201#step:10:9330
was (Author: mapohl):
before the fix was committed to master:
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9329
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:8057
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:9345
> KeyedPartitionWindowedStream and NonPartitionWindowedStream
> IllegalStateException in AZP
>
>
> Key: FLINK-34718
> URL: https://issues.apache.org/jira/browse/FLINK-34718
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Assignee: Ryan Skraba
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58320&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=9646]
> 18 of the KeyedPartitionWindowedStreamITCase and
> NonKeyedPartitionWindowedStreamITCase unit tests introduced in FLINK-34543
> are failing in the adaptive scheduler profile, with errors similar to:
> {code:java}
> Mar 15 01:54:12 Caused by: java.lang.IllegalStateException: The adaptive
> scheduler supports pipelined data exchanges (violated by MapPartition
> (org.apache.flink.streaming.runtime.tasks.OneInputStreamTask) ->
> ddb598ad156ed281023ba4eebbe487e3).
> Mar 15 01:54:12 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.assertPreconditions(AdaptiveScheduler.java:438)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.(AdaptiveScheduler.java:356)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerFactory.createInstance(AdaptiveSchedulerFactory.java:124)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:121)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:384)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:361)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
> Mar 15 01:54:12 at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
> Mar 15 01:54:12 at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> Mar 15 01:54:12 ... 4 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34643) JobIDLoggingITCase failed
[
https://issues.apache.org/jira/browse/FLINK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829485#comment-17829485
]
Matthias Pohl edited comment on FLINK-34643 at 3/21/24 11:08 AM:
-
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764925776#step:10:8913
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764920830#step:10:8727
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764903331#step:10:9336
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813901357#step:10:8952
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813876201#step:10:9327
*
https://github.com/apache/flink/actions/runs/8352823788/job/22863786799#step:10:8952
was (Author: mapohl):
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764925776#step:10:8913
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764920830#step:10:8727
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764903331#step:10:9336
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813901357#step:10:8952
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813876201#step:10:9327
> JobIDLoggingITCase failed
> -
>
> Key: FLINK-34643
> URL: https://issues.apache.org/jira/browse/FLINK-34643
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7897
> {code}
> Mar 09 01:24:23 01:24:23.498 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 4.209 s <<< FAILURE! -- in
> org.apache.flink.test.misc.JobIDLoggingITCase
> Mar 09 01:24:23 01:24:23.498 [ERROR]
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(ClusterClient)
> -- Time elapsed: 1.459 s <<< ERROR!
> Mar 09 01:24:23 java.lang.IllegalStateException: Too few log events recorded
> for org.apache.flink.runtime.jobmaster.JobMaster (12) - this must be a bug in
> the test code
> Mar 09 01:24:23 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.assertJobIDPresent(JobIDLoggingITCase.java:148)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(JobIDLoggingITCase.java:132)
> Mar 09 01:24:23 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 09 01:24:23 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Mar 09 01:24:23
> {code}
> The other test failures of this build were also caused by the same test:
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8349
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3&t=712ade8c-ca16-5b76-3acd-14df33bc1cb1&l=8209
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34718) KeyedPartitionWindowedStream and NonPartitionWindowedStream IllegalStateException in AZP
[
https://issues.apache.org/jira/browse/FLINK-34718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829484#comment-17829484
]
Matthias Pohl edited comment on FLINK-34718 at 3/21/24 11:08 AM:
-
before the fix was committed to master:
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9329
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:8057
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:9345
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813876201#step:10:9330
*
https://github.com/apache/flink/actions/runs/8352823788/job/22863772571#step:10:9347
was (Author: mapohl):
before the fix was committed to master:
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9329
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:8057
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:9345
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813876201#step:10:9330
> KeyedPartitionWindowedStream and NonPartitionWindowedStream
> IllegalStateException in AZP
>
>
> Key: FLINK-34718
> URL: https://issues.apache.org/jira/browse/FLINK-34718
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Assignee: Ryan Skraba
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58320&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=9646]
> 18 of the KeyedPartitionWindowedStreamITCase and
> NonKeyedPartitionWindowedStreamITCase unit tests introduced in FLINK-34543
> are failing in the adaptive scheduler profile, with errors similar to:
> {code:java}
> Mar 15 01:54:12 Caused by: java.lang.IllegalStateException: The adaptive
> scheduler supports pipelined data exchanges (violated by MapPartition
> (org.apache.flink.streaming.runtime.tasks.OneInputStreamTask) ->
> ddb598ad156ed281023ba4eebbe487e3).
> Mar 15 01:54:12 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.assertPreconditions(AdaptiveScheduler.java:438)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.(AdaptiveScheduler.java:356)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerFactory.createInstance(AdaptiveSchedulerFactory.java:124)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:121)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:384)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:361)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
> Mar 15 01:54:12 at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
> Mar 15 01:54:12 at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
> Mar 15 01:54:12 at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> Mar 15 01:54:12 ... 4 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34643) JobIDLoggingITCase failed
[
https://issues.apache.org/jira/browse/FLINK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829485#comment-17829485
]
Matthias Pohl edited comment on FLINK-34643 at 3/21/24 11:08 AM:
-
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764925776#step:10:8913
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764920830#step:10:8727
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764903331#step:10:9336
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813901357#step:10:8952
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813876201#step:10:9327
*
https://github.com/apache/flink/actions/runs/8352823788/job/22863786799#step:10:8952
*
https://github.com/apache/flink/actions/runs/8352823788/job/22863772571#step:10:9337
was (Author: mapohl):
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764925776#step:10:8913
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764920830#step:10:8727
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764903331#step:10:9336
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813901357#step:10:8952
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813876201#step:10:9327
*
https://github.com/apache/flink/actions/runs/8352823788/job/22863786799#step:10:8952
> JobIDLoggingITCase failed
> -
>
> Key: FLINK-34643
> URL: https://issues.apache.org/jira/browse/FLINK-34643
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7897
> {code}
> Mar 09 01:24:23 01:24:23.498 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 4.209 s <<< FAILURE! -- in
> org.apache.flink.test.misc.JobIDLoggingITCase
> Mar 09 01:24:23 01:24:23.498 [ERROR]
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(ClusterClient)
> -- Time elapsed: 1.459 s <<< ERROR!
> Mar 09 01:24:23 java.lang.IllegalStateException: Too few log events recorded
> for org.apache.flink.runtime.jobmaster.JobMaster (12) - this must be a bug in
> the test code
> Mar 09 01:24:23 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.assertJobIDPresent(JobIDLoggingITCase.java:148)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(JobIDLoggingITCase.java:132)
> Mar 09 01:24:23 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 09 01:24:23 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Mar 09 01:24:23
> {code}
> The other test failures of this build were also caused by the same test:
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8349
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3&t=712ade8c-ca16-5b76-3acd-14df33bc1cb1&l=8209
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34227) Job doesn't disconnect from ResourceManager
[
https://issues.apache.org/jira/browse/FLINK-34227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829499#comment-17829499
]
Matthias Pohl commented on FLINK-34227:
---
https://github.com/apache/flink/actions/runs/8352823891/job/22863768994#step:10:12399
> Job doesn't disconnect from ResourceManager
> ---
>
> Key: FLINK-34227
> URL: https://issues.apache.org/jira/browse/FLINK-34227
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.19.0, 1.18.1
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Critical
> Labels: github-actions, pull-request-available, test-stability
> Attachments: FLINK-34227.7e7d69daebb438b8d03b7392c9c55115.log,
> FLINK-34227.log
>
>
> https://github.com/XComp/flink/actions/runs/7634987973/job/20800205972#step:10:14557
> {code}
> [...]
> "main" #1 prio=5 os_prio=0 tid=0x7f4b7000 nid=0x24ec0 waiting on
> condition [0x7fccce1eb000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xbdd52618> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2131)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2099)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2077)
> at
> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:876)
> at
> org.apache.flink.table.planner.runtime.stream.sql.WindowDistinctAggregateITCase.testHopWindow_Cube(WindowDistinctAggregateITCase.scala:550)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34227) Job doesn't disconnect from ResourceManager
[
https://issues.apache.org/jira/browse/FLINK-34227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829499#comment-17829499
]
Matthias Pohl edited comment on FLINK-34227 at 3/21/24 11:11 AM:
-
SetOperatorsITCase:
https://github.com/apache/flink/actions/runs/8352823891/job/22863768994#step:10:12399
was (Author: mapohl):
https://github.com/apache/flink/actions/runs/8352823891/job/22863768994#step:10:12399
> Job doesn't disconnect from ResourceManager
> ---
>
> Key: FLINK-34227
> URL: https://issues.apache.org/jira/browse/FLINK-34227
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.19.0, 1.18.1
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Critical
> Labels: github-actions, pull-request-available, test-stability
> Attachments: FLINK-34227.7e7d69daebb438b8d03b7392c9c55115.log,
> FLINK-34227.log
>
>
> https://github.com/XComp/flink/actions/runs/7634987973/job/20800205972#step:10:14557
> {code}
> [...]
> "main" #1 prio=5 os_prio=0 tid=0x7f4b7000 nid=0x24ec0 waiting on
> condition [0x7fccce1eb000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xbdd52618> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2131)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2099)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2077)
> at
> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:876)
> at
> org.apache.flink.table.planner.runtime.stream.sql.WindowDistinctAggregateITCase.testHopWindow_Cube(WindowDistinctAggregateITCase.scala:550)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28440) EventTimeWindowCheckpointingITCase failed with restore
[
https://issues.apache.org/jira/browse/FLINK-28440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829500#comment-17829500
]
Matthias Pohl commented on FLINK-28440:
---
https://github.com/apache/flink/actions/runs/8360441603/job/22886656534#step:10:7536
> EventTimeWindowCheckpointingITCase failed with restore
> --
>
> Key: FLINK-28440
> URL: https://issues.apache.org/jira/browse/FLINK-28440
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / State Backends
>Affects Versions: 1.16.0, 1.17.0, 1.18.0, 1.19.0
>Reporter: Huang Xingbo
>Assignee: Yanfei Lei
>Priority: Critical
> Labels: auto-deprioritized-critical, pull-request-available,
> stale-assigned, test-stability
> Fix For: 1.20.0
>
> Attachments: image-2023-02-01-00-51-54-506.png,
> image-2023-02-01-01-10-01-521.png, image-2023-02-01-01-19-12-182.png,
> image-2023-02-01-16-47-23-756.png, image-2023-02-01-16-57-43-889.png,
> image-2023-02-02-10-52-56-599.png, image-2023-02-03-10-09-07-586.png,
> image-2023-02-03-12-03-16-155.png, image-2023-02-03-12-03-56-614.png
>
>
> {code:java}
> Caused by: java.lang.Exception: Exception while creating
> StreamOperatorStateContext.
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:256)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268)
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:722)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:698)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:665)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:904)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for WindowOperator_0a448493b4782967b150582570326227_(2/4) from
> any of the 1 provided restore options.
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:353)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:165)
> ... 11 more
> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException:
> /tmp/junit1835099326935900400/junit1113650082510421526/52ee65b7-033f-4429-8ddd-adbe85e27ced
> (No such file or directory)
> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
> at
> org.apache.flink.runtime.state.changelog.StateChangelogHandleStreamHandleReader$1.advance(StateChangelogHandleStreamHandleReader.java:87)
> at
> org.apache.flink.runtime.state.changelog.StateChangelogHandleStreamHandleReader$1.hasNext(StateChangelogHandleStreamHandleReader.java:69)
> at
> org.apache.flink.state.changelog.restore.ChangelogBackendRestoreOperation.readBackendHandle(ChangelogBackendRestoreOperation.java:96)
> at
> org.apache.flink.state.changelog.restore.ChangelogBackendRestoreOperation.restore(ChangelogBackendRestoreOperation.java:75)
> at
> org.apache.flink.state.changelog.ChangelogStateBackend.restore(ChangelogStateBackend.java:92)
> at
> org.apache.flink.state.changelog.AbstractChangelogStateBackend.createKeyedStateBackend(AbstractChangelogStateBackend.java:136)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:336)
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
> ... 13 more
> Caused by: java.io.FileNotFoundException:
> /tmp/junit1835099326935900400/junit1113650082510421526/52ee65b7-03
[jira] [Commented] (FLINK-33186) CheckpointAfterAllTasksFinishedITCase.testRestoreAfterSomeTasksFinished fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-33186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829501#comment-17829501
]
Matthias Pohl commented on FLINK-33186:
---
https://github.com/apache/flink/actions/runs/8369823390/job/22916375709#step:10:7894
> CheckpointAfterAllTasksFinishedITCase.testRestoreAfterSomeTasksFinished
> fails on AZP
> -
>
> Key: FLINK-33186
> URL: https://issues.apache.org/jira/browse/FLINK-33186
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.19.0, 1.18.1
>Reporter: Sergey Nuyanzin
>Assignee: Jiang Xin
>Priority: Critical
> Labels: test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53509&view=logs&j=baf26b34-3c6a-54e8-f93f-cf269b32f802&t=8c9d126d-57d2-5a9e-a8c8-ff53f7b35cd9&l=8762
> fails as
> {noformat}
> Sep 28 01:23:43 Caused by:
> org.apache.flink.runtime.checkpoint.CheckpointException: Task local
> checkpoint failure.
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.PendingCheckpoint.abort(PendingCheckpoint.java:550)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2248)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2235)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:817)
> Sep 28 01:23:43 at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> Sep 28 01:23:43 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Sep 28 01:23:43 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> Sep 28 01:23:43 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> Sep 28 01:23:43 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> Sep 28 01:23:43 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> Sep 28 01:23:43 at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34643) JobIDLoggingITCase failed
[
https://issues.apache.org/jira/browse/FLINK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829485#comment-17829485
]
Matthias Pohl edited comment on FLINK-34643 at 3/21/24 11:13 AM:
-
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764925776#step:10:8913
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764920830#step:10:8727
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764903331#step:10:9336
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813901357#step:10:8952
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813876201#step:10:9327
*
https://github.com/apache/flink/actions/runs/8352823788/job/22863786799#step:10:8952
*
https://github.com/apache/flink/actions/runs/8352823788/job/22863772571#step:10:9337
*
https://github.com/apache/flink/actions/runs/8368626493/job/22913270846#step:10:8418
was (Author: mapohl):
*
https://github.com/apache/flink/actions/runs/8290287716/job/22688325865#step:10:9328
*
https://github.com/apache/flink/actions/runs/8304571223/job/22730531076#step:10:9194
*
https://github.com/apache/flink/actions/runs/8312246651/job/22747312383#step:10:8539
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764925776#step:10:8913
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764920830#step:10:8727
*
https://github.com/apache/flink/actions/runs/8320242443/job/22764903331#step:10:9336
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813901357#step:10:8952
*
https://github.com/apache/flink/actions/runs/8336454518/job/22813876201#step:10:9327
*
https://github.com/apache/flink/actions/runs/8352823788/job/22863786799#step:10:8952
*
https://github.com/apache/flink/actions/runs/8352823788/job/22863772571#step:10:9337
> JobIDLoggingITCase failed
> -
>
> Key: FLINK-34643
> URL: https://issues.apache.org/jira/browse/FLINK-34643
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7897
> {code}
> Mar 09 01:24:23 01:24:23.498 [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 4.209 s <<< FAILURE! -- in
> org.apache.flink.test.misc.JobIDLoggingITCase
> Mar 09 01:24:23 01:24:23.498 [ERROR]
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(ClusterClient)
> -- Time elapsed: 1.459 s <<< ERROR!
> Mar 09 01:24:23 java.lang.IllegalStateException: Too few log events recorded
> for org.apache.flink.runtime.jobmaster.JobMaster (12) - this must be a bug in
> the test code
> Mar 09 01:24:23 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.assertJobIDPresent(JobIDLoggingITCase.java:148)
> Mar 09 01:24:23 at
> org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(JobIDLoggingITCase.java:132)
> Mar 09 01:24:23 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 09 01:24:23 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 09 01:24:23 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Mar 09 01:24:23
> {code}
> The other test failures of this build were also caused by the same test:
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8349
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3&t=712ade8c-ca16-5b76-3acd-14df33bc1cb1&l=8209
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34731][runtime] Remove SpeculativeScheduler and incorporate its features into AdaptiveBatchScheduler. [flink]
JunRuiLee commented on code in PR #24524: URL: https://github.com/apache/flink/pull/24524#discussion_r1533688677 ## flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultSpeculativeExecutionHandler.java: ## @@ -7,62 +7,37 @@ * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * - * http://www.apache.org/licenses/LICENSE-2.0 + * http://www.apache.org/licenses/LICENSE-2.0 * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. Review Comment: Fixed. What happened was that I removed the original SpeculativeScheduler.class file and added a new DefaultSpeculativeExecutionHandler.class file. However, during the commit, these two actions were somehow combined into a single modify operation... 🤦♂️🤦♂️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34731][runtime] Remove SpeculativeScheduler and incorporate its features into AdaptiveBatchScheduler. [flink]
JunRuiLee commented on PR #24524: URL: https://github.com/apache/flink/pull/24524#issuecomment-2011980990 Thanks @zhuzhurk for the review, I've updated this pr, PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-34908) [Feature][Pipeline] Mysql pipeline to doris and starrocks will lost precision for timestamp
[ https://issues.apache.org/jira/browse/FLINK-34908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu reassigned FLINK-34908: -- Assignee: Xin Gong > [Feature][Pipeline] Mysql pipeline to doris and starrocks will lost precision > for timestamp > --- > > Key: FLINK-34908 > URL: https://issues.apache.org/jira/browse/FLINK-34908 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Xin Gong >Assignee: Xin Gong >Priority: Minor > Fix For: cdc-3.1.0 > > > flink cdc pipeline will decide timestamp zone by config of pipeline. I found > mysql2doris and mysql2starracks will specific datetime format > -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we > should't set fixed datetime format, just return LocalDateTime object. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] Add mysql-pipeline-connector with table.exclude.list option to exclud… [flink-cdc]
shiyiky opened a new pull request, #3186: URL: https://github.com/apache/flink-cdc/pull/3186 desc :[FLINK-34903](https://issues.apache.org/jira/browse/FLINK-34903) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34908) [Feature][Pipeline] Mysql pipeline to doris and starrocks will lost precision for timestamp
[ https://issues.apache.org/jira/browse/FLINK-34908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829507#comment-17829507 ] Leonard Xu commented on FLINK-34908: Thanks [~pacinogong], assigned this ticket to you. > [Feature][Pipeline] Mysql pipeline to doris and starrocks will lost precision > for timestamp > --- > > Key: FLINK-34908 > URL: https://issues.apache.org/jira/browse/FLINK-34908 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Xin Gong >Assignee: Xin Gong >Priority: Minor > Fix For: cdc-3.1.0 > > > flink cdc pipeline will decide timestamp zone by config of pipeline. I found > mysql2doris and mysql2starracks will specific datetime format > -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we > should't set fixed datetime format, just return LocalDateTime object. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34731][runtime] Remove SpeculativeScheduler and incorporate its features into AdaptiveBatchScheduler. [flink]
zhuzhurk commented on code in PR #24524:
URL: https://github.com/apache/flink/pull/24524#discussion_r1533719880
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java:
##
@@ -232,8 +230,8 @@ protected void startSchedulingInternal() {
}
});
-speculativeExecutionHandler.startSlowTaskDetector(
-getExecutionGraph(), getMainThreadExecutor());
+speculativeExecutionHandler.init(
Review Comment:
I prefer this to happen at the beginning of `startSchedulingInternal`, doing
initialization before triggering scheduling.
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchSchedulerFactory.java:
##
@@ -162,6 +147,74 @@ public SchedulerNG createInstance(
jobGraph.getName(),
jobGraph.getJobID());
+return createScheduler(
+log,
+jobGraph,
+ioExecutor,
+jobMasterConfiguration,
+futureExecutor,
+userCodeLoader,
+checkpointRecoveryFactory,
+rpcTimeout,
+blobWriter,
+jobManagerJobMetricGroup,
+shuffleMaster,
+partitionTracker,
+executionDeploymentTracker,
+initializationTimestamp,
+mainThreadExecutor,
+jobStatusListener,
+failureEnrichers,
+blocklistOperations,
+new DefaultExecutionOperations(),
+allocatorFactory,
+restartBackoffTimeStrategy,
+new ScheduledExecutorServiceAdapter(futureExecutor),
+DefaultVertexParallelismAndInputInfosDecider.from(
+getDefaultMaxParallelism(jobMasterConfiguration,
executionConfig),
+jobMasterConfiguration));
+}
+
+public AdaptiveBatchScheduler createScheduler(
Review Comment:
Better to be static and `@VisibleForTesting`.
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchSchedulerFactory.java:
##
@@ -162,6 +147,74 @@ public SchedulerNG createInstance(
jobGraph.getName(),
jobGraph.getJobID());
+return createScheduler(
+log,
+jobGraph,
+ioExecutor,
+jobMasterConfiguration,
+futureExecutor,
+userCodeLoader,
+checkpointRecoveryFactory,
+rpcTimeout,
+blobWriter,
+jobManagerJobMetricGroup,
+shuffleMaster,
+partitionTracker,
+executionDeploymentTracker,
+initializationTimestamp,
+mainThreadExecutor,
+jobStatusListener,
+failureEnrichers,
+blocklistOperations,
+new DefaultExecutionOperations(),
+allocatorFactory,
+restartBackoffTimeStrategy,
+new ScheduledExecutorServiceAdapter(futureExecutor),
+DefaultVertexParallelismAndInputInfosDecider.from(
+getDefaultMaxParallelism(jobMasterConfiguration,
executionConfig),
+jobMasterConfiguration));
+}
+
+public AdaptiveBatchScheduler createScheduler(
+Logger log,
+JobGraph jobGraph,
+Executor ioExecutor,
+Configuration jobMasterConfiguration,
+ScheduledExecutorService futureExecutor,
+ClassLoader userCodeLoader,
+CheckpointRecoveryFactory checkpointRecoveryFactory,
+Time rpcTimeout,
+BlobWriter blobWriter,
+JobManagerJobMetricGroup jobManagerJobMetricGroup,
+ShuffleMaster shuffleMaster,
+JobMasterPartitionTracker partitionTracker,
+ExecutionDeploymentTracker executionDeploymentTracker,
+long initializationTimestamp,
+ComponentMainThreadExecutor mainThreadExecutor,
+JobStatusListener jobStatusListener,
+Collection failureEnrichers,
+BlocklistOperations blocklistOperations,
+ExecutionOperations executionOperations,
+ExecutionSlotAllocatorFactory allocatorFactory,
+RestartBackoffTimeStrategy restartBackoffTimeStrategy,
+ScheduledExecutor delayExecutor,
+VertexParallelismAndInputInfosDecider
vertexParallelismAndInputInfosDecider)
+throws Exception {
+
+checkState(
+jobGraph.getJobType() == JobType.BATCH,
+"Adaptive batch scheduler only supports batch jobs");
+checkAllExchangesAreSupported(jobGraph);
+
+final boolean enableSpeculativeExecuti
[jira] [Commented] (FLINK-34903) Add mysql-pipeline-connector with table.exclude.list option to exclude unnecessary tables
[
https://issues.apache.org/jira/browse/FLINK-34903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829512#comment-17829512
]
Thorne commented on FLINK-34903:
i take a pr in https://github.com/apache/flink-cdc/pull/3186
> Add mysql-pipeline-connector with table.exclude.list option to exclude
> unnecessary tables
> ---
>
> Key: FLINK-34903
> URL: https://issues.apache.org/jira/browse/FLINK-34903
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
>Reporter: Thorne
>Priority: Major
> Labels: cdc
> Fix For: cdc-3.1.0
>
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> When using the MySQL Pipeline connector for whole-database
> synchronization, users currently cannot exclude unnecessary tables. Taking
> reference from Debezium's parameters, specifically the
> {*}table.exclude.list{*}, if the *table.include.list* is declared, then the
> *table.exclude.list* parameter will not take effect. However, the tables
> specified in the tables parameter of the MySQL Pipeline connector are
> effectively added to the *table.include.list* in Debezium's context.
> In summary, it is necessary to introduce an externally-exposed
> *table.exclude.list* parameter within the MySQL Pipeline connector to
> facilitate the exclusion of tables. This is because the current setup does
> not allow for excluding unnecessary tables when including others through the
> tables parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34898) Cannot create named STRUCT with a single field
[
https://issues.apache.org/jira/browse/FLINK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829513#comment-17829513
]
Feng Jin commented on FLINK-34898:
--
I tried this case and indeed encountered an error, but I managed to pass the
test by trying a different approach.
{code:java}
//代码占位符
-- The query is normal and results can be obtained.
Flink SQL> SELECT cast(ARRAY[ROW(1)] as ARRAY>);
[INFO] Result retrieval cancelled.
-- Got the exception
Flink SQL> SELECT ARRAY[cast(ROW(1) as ROW)];
[ERROR] Could not execute SQL statement. Reason:
java.lang.UnsupportedOperationException: class
org.apache.calcite.sql.SqlBasicCall: ROW(1)
{code}
I think this might indeed be a bug, we need to follow up and fix it.
[~chloehe] Can you help me modify the title and content of Jira? Please
provide the specific query and corresponding error message.
> Cannot create named STRUCT with a single field
> --
>
> Key: FLINK-34898
> URL: https://issues.apache.org/jira/browse/FLINK-34898
> Project: Flink
> Issue Type: Bug
>Reporter: Chloe He
>Priority: Major
> Attachments: image-2024-03-21-12-00-00-183.png
>
>
> I'm trying to create named structs using Flink SQL and I found a previous
> ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use
> of the following syntax:
> {code:java}
> SELECT CAST(('a', 1) as ROW) AS row1;
> {code}
> However, my named struct has a single field and effectively it should look
> something like `\{"a": 1}`. I can't seem to be able to find a way to
> construct this. I have experimented with a few different syntax and it either
> throws parsing error or casting error:
> {code:java}
> Cast function cannot convert value of type INTEGER to type
> RecordType(VARCHAR(2147483647) a) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34903) Add mysql-pipeline-connector with table.exclude.list option to exclude unnecessary tables
[
https://issues.apache.org/jira/browse/FLINK-34903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thorne updated FLINK-34903:
---
Description:
When using the MySQL Pipeline connector for whole-database synchronization,
users currently cannot exclude unnecessary tables. Taking reference from
Debezium's parameters, specifically the {*}table.exclude.list{*}, if the
*table.include.list* is declared, then the *table.exclude.list* parameter will
not take effect. However, the tables specified in the tables parameter of the
MySQL Pipeline connector are effectively added to the *table.include.list* in
Debezium's context.
!screenshot-1.png!
!screenshot-2.png|width=834,height=86!
debezium opthion desc
!screenshot-3.png|width=831,height=217!
In summary, it is necessary to introduce an externally-exposed
*table.exclude.list* parameter within the MySQL Pipeline connector to
facilitate the exclusion of tables. This is because the current setup does not
allow for excluding unnecessary tables when including others through the tables
parameter.
was:
When using the MySQL Pipeline connector for whole-database synchronization,
users currently cannot exclude unnecessary tables. Taking reference from
Debezium's parameters, specifically the {*}table.exclude.list{*}, if the
*table.include.list* is declared, then the *table.exclude.list* parameter will
not take effect. However, the tables specified in the tables parameter of the
MySQL Pipeline connector are effectively added to the *table.include.list* in
Debezium's context.
In summary, it is necessary to introduce an externally-exposed
*table.exclude.list* parameter within the MySQL Pipeline connector to
facilitate the exclusion of tables. This is because the current setup does not
allow for excluding unnecessary tables when including others through the tables
parameter.
> Add mysql-pipeline-connector with table.exclude.list option to exclude
> unnecessary tables
> ---
>
> Key: FLINK-34903
> URL: https://issues.apache.org/jira/browse/FLINK-34903
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
>Reporter: Thorne
>Priority: Major
> Labels: cdc
> Fix For: cdc-3.1.0
>
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> When using the MySQL Pipeline connector for whole-database
> synchronization, users currently cannot exclude unnecessary tables. Taking
> reference from Debezium's parameters, specifically the
> {*}table.exclude.list{*}, if the *table.include.list* is declared, then the
> *table.exclude.list* parameter will not take effect. However, the tables
> specified in the tables parameter of the MySQL Pipeline connector are
> effectively added to the *table.include.list* in Debezium's context.
> !screenshot-1.png!
> !screenshot-2.png|width=834,height=86!
> debezium opthion desc
> !screenshot-3.png|width=831,height=217!
> In summary, it is necessary to introduce an externally-exposed
> *table.exclude.list* parameter within the MySQL Pipeline connector to
> facilitate the exclusion of tables. This is because the current setup does
> not allow for excluding unnecessary tables when including others through the
> tables parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34898) Cannot create named STRUCT with a single field
[
https://issues.apache.org/jira/browse/FLINK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829513#comment-17829513
]
Feng Jin edited comment on FLINK-34898 at 3/21/24 11:51 AM:
I tried this case and indeed encountered an error, but I managed to pass the
test by trying a different approach.
{code:java}
//代码占位符
-- The query is normal and results can be obtained.
Flink SQL> SELECT cast(ARRAY[ROW(1)] as ARRAY>);
[INFO] Result retrieval cancelled.
-- Got the exception
Flink SQL> SELECT ARRAY[cast(ROW(1) as ROW)];
[ERROR] Could not execute SQL statement. Reason:
java.lang.UnsupportedOperationException: class
org.apache.calcite.sql.SqlBasicCall: ROW(1)
{code}
I think this might indeed be a bug, we need to follow up and fix it.
[~chloehe] Can you help to modify the title and content of Jira? Please
provide the specific query and corresponding error message.
was (Author: hackergin):
I tried this case and indeed encountered an error, but I managed to pass the
test by trying a different approach.
{code:java}
//代码占位符
-- The query is normal and results can be obtained.
Flink SQL> SELECT cast(ARRAY[ROW(1)] as ARRAY>);
[INFO] Result retrieval cancelled.
-- Got the exception
Flink SQL> SELECT ARRAY[cast(ROW(1) as ROW)];
[ERROR] Could not execute SQL statement. Reason:
java.lang.UnsupportedOperationException: class
org.apache.calcite.sql.SqlBasicCall: ROW(1)
{code}
I think this might indeed be a bug, we need to follow up and fix it.
[~chloehe] Can you help me modify the title and content of Jira? Please
provide the specific query and corresponding error message.
> Cannot create named STRUCT with a single field
> --
>
> Key: FLINK-34898
> URL: https://issues.apache.org/jira/browse/FLINK-34898
> Project: Flink
> Issue Type: Bug
>Reporter: Chloe He
>Priority: Major
> Attachments: image-2024-03-21-12-00-00-183.png
>
>
> I'm trying to create named structs using Flink SQL and I found a previous
> ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use
> of the following syntax:
> {code:java}
> SELECT CAST(('a', 1) as ROW) AS row1;
> {code}
> However, my named struct has a single field and effectively it should look
> something like `\{"a": 1}`. I can't seem to be able to find a way to
> construct this. I have experimented with a few different syntax and it either
> throws parsing error or casting error:
> {code:java}
> Cast function cannot convert value of type INTEGER to type
> RecordType(VARCHAR(2147483647) a) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34743][autoscaler] Memory tuning takes effect even if the parallelism isn't changed [flink-kubernetes-operator]
mxm commented on PR #799: URL: https://github.com/apache/flink-kubernetes-operator/pull/799#issuecomment-2012067487 > > Autoscaling wouldn't have a chance to realize its SLOs. > > You are right. Autoscaler supports scaling parallelism and memory for now. As I understand, the downtime cannot be guaranteed even if users only use scaling parallelism. For example, flink jobs don't use the Adaptive Scheduler and the input rate is always changed, then flink jobs will be scaled frequently. I agree that there are edge cases where the autoscaler cannot fulfill its service objectives. However, that doesn't mean we need to give up on them entirely. With restarts due to autotuning at any point in time, the autoscaling algorithm is inherently broken because downtime is never factored into the autoscaling decision. You mentioned the adaptive scheduler. Frankly, the use of the adaptive scheduler with autoscaling isn't fully developed. I would discourage users from using it with autoscaling at its current state. > Fortunately, scaling parallelism consider the restart time than scaling memory, and then increase some parallelisms. +1 > > > For this feature to be mergable, it will either have to be disabled by default (opt-in via config) > > IIUC, `job.autoscaler.memory.tuning.enabled` is disabled by default. It means the memory tuning is turned off by default even if this PR is merged, right? Autoscaling is also disabled by default. I think we want to make sure autoscaling and autotuning work together collaboratively. > > > or be integrated with autoscaling, i.e. figure out a way to balance tuning / autoscaling decisions and feed back tuning decisions to the autoscaling algorithm to scale up whenever we redeploy for memory changes to avoid falling behind and preventing autoscaling to scale up after downtime due to memory reconfigurations. > > The restartTime has been considered during `computeScalingSummary`, but we may ignore it due to the new parallelism is `WithinUtilizationTarget`. Do you mean we force adjust the parallelism to the new parallelism when scaling memory happens even if the new parallelism `WithinUtilizationTarget`? True, the rescale time has been considered for the downscale / upscale processing capacity, but the current processing capacity doesn't factor in downtime. Unplanned restarts would reduce the processing capacity. If we know we are going to restart, the autoscaling algorithm should factor this in, e.g. by reducing the calculated processing capacity accordingly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
