Re: [PR] [FLINK-33328] Allow TwoPhaseCommittingSink WithPreCommitTopology to alter the type of the Committable - mixin approach [flink]
leonardBang commented on code in PR #23912:
URL: https://github.com/apache/flink/pull/23912#discussion_r1433677957
##
flink-core/src/main/java/org/apache/flink/api/connector/sink2/TwoPhaseCommittingSink.java:
##
@@ -40,17 +37,9 @@
* @param The type of the committables.
*/
@PublicEvolving
-public interface TwoPhaseCommittingSink extends Sink {
-
-/**
- * Creates a {@link PrecommittingSinkWriter} that creates committables on
checkpoint or end of
- * input.
- *
- * @param context the runtime context.
- * @return A sink writer for the two-phase commit protocol.
- * @throws IOException for any failure during creation.
- */
-PrecommittingSinkWriter createWriter(InitContext context)
throws IOException;
Review Comment:
We'd better save the method which is better than `SinkWriter
createWriter(InitContext context)` as it return a `PrecommittingSinkWriter`
which contains more information than `SinkWriter`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33743][runtime] Support consuming multiple subpartitions on a single channel [flink]
yunfengzhou-hub commented on code in PR #23927:
URL: https://github.com/apache/flink/pull/23927#discussion_r1433659493
##
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java:
##
@@ -533,7 +533,7 @@ public void close() {
}
@Override
-public ResultSubpartitionView createSubpartitionView(
+protected ResultSubpartitionView createSubpartitionView(
Review Comment:
This is because the method is only used in internal implementations of
ResultPartition. The public interface has been modified to expose IndexSet
parameter instead of int.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33743][runtime] Support consuming multiple subpartitions on a single channel [flink]
yunfengzhou-hub commented on code in PR #23927:
URL: https://github.com/apache/flink/pull/23927#discussion_r1433656381
##
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultSubpartitionIndexSet.java:
##
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import java.io.Serializable;
+
+/** A collection of subpartition indexes. */
+public interface ResultSubpartitionIndexSet extends Iterable,
Serializable {
Review Comment:
I agree with it that we should reduce unnecessary inheritance. I'll defined
a method here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33877][streaming] Adjusting CollectSinkFunctionTest.testConfiguredPortIsUsed's assertion message to work with JDK 17 & 21 [flink]
Jiabao-Sun commented on PR #23976: URL: https://github.com/apache/flink/pull/23976#issuecomment-1865720524 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-27756) Fix intermittently failing test in AsyncSinkWriterTest.checkLoggedSendTimesAreWithinBounds
[
https://issues.apache.org/jira/browse/FLINK-27756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799285#comment-17799285
]
Jiabao Sun commented on FLINK-27756:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55774&view=logs&j=2e8cb2f7-b2d3-5c62-9c05-cd756d33a819&t=2dd510a3-5041-5201-6dc3-54d310f68906
> Fix intermittently failing test in
> AsyncSinkWriterTest.checkLoggedSendTimesAreWithinBounds
> --
>
> Key: FLINK-27756
> URL: https://issues.apache.org/jira/browse/FLINK-27756
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.15.0, 1.17.0, 1.19.0
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Critical
> Labels: test-stability
> Fix For: 1.16.0
>
>
> h2. Motivation
> - One of the integration tests ({{checkLoggedSendTimesAreWithinBounds}}) of
> {{AsyncSinkWriterTest}} has been reported to fail intermittently on build
> pipeline causing blocking of new changes.
> - Reporting build is [linked
> |https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=36009&view=logs&j=aa18c3f6-13b8-5f58-86bb-c1cffb239496&t=502fb6c0-30a2-5e49-c5c2-a00fa3acb203]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-33502) HybridShuffleITCase caused a fatal error
[
https://issues.apache.org/jira/browse/FLINK-33502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo closed FLINK-33502.
--
Resolution: Fixed
master(1.19) 593e443cd85962ffa512e26d1a900eccf5f705ca.
Feel free to reopen it if this fix doesn't work, but I hope this is the last
time we patch it. :)
> HybridShuffleITCase caused a fatal error
>
>
> Key: FLINK-33502
> URL: https://issues.apache.org/jira/browse/FLINK-33502
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Assignee: Wencong Liu
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
> Attachments: image-2023-11-20-14-37-37-321.png
>
>
> [https://github.com/XComp/flink/actions/runs/6789774296/job/18458197040#step:12:9177]
> {code:java}
> Error: 21:21:35 21:21:35.379 [ERROR] Error occurred in starting fork, check
> output in log
> 9168Error: 21:21:35 21:21:35.379 [ERROR] Process Exit Code: 239
> 9169Error: 21:21:35 21:21:35.379 [ERROR] Crashed tests:
> 9170Error: 21:21:35 21:21:35.379 [ERROR]
> org.apache.flink.test.runtime.HybridShuffleITCase
> 9171Error: 21:21:35 21:21:35.379 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> 9172Error: 21:21:35 21:21:35.379 [ERROR] Command was /bin/sh -c cd
> /root/flink/flink-tests && /usr/lib/jvm/jdk-11.0.19+7/bin/java -XX:+UseG1GC
> -Xms256m -XX:+IgnoreUnrecognizedVMOptions
> --add-opens=java.base/java.util=ALL-UNNAMED
> --add-opens=java.base/java.io=ALL-UNNAMED -Xmx1536m -jar
> /root/flink/flink-tests/target/surefire/surefirebooter10811559899200556131.jar
> /root/flink/flink-tests/target/surefire 2023-11-07T20-32-50_466-jvmRun4
> surefire6242806641230738408tmp surefire_1603959900047297795160tmp
> 9173Error: 21:21:35 21:21:35.379 [ERROR] Error occurred in starting fork,
> check output in log
> 9174Error: 21:21:35 21:21:35.379 [ERROR] Process Exit Code: 239
> 9175Error: 21:21:35 21:21:35.379 [ERROR] Crashed tests:
> 9176Error: 21:21:35 21:21:35.379 [ERROR]
> org.apache.flink.test.runtime.HybridShuffleITCase
> 9177Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
> 9178Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:479)
> 9179Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:322)
> 9180Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
> [...] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-25476][table-planner] support CHAR type in function MAX and MIN [flink]
xuyangzhong commented on code in PR #18375:
URL: https://github.com/apache/flink/pull/18375#discussion_r1433633656
##
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/AggregateITCase.scala:
##
@@ -1331,6 +1331,81 @@ class AggregateITCase(aggMode: AggMode, miniBatch:
MiniBatchMode, backend: State
assertThat(sink.getRetractResults.sorted).isEqualTo(expected.sorted)
}
+ @TestTemplate
+ def testMinMaxWithChar(): Unit = {
+val data =
+ List(
+rowOf(1, "a"),
+rowOf(1, "b"),
+rowOf(2, "d"),
+rowOf(2, "c")
+ )
+val dataId = TestValuesTableFactory.registerData(data)
+tEnv.executeSql(s"""
+ |CREATE TABLE src(
+ | `id` INT,
+ | `char` CHAR(1)
+ |) WITH (
+ | 'connector' = 'values',
+ | 'data-id' = '$dataId'
+ |)
+ |""".stripMargin)
+
+val sql =
+ """
+|select `id`, count(*), min(`char`), max(`char`) from src group by `id`
+ """.stripMargin
+
+val sink = new TestingRetractSink()
+tEnv.sqlQuery(sql).toRetractStream[Row].addSink(sink)
+env.execute()
+
+val expected = List("1,2,a,b", "2,2,c,d")
+assertThat(sink.getRetractResults.sorted).isEqualTo(expected.sorted)
+ }
+
+ @TestTemplate
+ def testRetractMinMaxWithChar(): Unit = {
+val data =
+ List(
+changelogRow("+I", Int.box(1), "a"),
+changelogRow("+I", Int.box(1), "b"),
+changelogRow("+I", Int.box(1), "c"),
+changelogRow("-D", Int.box(1), "c"),
+changelogRow("-D", Int.box(1), "a"),
+changelogRow("+I", Int.box(2), "a"),
+changelogRow("+I", Int.box(2), "b"),
+changelogRow("+I", Int.box(2), "c"),
+changelogRow("-U", Int.box(2), "b"),
+changelogRow("+U", Int.box(2), "d"),
+changelogRow("-U", Int.box(2), "a"),
+changelogRow("+U", Int.box(2), "b")
+ )
+val dataId = TestValuesTableFactory.registerData(data)
+tEnv.executeSql(s"""
+ |CREATE TABLE src(
+ | `id` INT,
+ | `char` CHAR(1)
+ |) WITH (
+ | 'connector' = 'values',
Review Comment:
I have updated the test to add a new field with type "char(2)", and test it
with data that actually has a length of 1 or 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33502][network] Refactor DiskIOScheduler to handle exceptions more gracefully and improve error handling logic [flink]
reswqa merged PR #23954: URL: https://github.com/apache/flink/pull/23954 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33502][network] Refactor DiskIOScheduler to handle exceptions more gracefully and improve error handling logic [flink]
reswqa commented on code in PR #23954:
URL: https://github.com/apache/flink/pull/23954#discussion_r1433628981
##
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/disk/DiskIOScheduler.java:
##
@@ -206,9 +206,9 @@ private int readBuffersFromFile() {
Queue buffers;
try {
buffers = allocateBuffers();
-} catch (Exception exception) {
-failScheduledReaders(scheduledReaders, exception);
-LOG.error("Failed to request buffers for data reading.",
exception);
+} catch (Exception e) {
Review Comment:
Alright, `BatchShuffleReadBufferPool#requestBuffers()` abuses the
declaration of ambiguous exception. It should be limited to it's own home, but
this indeed out of the scope of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33502][network] Refactor DiskIOScheduler to handle exceptions more gracefully and improve error handling logic [flink]
reswqa commented on code in PR #23954:
URL: https://github.com/apache/flink/pull/23954#discussion_r1433628981
##
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/disk/DiskIOScheduler.java:
##
@@ -206,9 +206,9 @@ private int readBuffersFromFile() {
Queue buffers;
try {
buffers = allocateBuffers();
-} catch (Exception exception) {
-failScheduledReaders(scheduledReaders, exception);
-LOG.error("Failed to request buffers for data reading.",
exception);
+} catch (Exception e) {
Review Comment:
Alright, `BatchShuffleReadBufferPool#requestBuffers()` abuses the
declaration of ambiguous exception. It should be limited to it's own home, but
this indeed out of the scope of this PR. So let's keep as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33502][network] Refactor DiskIOScheduler to handle exceptions more gracefully and improve error handling logic [flink]
WencongLiu commented on code in PR #23954:
URL: https://github.com/apache/flink/pull/23954#discussion_r1433606853
##
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/disk/DiskIOScheduler.java:
##
@@ -206,9 +206,9 @@ private int readBuffersFromFile() {
Queue buffers;
try {
buffers = allocateBuffers();
-} catch (Exception exception) {
-failScheduledReaders(scheduledReaders, exception);
-LOG.error("Failed to request buffers for data reading.",
exception);
+} catch (Exception e) {
Review Comment:
The `allocateBuffers()` method will call
`BatchShuffleReadBufferPool#requestBuffers()`, which throws `Exception`in the
method declaration.
Modifying the `BatchShuffleReadBufferPool#requestBuffers()` is a little
unnecessary. Catching the `Exception` is required here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33502][network] Refactor DiskIOScheduler to handle exceptions more gracefully and improve error handling logic [flink]
reswqa commented on code in PR #23954:
URL: https://github.com/apache/flink/pull/23954#discussion_r1433587339
##
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/disk/DiskIOScheduler.java:
##
@@ -206,9 +206,9 @@ private int readBuffersFromFile() {
Queue buffers;
try {
buffers = allocateBuffers();
-} catch (Exception exception) {
-failScheduledReaders(scheduledReaders, exception);
-LOG.error("Failed to request buffers for data reading.",
exception);
+} catch (Exception e) {
Review Comment:
Could we catch only `TimeoutException` here, other exceptions should be
unexpected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33440] Update Flink version matrix, add 1.18.0 and 1.19-SNAPSHOT to GH workflows [flink-connector-hbase]
ferenc-csaky commented on PR #35: URL: https://github.com/apache/flink-connector-hbase/pull/35#issuecomment-1865506167 cc @MartijnVisser @snuyanzin -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33856) Add metrics to monitor the interaction performance between task and external storage system in the process of checkpoint making
[ https://issues.apache.org/jira/browse/FLINK-33856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799272#comment-17799272 ] Weijie Guo commented on FLINK-33856: Yep, it seems feasible. I have assigned this to you. Before you start preparing for PR, I would like to ask [~masteryhx] to check this one. > Add metrics to monitor the interaction performance between task and external > storage system in the process of checkpoint making > --- > > Key: FLINK-33856 > URL: https://issues.apache.org/jira/browse/FLINK-33856 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.18.0 >Reporter: Jufang He >Priority: Major > > When Flink makes a checkpoint, the interaction performance with the external > file system has a great impact on the overall time-consuming. Therefore, it > is easy to observe the bottleneck point by adding performance indicators when > the task interacts with the external file storage system. These include: the > rate of file write , the latency to write the file, the latency to close the > file. > In flink side add the above metrics has the following advantages: convenient > statistical different task E2E time-consuming; do not need to distinguish the > type of external storage system, can be unified in the > FsCheckpointStreamFactory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33856) Add metrics to monitor the interaction performance between task and external storage system in the process of checkpoint making
[ https://issues.apache.org/jira/browse/FLINK-33856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo reassigned FLINK-33856: -- Assignee: Jufang He > Add metrics to monitor the interaction performance between task and external > storage system in the process of checkpoint making > --- > > Key: FLINK-33856 > URL: https://issues.apache.org/jira/browse/FLINK-33856 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.18.0 >Reporter: Jufang He >Assignee: Jufang He >Priority: Major > > When Flink makes a checkpoint, the interaction performance with the external > file system has a great impact on the overall time-consuming. Therefore, it > is easy to observe the bottleneck point by adding performance indicators when > the task interacts with the external file storage system. These include: the > rate of file write , the latency to write the file, the latency to close the > file. > In flink side add the above metrics has the following advantages: convenient > statistical different task E2E time-consuming; do not need to distinguish the > type of external storage system, can be unified in the > FsCheckpointStreamFactory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33856) Add metrics to monitor the interaction performance between task and external storage system in the process of checkpoint making
[ https://issues.apache.org/jira/browse/FLINK-33856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799254#comment-17799254 ] Jiang Xin commented on FLINK-33856: --- [~hejufang001] I'm sorry, I have no permission to assign but I will keep following up on this issue. [~Weijie Guo] Could you help assign this issue? > Add metrics to monitor the interaction performance between task and external > storage system in the process of checkpoint making > --- > > Key: FLINK-33856 > URL: https://issues.apache.org/jira/browse/FLINK-33856 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.18.0 >Reporter: Jufang He >Priority: Major > > When Flink makes a checkpoint, the interaction performance with the external > file system has a great impact on the overall time-consuming. Therefore, it > is easy to observe the bottleneck point by adding performance indicators when > the task interacts with the external file storage system. These include: the > rate of file write , the latency to write the file, the latency to close the > file. > In flink side add the above metrics has the following advantages: convenient > statistical different task E2E time-consuming; do not need to distinguish the > type of external storage system, can be unified in the > FsCheckpointStreamFactory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] FLINK-25029: Hadoop Caller Context Setting in Flink [flink]
liangyu-1 commented on PR #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-1865455819 Is there anyone still working on this issue? I met the same problem in my scenario. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33856) Add metrics to monitor the interaction performance between task and external storage system in the process of checkpoint making
[ https://issues.apache.org/jira/browse/FLINK-33856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799247#comment-17799247 ] Jufang He commented on FLINK-33856: --- [~Jiang Xin] Yes, this metrics optimization has been implemented in our company, which is very helpful for locating checkpoint bottlenecks. Now I want to contribute to the community, can you assign this to me? > Add metrics to monitor the interaction performance between task and external > storage system in the process of checkpoint making > --- > > Key: FLINK-33856 > URL: https://issues.apache.org/jira/browse/FLINK-33856 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.18.0 >Reporter: Jufang He >Priority: Major > > When Flink makes a checkpoint, the interaction performance with the external > file system has a great impact on the overall time-consuming. Therefore, it > is easy to observe the bottleneck point by adding performance indicators when > the task interacts with the external file storage system. These include: the > rate of file write , the latency to write the file, the latency to close the > file. > In flink side add the above metrics has the following advantages: convenient > statistical different task E2E time-consuming; do not need to distinguish the > type of external storage system, can be unified in the > FsCheckpointStreamFactory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33490) Validate the name conflicts when creating view
[ https://issues.apache.org/jira/browse/FLINK-33490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shengkai Fang closed FLINK-33490. - Resolution: Fixed > Validate the name conflicts when creating view > -- > > Key: FLINK-33490 > URL: https://issues.apache.org/jira/browse/FLINK-33490 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.19.0 >Reporter: Shengkai Fang >Assignee: xuyang >Priority: Major > Labels: pull-request-available > > We should forbid > ``` > CREATE VIEW id_view AS > SELECT id, uid AS id FROM id_table > ``` > As the SQL standards states, > If is specified, then: > i) If any two columns in the table specified by the have > equivalent s, or if any column of that table has an > implementation-dependent name, then a shall be specified. > ii) Equivalent s shall not be specified more than once in the > . > Many databases also throw exception when view name conflicts, e.g. mysql, > postgres. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33490) Validate the name conflicts when creating view
[ https://issues.apache.org/jira/browse/FLINK-33490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799240#comment-17799240 ] Shengkai Fang commented on FLINK-33490: --- Merged into master: a4fe01cb9d678b293107b0a6278fec1a749913cc > Validate the name conflicts when creating view > -- > > Key: FLINK-33490 > URL: https://issues.apache.org/jira/browse/FLINK-33490 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.19.0 >Reporter: Shengkai Fang >Assignee: xuyang >Priority: Major > Labels: pull-request-available > > We should forbid > ``` > CREATE VIEW id_view AS > SELECT id, uid AS id FROM id_table > ``` > As the SQL standards states, > If is specified, then: > i) If any two columns in the table specified by the have > equivalent s, or if any column of that table has an > implementation-dependent name, then a shall be specified. > ii) Equivalent s shall not be specified more than once in the > . > Many databases also throw exception when view name conflicts, e.g. mysql, > postgres. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33490][table-planner] Validate the column name conflicts in view when creating view [flink]
fsk119 merged PR #23760: URL: https://github.com/apache/flink/pull/23760 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33743][runtime] Support consuming multiple subpartitions on a single channel [flink]
TanYuxin-tyx commented on code in PR #23927:
URL: https://github.com/apache/flink/pull/23927#discussion_r1432220667
##
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/Buffer.java:
##
@@ -304,7 +306,15 @@ enum DataType {
RECOVERY_COMPLETION(false, true, true, false, false),
/** {@link #END_OF_SEGMENT} indicates that a segment is finished in a
subpartition. */
-END_OF_SEGMENT(false, true, false, false, false);
+END_OF_SEGMENT(false, true, false, false, false),
+
+/**
+ * {@link #END_OF_DATA} indicates that there will be no more data
buffer in a subpartition.
+ */
+END_OF_DATA(false, true, false, false, false),
Review Comment:
We already had `EndOfData` and `EndOfPartitionEvent` events, and why not
reuse them here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-33897) Allow triggering unaligned checkpoint via CLI
[
https://issues.apache.org/jira/browse/FLINK-33897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799233#comment-17799233
]
Zakelly Lan commented on FLINK-33897:
-
This also require the {{SingleCheckpointBarrierHandler}} changing from aligned
to unaligned state when receiving an unaligned barrier. Would like to hear your
thoughts [~pnowojski] [~dwysakowicz]
> Allow triggering unaligned checkpoint via CLI
> -
>
> Key: FLINK-33897
> URL: https://issues.apache.org/jira/browse/FLINK-33897
> Project: Flink
> Issue Type: Improvement
> Components: Command Line Client, Runtime / Checkpointing
>Reporter: Zakelly Lan
>Assignee: Zakelly Lan
>Priority: Major
>
> After FLINK-6755, user could trigger checkpoint through CLI. However I
> noticed there would be value supporting trigger it in unaligned way, since
> the job may encounter a high back-pressure and an aligned checkpoint would
> fail.
>
> I suggest we provide an option '-unaligned' in CLI to support that.
>
> Similar option would also be useful for REST api
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33881) [TtlListState]Avoid copy and update value in TtlListState#getUnexpiredOrNull
[
https://issues.apache.org/jira/browse/FLINK-33881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799232#comment-17799232
]
Zakelly Lan commented on FLINK-33881:
-
[~lijinzhong] Thus I have no problem. And actually I think there may be some
value if we could make sure it is safe to do shallow copy.
> [TtlListState]Avoid copy and update value in TtlListState#getUnexpiredOrNull
>
>
> Key: FLINK-33881
> URL: https://issues.apache.org/jira/browse/FLINK-33881
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Jinzhong Li
>Priority: Minor
> Attachments: image-2023-12-19-21-25-21-446.png,
> image-2023-12-19-21-26-43-518.png
>
>
> In some scenarios, 'TtlListState#getUnexpiredOrNull ->
> elementSerializer.copy(ttlValue)' consumes a lot of cpu resources.
> !image-2023-12-19-21-25-21-446.png|width=529,height=119!
> I found that for TtlListState#getUnexpiredOrNull， if none of the elements
> have expired, it still needs to copy all the elements and update the whole
> list/map in TtlIncrementalCleanup#runCleanup();
> !image-2023-12-19-21-26-43-518.png|width=505,height=266!
> I think we could optimize TtlListState#getUnexpiredOrNull by:
> 1）find the first expired element index in the list;
> 2）If not found, return to the original list;
> 3）If found, then constrct the unexpire list (puts the previous elements into
> the list), and go through the subsequent elements, adding expired elements
> into the list.
> {code:java}
> public List> getUnexpiredOrNull(@Nonnull List>
> ttlValues) {
> //...
> int firstExpireIndex = -1;
> for (int i = 0; i < ttlValues.size(); i++) {
> if (TtlUtils.expired(ttlValues.get(i), ttl, currentTimestamp)) {
> firstExpireIndex = i;
> break;
> }
> }
> if (firstExpireIndex == -1) {
> return ttlValues; //return the original ttlValues
> }
> List> unexpired = new ArrayList<>(ttlValues.size());
> for (int i = 0; i < ttlValues.size(); i++) {
> if (i < firstExpireIndex) {
> // unexpired.add(ttlValues.get(i));
> unexpired.add(elementSerializer.copy(ttlValues.get(i)));
> }
> if (i > firstExpireIndex) {
> if (!TtlUtils.expired(ttlValues.get(i), ttl, currentTimestamp)) {
> // unexpired.add(ttlValues.get(i));
> unexpired.add(elementSerializer.copy(ttlValues.get(i)));
> }
> }
> }
> // .
> } {code}
> *In this way, the extra iteration overhead is actually very very small, but
> the benefit when there are no expired elements is significant.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33881) [TtlListState]Avoid copy and update value in TtlListState#getUnexpiredOrNull
[
https://issues.apache.org/jira/browse/FLINK-33881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799232#comment-17799232
]
Zakelly Lan edited comment on FLINK-33881 at 12/21/23 2:28 AM:
---
[~lijinzhong] Thus I have no question. And actually I think there may be some
value if we could make sure it is safe to do shallow copy.
was (Author: zakelly):
[~lijinzhong] Thus I have no problem. And actually I think there may be some
value if we could make sure it is safe to do shallow copy.
> [TtlListState]Avoid copy and update value in TtlListState#getUnexpiredOrNull
>
>
> Key: FLINK-33881
> URL: https://issues.apache.org/jira/browse/FLINK-33881
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Jinzhong Li
>Priority: Minor
> Attachments: image-2023-12-19-21-25-21-446.png,
> image-2023-12-19-21-26-43-518.png
>
>
> In some scenarios, 'TtlListState#getUnexpiredOrNull ->
> elementSerializer.copy(ttlValue)' consumes a lot of cpu resources.
> !image-2023-12-19-21-25-21-446.png|width=529,height=119!
> I found that for TtlListState#getUnexpiredOrNull， if none of the elements
> have expired, it still needs to copy all the elements and update the whole
> list/map in TtlIncrementalCleanup#runCleanup();
> !image-2023-12-19-21-26-43-518.png|width=505,height=266!
> I think we could optimize TtlListState#getUnexpiredOrNull by:
> 1）find the first expired element index in the list;
> 2）If not found, return to the original list;
> 3）If found, then constrct the unexpire list (puts the previous elements into
> the list), and go through the subsequent elements, adding expired elements
> into the list.
> {code:java}
> public List> getUnexpiredOrNull(@Nonnull List>
> ttlValues) {
> //...
> int firstExpireIndex = -1;
> for (int i = 0; i < ttlValues.size(); i++) {
> if (TtlUtils.expired(ttlValues.get(i), ttl, currentTimestamp)) {
> firstExpireIndex = i;
> break;
> }
> }
> if (firstExpireIndex == -1) {
> return ttlValues; //return the original ttlValues
> }
> List> unexpired = new ArrayList<>(ttlValues.size());
> for (int i = 0; i < ttlValues.size(); i++) {
> if (i < firstExpireIndex) {
> // unexpired.add(ttlValues.get(i));
> unexpired.add(elementSerializer.copy(ttlValues.get(i)));
> }
> if (i > firstExpireIndex) {
> if (!TtlUtils.expired(ttlValues.get(i), ttl, currentTimestamp)) {
> // unexpired.add(ttlValues.get(i));
> unexpired.add(elementSerializer.copy(ttlValues.get(i)));
> }
> }
> }
> // .
> } {code}
> *In this way, the extra iteration overhead is actually very very small, but
> the benefit when there are no expired elements is significant.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-33881) [TtlListState]Avoid copy and update value in TtlListState#getUnexpiredOrNull
[
https://issues.apache.org/jira/browse/FLINK-33881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zakelly Lan reassigned FLINK-33881:
---
Assignee: Jinzhong Li
> [TtlListState]Avoid copy and update value in TtlListState#getUnexpiredOrNull
>
>
> Key: FLINK-33881
> URL: https://issues.apache.org/jira/browse/FLINK-33881
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Jinzhong Li
>Assignee: Jinzhong Li
>Priority: Minor
> Attachments: image-2023-12-19-21-25-21-446.png,
> image-2023-12-19-21-26-43-518.png
>
>
> In some scenarios, 'TtlListState#getUnexpiredOrNull ->
> elementSerializer.copy(ttlValue)' consumes a lot of cpu resources.
> !image-2023-12-19-21-25-21-446.png|width=529,height=119!
> I found that for TtlListState#getUnexpiredOrNull， if none of the elements
> have expired, it still needs to copy all the elements and update the whole
> list/map in TtlIncrementalCleanup#runCleanup();
> !image-2023-12-19-21-26-43-518.png|width=505,height=266!
> I think we could optimize TtlListState#getUnexpiredOrNull by:
> 1）find the first expired element index in the list;
> 2）If not found, return to the original list;
> 3）If found, then constrct the unexpire list (puts the previous elements into
> the list), and go through the subsequent elements, adding expired elements
> into the list.
> {code:java}
> public List> getUnexpiredOrNull(@Nonnull List>
> ttlValues) {
> //...
> int firstExpireIndex = -1;
> for (int i = 0; i < ttlValues.size(); i++) {
> if (TtlUtils.expired(ttlValues.get(i), ttl, currentTimestamp)) {
> firstExpireIndex = i;
> break;
> }
> }
> if (firstExpireIndex == -1) {
> return ttlValues; //return the original ttlValues
> }
> List> unexpired = new ArrayList<>(ttlValues.size());
> for (int i = 0; i < ttlValues.size(); i++) {
> if (i < firstExpireIndex) {
> // unexpired.add(ttlValues.get(i));
> unexpired.add(elementSerializer.copy(ttlValues.get(i)));
> }
> if (i > firstExpireIndex) {
> if (!TtlUtils.expired(ttlValues.get(i), ttl, currentTimestamp)) {
> // unexpired.add(ttlValues.get(i));
> unexpired.add(elementSerializer.copy(ttlValues.get(i)));
> }
> }
> }
> // .
> } {code}
> *In this way, the extra iteration overhead is actually very very small, but
> the benefit when there are no expired elements is significant.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
flashJd commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1433376746
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##
@@ -264,4 +262,24 @@ private static Map
getVertexParallelismOverrides(
});
return overrides;
}
+
+private boolean blockScalingExecution(
+Context context,
+Map scalingSummaries,
+Configuration conf,
+Instant now) {
+var scaleEnabled = conf.get(SCALING_ENABLED);
+var isExcluded = CalendarUtils.inExcludedPeriods(conf, now);
+autoScalerEventHandler.handleScalingEvent(
+context,
+scalingSummaries,
+scaleEnabled,
+isExcluded,
+conf.get(SCALING_EVENT_INTERVAL));
+
+if (!scaleEnabled || isExcluded) {
+return true;
+}
+return false;
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33877][streaming] Adjusting CollectSinkFunctionTest.testConfiguredPortIsUsed's assertion message to work with JDK 21 [flink]
flinkbot commented on PR #23976: URL: https://github.com/apache/flink/pull/23976#issuecomment-1865386058 ## CI report: * 20523a5f4595ea63effa59fb209cf610071c548f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33877][streaming] Adjusting CollectSinkFunctionTest.testConfiguredPortIsUsed's assertion message to work with JDK 21 [flink]
Jiabao-Sun commented on PR #23976: URL: https://github.com/apache/flink/pull/23976#issuecomment-1865385787 Hi @snuyanzin, please take a look. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33877) CollectSinkFunctionTest.testConfiguredPortIsUsed fails due to BindException
[
https://issues.apache.org/jira/browse/FLINK-33877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799231#comment-17799231
]
Jiabao Sun commented on FLINK-33877:
Thanks [~Sergey Nuyanzin] to report this.
I noticed that it will fail only when JDK21, maybe because JDK21 has changed
the error message, lacking (Bind failed).
{noformat}
Dec 20 04:05:36 Expecting throwable message:
Dec 20 04:05:36 "Address already in use"
Dec 20 04:05:36 to contain:
Dec 20 04:05:36 "Address already in use (Bind failed)"
Dec 20 04:05:36 but did not.
{noformat}
> CollectSinkFunctionTest.testConfiguredPortIsUsed fails due to BindException
> ---
>
> Key: FLINK-33877
> URL: https://issues.apache.org/jira/browse/FLINK-33877
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.19.0
>Reporter: Jiabao Sun
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55646&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=24c3384f-1bcb-57b3-224f-51bf973bbee8&l=9482
> {noformat}
> Dec 18 17:49:57 17:49:57.241 [ERROR]
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunctionTest.testConfiguredPortIsUsed
> -- Time elapsed: 0.021 s <<< ERROR!
> Dec 18 17:49:57 java.net.BindException: Address already in use (Bind failed)
> Dec 18 17:49:57 at java.net.PlainSocketImpl.socketBind(Native Method)
> Dec 18 17:49:57 at
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
> Dec 18 17:49:57 at java.net.ServerSocket.bind(ServerSocket.java:390)
> Dec 18 17:49:57 at java.net.ServerSocket.(ServerSocket.java:252)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction$ServerThread.(CollectSinkFunction.java:375)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction$ServerThread.(CollectSinkFunction.java:362)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction.open(CollectSinkFunction.java:252)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.utils.CollectSinkFunctionTestWrapper.openFunction(CollectSinkFunctionTestWrapper.java:103)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunctionTest.testConfiguredPortIsUsed(CollectSinkFunctionTest.java:138)
> Dec 18 17:49:57 at java.lang.reflect.Method.invoke(Method.java:498)
> Dec 18 17:49:57 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-33877][streaming] Adjusting CollectSinkFunctionTest.testConfiguredPortIsUsed's assertion message to work with JDK 21 [flink]
Jiabao-Sun opened a new pull request, #23976: URL: https://github.com/apache/flink/pull/23976 ## What is the purpose of the change [FLINK-33877][streaming] Adjusting CollectSinkFunctionTest.testConfiguredPortIsUsed's assertion message to work with JDK 21 https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=675bf62c-8558-587e-2555-dcad13acefb5&t=5878eed3-cc1e-5b12-1ed0-9e7139ce0992&l=9924 ``` Dec 20 04:05:36 [Any cause is instance of class 'class java.net.BindException' and contains message 'Address already in use (Bind failed)'] Dec 20 04:05:36 Expecting any element of: Dec 20 04:05:36 [java.net.BindException: Address already in use Dec 20 04:05:36 at java.base/sun.nio.ch.Net.bind0(Native Method) Dec 20 04:05:36 at java.base/sun.nio.ch.Net.bind(Net.java:555) Dec 20 04:05:36 at java.base/sun.nio.ch.Net.bind(Net.java:544) Dec 20 04:05:36 ...(71 remaining lines not displayed - this can be changed with Assertions.setMaxStackTraceElementsDisplayed)] Dec 20 04:05:36 to satisfy the given assertions requirements but none did: Dec 20 04:05:36 Dec 20 04:05:36 java.net.BindException: Address already in use Dec 20 04:05:36 at java.base/sun.nio.ch.Net.bind0(Native Method) Dec 20 04:05:36 at java.base/sun.nio.ch.Net.bind(Net.java:555) Dec 20 04:05:36 at java.base/sun.nio.ch.Net.bind(Net.java:544) Dec 20 04:05:36 ...(71 remaining lines not displayed - this can be changed with Assertions.setMaxStackTraceElementsDisplayed) Dec 20 04:05:36 error: Dec 20 04:05:36 Expecting throwable message: Dec 20 04:05:36 "Address already in use" Dec 20 04:05:36 to contain: Dec 20 04:05:36 "Address already in use (Bind failed)" Dec 20 04:05:36 but did not. ``` ## Verifying this change This change is already covered by existing tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33880) Introducing Retry Mechanism for Listing TaskManager Pods to Prevent API Server Connection Failures
[ https://issues.apache.org/jira/browse/FLINK-33880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Huang updated FLINK-33880: Attachment: image-2023-12-21-10-12-37-667.png Description: When operating in Kubernetes mode, if the JobManager undergoes a restart, it attempts to establish a connection with the API server to retrieve the complete list of TaskManager Pods, facilitating the recovery of previous TaskManagers. In the context of a large Kubernetes cluster with potentially thousands of concurrently running jobs, a scenario may arise where all JobManagers undergo a restart and subsequently connect to the API server (e.g., during disaster recovery). This influx of requests may overwhelm the API server, reaching its maximum capacity and leading to the refusal of some JobManager requests. Consequently, certain JobManagers may experience failures and initiate reconnection attempts to the API server. !image-2023-12-21-10-12-37-667.png|width=609,height=305! !image-2023-12-19-18-44-13-623.png|width=505,height=206! To enhance this process, we can propose the implementation of a retry mechanism. In the event of a failed connection attempt to the API server, Flink will introduce a waiting period before making subsequent connection attempts, mitigating the risk of overwhelming the server and improving the overall resilience of the system. was: When operating in Kubernetes mode, if the JobManager undergoes a restart, it attempts to establish a connection with the API server to retrieve the complete list of TaskManager Pods, facilitating the recovery of previous TaskManagers. In the context of a large Kubernetes cluster with potentially thousands of concurrently running jobs, a scenario may arise where all JobManagers undergo a restart and subsequently connect to the API server (e.g., during disaster recovery). This influx of requests may overwhelm the API server, reaching its maximum capacity and leading to the refusal of some JobManager requests. Consequently, certain JobManagers may experience failures and initiate reconnection attempts to the API server. !image-2023-12-19-18-44-13-623.png|width=505,height=206! To enhance this process, we can propose the implementation of a retry mechanism. In the event of a failed connection attempt to the API server, Flink will introduce a waiting period before making subsequent connection attempts, mitigating the risk of overwhelming the server and improving the overall resilience of the system. > Introducing Retry Mechanism for Listing TaskManager Pods to Prevent API > Server Connection Failures > -- > > Key: FLINK-33880 > URL: https://issues.apache.org/jira/browse/FLINK-33880 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Affects Versions: 1.17.2 >Reporter: Yuan Huang >Priority: Major > Attachments: image-2023-12-19-18-41-41-308.png, > image-2023-12-19-18-44-13-623.png, image-2023-12-21-10-12-37-667.png > > > When operating in Kubernetes mode, if the JobManager undergoes a restart, it > attempts to establish a connection with the API server to retrieve the > complete list of TaskManager Pods, facilitating the recovery of previous > TaskManagers. > In the context of a large Kubernetes cluster with potentially thousands of > concurrently running jobs, a scenario may arise where all JobManagers undergo > a restart and subsequently connect to the API server (e.g., during disaster > recovery). This influx of requests may overwhelm the API server, reaching its > maximum capacity and leading to the refusal of some JobManager requests. > Consequently, certain JobManagers may experience failures and initiate > reconnection attempts to the API server. > !image-2023-12-21-10-12-37-667.png|width=609,height=305! > !image-2023-12-19-18-44-13-623.png|width=505,height=206! > To enhance this process, we can propose the implementation of a retry > mechanism. In the event of a failed connection attempt to the API server, > Flink will introduce a waiting period before making subsequent connection > attempts, mitigating the risk of overwhelming the server and improving the > overall resilience of the system. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33879] Avoids the potential hang of Hybrid Shuffle during redistribution [flink]
jiangxin369 commented on PR #23957: URL: https://github.com/apache/flink/pull/23957#issuecomment-1865364340 @TanYuxin-tyx Could you help review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33788) Table not found for catalogname is capital
[
https://issues.apache.org/jira/browse/FLINK-33788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799223#comment-17799223
]
luoyuxia commented on FLINK-33788:
--
[~shaoguang] What's the problem you meet that casue the push fail?
> Table not found for catalogname is capital
> --
>
> Key: FLINK-33788
> URL: https://issues.apache.org/jira/browse/FLINK-33788
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.16.1
>Reporter: lishaoguang
>Assignee: lishaoguang
>Priority: Major
> Attachments: image-2023-12-09-20-26-41-150.png, screenshot-1.png,
> screenshot-2.png
>
>
> {code:java}
> String runMode=args[0];
> String sql = "select * from student";
> String sql = "insert into table student select * from student;";
> String sql = args[1];
> String catalogName = "myHive";
> String defaultDatabase = args[2];
> RuntimeExecutionMode executionMode = "batch".equals(runMode) ?
> RuntimeExecutionMode.BATCH : RuntimeExecutionMode.STREAMING;
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setRuntimeMode(executionMode);
> StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
>
> tenv.getConfig().setSqlDialect(SqlDialect.HIVE);
> Configuration conf = new Configuration();
> conf.addResource("/ect/hadoop/conf");
> conf.addResource("/ect/hive/conf");
> HiveConf hiveConf = new HiveConf(conf, Configuration.class);
> String hiveVersion = "3.1.3";
> HiveCatalog hiveCatalog = new HiveCatalog(catalogName,
> defaultDatabase, hiveConf, hiveVersion, true);
> tenv.registerCatalog(catalogName, hiveCatalog);
> tenv.useCatalog(catalogName);
> tenv.useDatabase(defaultDatabase);
> tenv.executeSql(sql).print(); {code}
> Even if the table exists in hive, the query will still report an error
> like"Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:14
> Table not found 'student'"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLIP 400] Introduces AsyncScalarFunction as a new UDF type [flink]
AlanConfluent commented on code in PR #23975: URL: https://github.com/apache/flink/pull/23975#discussion_r1433297124 ## flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/AsyncCalcSyncModeRule.java: ## @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.table.planner.plan.rules.logical; + +import org.apache.flink.table.planner.plan.nodes.logical.FlinkLogicalCalc; +import org.apache.flink.table.planner.plan.nodes.logical.FlinkLogicalJoin; +import org.apache.flink.table.planner.plan.nodes.logical.FlinkLogicalRel; +import org.apache.flink.table.planner.plan.trait.AsyncOperatorModeTrait; +import org.apache.flink.table.planner.plan.utils.AsyncUtil; + +import org.apache.calcite.plan.RelOptRule; +import org.apache.calcite.plan.RelOptRuleCall; +import org.apache.calcite.plan.hep.HepRelVertex; +import org.apache.calcite.rel.RelNode; +import org.apache.calcite.rel.core.Calc; +import org.apache.calcite.rel.core.Join; +import org.apache.calcite.rex.RexNode; + +import java.util.List; +import java.util.Optional; +import java.util.stream.Collectors; + +/** Review Comment: I'll likely remove these rules as well since with ORDERED only, they don't correspond to real things that we want to run in sync mode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLIP 400] Introduces AsyncScalarFunction as a new UDF type [flink]
flinkbot commented on PR #23975: URL: https://github.com/apache/flink/pull/23975#issuecomment-1865305626 ## CI report: * 0a5a00a39da8ce1ab4db47295b49251fa8d1c631 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLIP 400] Introduces AsyncScalarFunction as a new UDF type [flink]
AlanConfluent commented on code in PR #23975: URL: https://github.com/apache/flink/pull/23975#discussion_r1433296125 ## flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/trait/AsyncOperatorModeTrait.java: ## @@ -0,0 +1,47 @@ +package org.apache.flink.table.planner.plan.trait; Review Comment: I think given where we left things with UNORDERED not covered in this first version, there's no need for sync mode and these trait classes and operator sync mode support can be removed for now. What do you think @twalthr? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLIP 400] Introduces AsyncScalarFunction as a new UDF type [flink]
AlanConfluent opened a new pull request, #23975: URL: https://github.com/apache/flink/pull/23975 ## What is the purpose of the change This introduces `AsyncScalarFunction` a new UDF type which allows asynchronous responses. The functionality here is covered in FLIP-400: https://cwiki.apache.org/confluence/display/FLINK/FLIP-400%3A+AsyncScalarFunction+for+asynchronous+scalar+function+support ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) It changes it only for new uses of the AsyncScalarFunction, not existing code paths. - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (**yes** / no) - If yes, how is the feature documented? (not applicable / docs / **JavaDocs** / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33817) Allow ReadDefaultValues = False for non primitive types on Proto3
[ https://issues.apache.org/jira/browse/FLINK-33817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Sharath Dandi updated FLINK-33817: -- Description: *Background* The current Protobuf format [implementation|https://github.com/apache/flink/blob/c3e2d163a637dca5f49522721109161bd7ebb723/flink-formats/flink-protobuf/src/main/java/org/apache/flink/formats/protobuf/deserialize/ProtoToRowConverter.java] always sets ReadDefaultValues=False when using Proto3 version. This can cause severe performance degradation for large Protobuf schemas with OneOf fields as the entire generated code needs to be executed during deserialization even when certain fields are not present in the data to be deserialized and all the subsequent nested Fields can be skipped. Proto3 supports hasXXX() methods for checking field presence for non primitive types since Proto version [3.15|https://github.com/protocolbuffers/protobuf/releases/tag/v3.15.0]. In the internal performance benchmarks in our company, we've seen almost 10x difference in performance for one of our real production usecase when allowing to set ReadDefaultValues=False with proto3 version. The exact difference in performance depends on the schema complexity and data payload but we should allow readDefaultValue=False in general. *Solution* Support using ReadDefaultValues=False when using Proto3 version. We need to be careful to check for field presence only on non-primitive types if ReadDefaultValues is false and version used is Proto3 was: *Background* The current Protobuf format [implementation|https://github.com/apache/flink/blob/c3e2d163a637dca5f49522721109161bd7ebb723/flink-formats/flink-protobuf/src/main/java/org/apache/flink/formats/protobuf/deserialize/ProtoToRowConverter.java] always sets ReadDefaultValues=False when using Proto3 version. This can cause severe performance degradation for large Protobuf schemas with OneOf fields as the entire generated code needs to be executed during deserialization even when certain fields are not present in the data to be deserialized and all the subsequent nested Fields can be skipped. Proto3 supports hasXXX() methods for checking field presence for non primitive types since Proto version [3.15|https://github.com/protocolbuffers/protobuf/releases/tag/v3.15.0]. In the internal performance benchmarks in our company, we've seen almost 10x difference in performance when allowing to set ReadDefaultValues=False with proto3 version *Solution* Support using ReadDefaultValues=False when using Proto3 version. We need to be careful to check for field presence only on non-primitive types if ReadDefaultValues is false and version used is Proto3 > Allow ReadDefaultValues = False for non primitive types on Proto3 > - > > Key: FLINK-33817 > URL: https://issues.apache.org/jira/browse/FLINK-33817 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.18.0 >Reporter: Sai Sharath Dandi >Priority: Major > > *Background* > > The current Protobuf format > [implementation|https://github.com/apache/flink/blob/c3e2d163a637dca5f49522721109161bd7ebb723/flink-formats/flink-protobuf/src/main/java/org/apache/flink/formats/protobuf/deserialize/ProtoToRowConverter.java] > always sets ReadDefaultValues=False when using Proto3 version. This can > cause severe performance degradation for large Protobuf schemas with OneOf > fields as the entire generated code needs to be executed during > deserialization even when certain fields are not present in the data to be > deserialized and all the subsequent nested Fields can be skipped. Proto3 > supports hasXXX() methods for checking field presence for non primitive types > since Proto version > [3.15|https://github.com/protocolbuffers/protobuf/releases/tag/v3.15.0]. In > the internal performance benchmarks in our company, we've seen almost 10x > difference in performance for one of our real production usecase when > allowing to set ReadDefaultValues=False with proto3 version. The exact > difference in performance depends on the schema complexity and data payload > but we should allow readDefaultValue=False in general. > > *Solution* > > Support using ReadDefaultValues=False when using Proto3 version. We need to > be careful to check for field presence only on non-primitive types if > ReadDefaultValues is false and version used is Proto3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33611] [flink-protobuf] Support Large Protobuf Schemas [flink]
sharath1709 commented on PR #23937: URL: https://github.com/apache/flink/pull/23937#issuecomment-1865255265 @libenchao Can you please review this PR? I see you're one of the main reviewer for this module in the past -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-21224) Remove BatchExecExchange and StreamExecExchange, and replace their functionality with ExecEdge
[ https://issues.apache.org/jira/browse/FLINK-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799190#comment-17799190 ] Jim Hughes edited comment on FLINK-21224 at 12/20/23 10:08 PM: --- As a note, if this node is used, it would need a compiled plan representation. was (Author: JIRAUSER284726): As a note, if this node is used, it would need a compilled plan representation. > Remove BatchExecExchange and StreamExecExchange, and replace their > functionality with ExecEdge > -- > > Key: FLINK-21224 > URL: https://issues.apache.org/jira/browse/FLINK-21224 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: godfrey he >Priority: Major > Labels: auto-unassigned > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-21224) Remove BatchExecExchange and StreamExecExchange, and replace their functionality with ExecEdge
[ https://issues.apache.org/jira/browse/FLINK-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799190#comment-17799190 ] Jim Hughes commented on FLINK-21224: As a note, if this node is used, it would need a compilled plan representation. > Remove BatchExecExchange and StreamExecExchange, and replace their > functionality with ExecEdge > -- > > Key: FLINK-21224 > URL: https://issues.apache.org/jira/browse/FLINK-21224 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: godfrey he >Priority: Major > Labels: auto-unassigned > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33902][ci] Adds -legacy to openssl command [flink]
XComp commented on code in PR #23961:
URL: https://github.com/apache/flink/pull/23961#discussion_r1433172828
##
flink-end-to-end-tests/test-scripts/common_ssl.sh:
##
@@ -67,8 +67,15 @@ function _set_conf_ssl_helper {
keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/ca.cer" -alias ca -noprompt
keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/node.cer" -alias node -noprompt
+local additional_params
+additional_params=""
+if [[ "$(openssl version)" =~ OpenSSL\ 3 ]]; then
Review Comment:
I was wondering that as well. But I thought that we would have to revisit
the issue anyway, if OpenSSL 4 comes out. I would assume that the legacy
algorithms might be gone by then. :thinking:
But thinking about it once more - you might be right: It would fall back to
having no parameter which is probably also not what we want. I'm gonna change
it to using `-legacy` also for versions 4.x+. Either `-legacy` works also then
or it would fail because of an unknown parameter error which is more explicit.
:+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-27082][ci] Disables tests that relies on disabling file permissions [flink]
snuyanzin commented on code in PR #23962:
URL: https://github.com/apache/flink/pull/23962#discussion_r1433172250
##
flink-core/src/test/java/org/apache/flink/core/fs/local/LocalFileSystemTest.java:
##
@@ -236,6 +237,7 @@ public void testRenameNonExistingFile() throws IOException {
}
@Test
+@Ignore("FLINK-27082: Fails in Docker with root user")
Review Comment:
Wouldn't it be better to create interface something like
`FailsOnDockerWithRoot`
and with help of `@Category` mark tests failing for the specific maven
profile?
Like it is done for
https://github.com/apache/flink/blob/cabb28d25c4c58af3ee23fc4a63f9564aefd6146/flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/testutils/junit/FailsOnJava17.java#L21
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-27082][ci] Disables tests that relies on disabling file permissions [flink]
snuyanzin commented on code in PR #23962:
URL: https://github.com/apache/flink/pull/23962#discussion_r1433172250
##
flink-core/src/test/java/org/apache/flink/core/fs/local/LocalFileSystemTest.java:
##
@@ -236,6 +237,7 @@ public void testRenameNonExistingFile() throws IOException {
}
@Test
+@Ignore("FLINK-27082: Fails in Docker with root user")
Review Comment:
Wouldn't it be better to create interface something like
`FailsOnDockerWithRoot`
and with help of `@Category` mark tests failing for the specific maven
profile
like it is done for
https://github.com/apache/flink/blob/cabb28d25c4c58af3ee23fc4a63f9564aefd6146/flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/testutils/junit/FailsOnJava17.java#L21
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33902][ci] Adds -legacy to openssl command [flink]
snuyanzin commented on code in PR #23961:
URL: https://github.com/apache/flink/pull/23961#discussion_r1433167209
##
flink-end-to-end-tests/test-scripts/common_ssl.sh:
##
@@ -67,8 +67,15 @@ function _set_conf_ssl_helper {
keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/ca.cer" -alias ca -noprompt
keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/node.cer" -alias node -noprompt
+local additional_params
+additional_params=""
+if [[ "$(openssl version)" =~ OpenSSL\ 3 ]]; then
Review Comment:
am I right that in case there will be OpenSSL 4+ it will stop working?
should we have something more generic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-33904) Add zip as a package to GitHub Actions runners
[
https://issues.apache.org/jira/browse/FLINK-33904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl closed FLINK-33904.
-
Resolution: Won't Fix
It will be fixed in the workflow
> Add zip as a package to GitHub Actions runners
> ---
>
> Key: FLINK-33904
> URL: https://issues.apache.org/jira/browse/FLINK-33904
> Project: Flink
> Issue Type: Sub-task
> Components: Build System / CI
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: github-actions, pull-request-available
>
> FLINK-33253 shows that {{test_pyflink.sh}} fails in GHA because it doesn't
> find {{{}zip{}}}. We should add this as a dependency in the e2e test.
> {code:java}
> /root/flink/flink-end-to-end-tests/test-scripts/test_pyflink.sh: line 107:
> zip: command not found {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33904][test] Installs missing zip package [flink]
XComp commented on PR #23963: URL: https://github.com/apache/flink/pull/23963#issuecomment-1865091763 Closing this one because we should prep the runners in the workflow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33904][test] Installs missing zip package [flink]
XComp closed pull request #23963: [FLINK-33904][test] Installs missing zip package URL: https://github.com/apache/flink/pull/23963 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33877) CollectSinkFunctionTest.testConfiguredPortIsUsed fails due to BindException
[
https://issues.apache.org/jira/browse/FLINK-33877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799161#comment-17799161
]
Sergey Nuyanzin commented on FLINK-33877:
-
[~jiabao.sun]could you have a look please?
> CollectSinkFunctionTest.testConfiguredPortIsUsed fails due to BindException
> ---
>
> Key: FLINK-33877
> URL: https://issues.apache.org/jira/browse/FLINK-33877
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.19.0
>Reporter: Jiabao Sun
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55646&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=24c3384f-1bcb-57b3-224f-51bf973bbee8&l=9482
> {noformat}
> Dec 18 17:49:57 17:49:57.241 [ERROR]
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunctionTest.testConfiguredPortIsUsed
> -- Time elapsed: 0.021 s <<< ERROR!
> Dec 18 17:49:57 java.net.BindException: Address already in use (Bind failed)
> Dec 18 17:49:57 at java.net.PlainSocketImpl.socketBind(Native Method)
> Dec 18 17:49:57 at
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
> Dec 18 17:49:57 at java.net.ServerSocket.bind(ServerSocket.java:390)
> Dec 18 17:49:57 at java.net.ServerSocket.(ServerSocket.java:252)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction$ServerThread.(CollectSinkFunction.java:375)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction$ServerThread.(CollectSinkFunction.java:362)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction.open(CollectSinkFunction.java:252)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.utils.CollectSinkFunctionTestWrapper.openFunction(CollectSinkFunctionTestWrapper.java:103)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunctionTest.testConfiguredPortIsUsed(CollectSinkFunctionTest.java:138)
> Dec 18 17:49:57 at java.lang.reflect.Method.invoke(Method.java:498)
> Dec 18 17:49:57 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33877) CollectSinkFunctionTest.testConfiguredPortIsUsed fails due to BindException
[
https://issues.apache.org/jira/browse/FLINK-33877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799159#comment-17799159
]
Sergey Nuyanzin edited comment on FLINK-33877 at 12/20/23 8:07 PM:
---
Have to reopen since it is reproduced again with same issue
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=675bf62c-8558-587e-2555-dcad13acefb5&t=5878eed3-cc1e-5b12-1ed0-9e7139ce0992&l=9924
at the same time commit from this PR is already present
UPD:
it is reproduced even more than once
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=d06b80b4-9e88-5d40-12a2-18072cf60528&t=609ecd5a-3f6e-5d0c-2239-2096b155a4d0&l=9951
was (Author: sergey nuyanzin):
Have to reopen since it is reproduced again with same issue
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=675bf62c-8558-587e-2555-dcad13acefb5&t=5878eed3-cc1e-5b12-1ed0-9e7139ce0992&l=9924
at the same time commit from this PR is already present
> CollectSinkFunctionTest.testConfiguredPortIsUsed fails due to BindException
> ---
>
> Key: FLINK-33877
> URL: https://issues.apache.org/jira/browse/FLINK-33877
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.19.0
>Reporter: Jiabao Sun
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55646&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=24c3384f-1bcb-57b3-224f-51bf973bbee8&l=9482
> {noformat}
> Dec 18 17:49:57 17:49:57.241 [ERROR]
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunctionTest.testConfiguredPortIsUsed
> -- Time elapsed: 0.021 s <<< ERROR!
> Dec 18 17:49:57 java.net.BindException: Address already in use (Bind failed)
> Dec 18 17:49:57 at java.net.PlainSocketImpl.socketBind(Native Method)
> Dec 18 17:49:57 at
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
> Dec 18 17:49:57 at java.net.ServerSocket.bind(ServerSocket.java:390)
> Dec 18 17:49:57 at java.net.ServerSocket.(ServerSocket.java:252)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction$ServerThread.(CollectSinkFunction.java:375)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction$ServerThread.(CollectSinkFunction.java:362)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction.open(CollectSinkFunction.java:252)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.utils.CollectSinkFunctionTestWrapper.openFunction(CollectSinkFunctionTestWrapper.java:103)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunctionTest.testConfiguredPortIsUsed(CollectSinkFunctionTest.java:138)
> Dec 18 17:49:57 at java.lang.reflect.Method.invoke(Method.java:498)
> Dec 18 17:49:57 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (FLINK-33877) CollectSinkFunctionTest.testConfiguredPortIsUsed fails due to BindException
[
https://issues.apache.org/jira/browse/FLINK-33877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin reopened FLINK-33877:
-
Have to reopen since it is reproduced again with same issue
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=675bf62c-8558-587e-2555-dcad13acefb5&t=5878eed3-cc1e-5b12-1ed0-9e7139ce0992&l=9924
at the same time commit from this PR is already present
> CollectSinkFunctionTest.testConfiguredPortIsUsed fails due to BindException
> ---
>
> Key: FLINK-33877
> URL: https://issues.apache.org/jira/browse/FLINK-33877
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.19.0
>Reporter: Jiabao Sun
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55646&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=24c3384f-1bcb-57b3-224f-51bf973bbee8&l=9482
> {noformat}
> Dec 18 17:49:57 17:49:57.241 [ERROR]
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunctionTest.testConfiguredPortIsUsed
> -- Time elapsed: 0.021 s <<< ERROR!
> Dec 18 17:49:57 java.net.BindException: Address already in use (Bind failed)
> Dec 18 17:49:57 at java.net.PlainSocketImpl.socketBind(Native Method)
> Dec 18 17:49:57 at
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
> Dec 18 17:49:57 at java.net.ServerSocket.bind(ServerSocket.java:390)
> Dec 18 17:49:57 at java.net.ServerSocket.(ServerSocket.java:252)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction$ServerThread.(CollectSinkFunction.java:375)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction$ServerThread.(CollectSinkFunction.java:362)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction.open(CollectSinkFunction.java:252)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.utils.CollectSinkFunctionTestWrapper.openFunction(CollectSinkFunctionTestWrapper.java:103)
> Dec 18 17:49:57 at
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunctionTest.testConfiguredPortIsUsed(CollectSinkFunctionTest.java:138)
> Dec 18 17:49:57 at java.lang.reflect.Method.invoke(Method.java:498)
> Dec 18 17:49:57 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Dec 18 17:49:57 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33919) AutoRescalingITCase hangs on AZP
Sergey Nuyanzin created FLINK-33919:
---
Summary: AutoRescalingITCase hangs on AZP
Key: FLINK-33919
URL: https://issues.apache.org/jira/browse/FLINK-33919
Project: Flink
Issue Type: Improvement
Components: Runtime / Checkpointing
Affects Versions: 1.19.0
Reporter: Sergey Nuyanzin
This build fails on AZP
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=a657ddbf-d986-5381-9649-342d9c92e7fb&t=dc085d4a-05c8-580e-06ab-21f5624dab16&l=8608
because of waiting
{noformat}
Dec 20 02:07:46 "main" #1 [14299] prio=5 os_prio=0 cpu=12675.70ms
elapsed=3115.94s tid=0x7f3f71481600 nid=14299 waiting on condition
[0x7f3f74913000]
Dec 20 02:07:46java.lang.Thread.State: TIMED_WAITING (sleeping)
Dec 20 02:07:46 at java.lang.Thread.sleep0(java.base@21.0.1/Native
Method)
Dec 20 02:07:46 at
java.lang.Thread.sleep(java.base@21.0.1/Thread.java:509)
Dec 20 02:07:46 at
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:152)
Dec 20 02:07:46 at
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:145)
Dec 20 02:07:46 at
org.apache.flink.runtime.testutils.CommonTestUtils.waitForOneMoreCheckpoint(CommonTestUtils.java:374)
Dec 20 02:07:46 at
org.apache.flink.test.checkpointing.AutoRescalingITCase.testCheckpointRescalingKeyedState(AutoRescalingITCase.java:265)
Dec 20 02:07:46 at
org.apache.flink.test.checkpointing.AutoRescalingITCase.testCheckpointRescalingInKeyedState(AutoRescalingITCase.java:196)
Dec 20 02:07:46 at
java.lang.invoke.LambdaForm$DMH/0x7f3f0f201400.invokeVirtual(java.base@21.0.1/LambdaForm$DMH)
Dec 20 02:07:46 at
java.lang.invoke.LambdaForm$MH/0x7f3f0f20c000.invoke(java.base@21.0.1/LambdaForm$MH)
Dec 20 02:07:46 at
java.lang.invoke.Invokers$Holder.invokeExact_MT(java.base@21.0.1/Invokers$Holder)
Dec 20 02:07:46 at
jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(java.base@21.0.1/DirectMethodHandleAccessor.java:153)
Dec 20 02:07:46 at
jdk.internal.reflect.DirectMethodHandleAccessor.invoke(java.base@21.0.1/DirectMethodHandleAccessor.java:103)
Dec 20 02:07:46 at
java.lang.reflect.Method.invoke(java.base@21.0.1/Method.java:580)
Dec 20 02:07:46 at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
Dec 20 02:07:46 at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Dec 20 02:07:46 at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
Dec 20 02:07:46 at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Dec 20 02:07:46 at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
Dec 20 02:07:46 at
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
Dec 20 02:07:46 at
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
Dec 20 02:07:46 at
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33919) AutoRescalingITCase hangs on AZP
[
https://issues.apache.org/jira/browse/FLINK-33919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin updated FLINK-33919:
Issue Type: Bug (was: Improvement)
> AutoRescalingITCase hangs on AZP
>
>
> Key: FLINK-33919
> URL: https://issues.apache.org/jira/browse/FLINK-33919
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: test-stability
>
> This build fails on AZP
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=a657ddbf-d986-5381-9649-342d9c92e7fb&t=dc085d4a-05c8-580e-06ab-21f5624dab16&l=8608
> because of waiting
> {noformat}
> Dec 20 02:07:46 "main" #1 [14299] prio=5 os_prio=0 cpu=12675.70ms
> elapsed=3115.94s tid=0x7f3f71481600 nid=14299 waiting on condition
> [0x7f3f74913000]
> Dec 20 02:07:46java.lang.Thread.State: TIMED_WAITING (sleeping)
> Dec 20 02:07:46 at java.lang.Thread.sleep0(java.base@21.0.1/Native
> Method)
> Dec 20 02:07:46 at
> java.lang.Thread.sleep(java.base@21.0.1/Thread.java:509)
> Dec 20 02:07:46 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:152)
> Dec 20 02:07:46 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:145)
> Dec 20 02:07:46 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitForOneMoreCheckpoint(CommonTestUtils.java:374)
> Dec 20 02:07:46 at
> org.apache.flink.test.checkpointing.AutoRescalingITCase.testCheckpointRescalingKeyedState(AutoRescalingITCase.java:265)
> Dec 20 02:07:46 at
> org.apache.flink.test.checkpointing.AutoRescalingITCase.testCheckpointRescalingInKeyedState(AutoRescalingITCase.java:196)
> Dec 20 02:07:46 at
> java.lang.invoke.LambdaForm$DMH/0x7f3f0f201400.invokeVirtual(java.base@21.0.1/LambdaForm$DMH)
> Dec 20 02:07:46 at
> java.lang.invoke.LambdaForm$MH/0x7f3f0f20c000.invoke(java.base@21.0.1/LambdaForm$MH)
> Dec 20 02:07:46 at
> java.lang.invoke.Invokers$Holder.invokeExact_MT(java.base@21.0.1/Invokers$Holder)
> Dec 20 02:07:46 at
> jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(java.base@21.0.1/DirectMethodHandleAccessor.java:153)
> Dec 20 02:07:46 at
> jdk.internal.reflect.DirectMethodHandleAccessor.invoke(java.base@21.0.1/DirectMethodHandleAccessor.java:103)
> Dec 20 02:07:46 at
> java.lang.reflect.Method.invoke(java.base@21.0.1/Method.java:580)
> Dec 20 02:07:46 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Dec 20 02:07:46 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Dec 20 02:07:46 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Dec 20 02:07:46 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Dec 20 02:07:46 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> Dec 20 02:07:46 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Dec 20 02:07:46 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> Dec 20 02:07:46 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-33918) Fix AsyncSinkWriterThrottlingTest test failure
[
https://issues.apache.org/jira/browse/FLINK-33918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Hughes closed FLINK-33918.
--
Resolution: Duplicate
Duplicate of https://issues.apache.org/jira/browse/FLINK-31472
> Fix AsyncSinkWriterThrottlingTest test failure
> --
>
> Key: FLINK-33918
> URL: https://issues.apache.org/jira/browse/FLINK-33918
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Jim Hughes
>Priority: Major
> Labels: test-stability
>
> From
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd]
> {code:java}
> Dec 20 03:09:03 03:09:03.411 [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> – Time elapsed: 0.879 s <<< ERROR!
> Dec 20 03:09:03 java.lang.IllegalStateException: Illegal thread detected.
> This method must be called from inside the mailbox thread!
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
>
> Dec 20 03:09:03 at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
>
> Dec 20 03:09:03 at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
>
> Dec 20 03:09:03 at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
>
> Dec 20 03:09:03 at java.lang.reflect.Method.invoke(Method.java:498) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33918) Fix AsyncSinkWriterThrottlingTest test failure
[
https://issues.apache.org/jira/browse/FLINK-33918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799149#comment-17799149
]
Jim Hughes commented on FLINK-33918:
Sorry for the duplicate!
> Fix AsyncSinkWriterThrottlingTest test failure
> --
>
> Key: FLINK-33918
> URL: https://issues.apache.org/jira/browse/FLINK-33918
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Jim Hughes
>Priority: Major
> Labels: test-stability
>
> From
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd]
> {code:java}
> Dec 20 03:09:03 03:09:03.411 [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> – Time elapsed: 0.879 s <<< ERROR!
> Dec 20 03:09:03 java.lang.IllegalStateException: Illegal thread detected.
> This method must be called from inside the mailbox thread!
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
>
> Dec 20 03:09:03 at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
>
> Dec 20 03:09:03 at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
>
> Dec 20 03:09:03 at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
>
> Dec 20 03:09:03 at java.lang.reflect.Method.invoke(Method.java:498) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33918) Fix AsyncSinkWriterThrottlingTest test failure
[
https://issues.apache.org/jira/browse/FLINK-33918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799147#comment-17799147
]
Martijn Visser commented on FLINK-33918:
[~jhughes] I think you might have to rebase, it appears to have been fixed via
FLINK-31472
> Fix AsyncSinkWriterThrottlingTest test failure
> --
>
> Key: FLINK-33918
> URL: https://issues.apache.org/jira/browse/FLINK-33918
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Jim Hughes
>Priority: Major
> Labels: test-stability
>
> From
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd]
> {code:java}
> Dec 20 03:09:03 03:09:03.411 [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> – Time elapsed: 0.879 s <<< ERROR!
> Dec 20 03:09:03 java.lang.IllegalStateException: Illegal thread detected.
> This method must be called from inside the mailbox thread!
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
>
> Dec 20 03:09:03 at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
>
> Dec 20 03:09:03 at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
>
> Dec 20 03:09:03 at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
>
> Dec 20 03:09:03 at java.lang.reflect.Method.invoke(Method.java:498) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33917) IllegalArgumentException: hostname can't be null
[
https://issues.apache.org/jira/browse/FLINK-33917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-33917:
---
Labels: pull-request-available (was: )
> IllegalArgumentException: hostname can't be null
>
>
> Key: FLINK-33917
> URL: https://issues.apache.org/jira/browse/FLINK-33917
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Tom
>Priority: Major
> Labels: pull-request-available
>
> In certain scenarios, if the hostname contains certain characters it will
> throw an exception when it tries to initialize the `InetSocketAddress`
>
> {code:java}
> java.lang.IllegalArgumentException: hostname can't be null at
> java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
> at
> java.base/java.net.InetSocketAddress.(InetSocketAddress.java:216) {code}
>
> [https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
>
> {code:java}
> @Override
> public boolean isJobManagerPortReady(Configuration config) {
> final URI uri;
> try (var clusterClient = getClusterClient(config)) {
> uri = URI.create(clusterClient.getWebInterfaceURL());
> } catch (Exception ex) {
> throw new FlinkRuntimeException(ex);
> }
> SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
> uri.getPort());
> Socket socket = new Socket();
> try {
> socket.connect(socketAddress, 1000);
> socket.close();
> return true;
> } catch (IOException e) {
> return false;
> }
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-33917] fix hostname can't be null for certain edgecases [flink-kubernetes-operator]
ensctom opened a new pull request, #738: URL: https://github.com/apache/flink-kubernetes-operator/pull/738 ## What is the purpose of the change Fix hostname is null under certain edgecases. ## Brief change log *(for example:)* - refactor AbstractFlinkService#isJobManagerReady to use URL instead of URI ## Verifying this change This change added tests and can be verified as follows: *(example:)* - *Add new test cases to test different url form factors - *Manually verified that the reconciliation loop still works as expected ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): NO - The public API, i.e., is any changes to the `CustomResourceDescriptors`: NO - Core observer or reconciler logic that is regularly executed: YES ## Documentation - Does this pull request introduce a new feature? NO -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33918) Fix AsyncSinkWriterThrottlingTest test failure
[
https://issues.apache.org/jira/browse/FLINK-33918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Hughes updated FLINK-33918:
---
Description:
>From
>[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd]
{code:java}
Dec 20 03:09:03 03:09:03.411 [ERROR]
org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
– Time elapsed: 0.879 s <<< ERROR!
Dec 20 03:09:03 java.lang.IllegalStateException: Illegal thread detected. This
method must be called from inside the mailbox thread!
Dec 20 03:09:03 at
org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
Dec 20 03:09:03 at
org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
Dec 20 03:09:03 at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
Dec 20 03:09:03 at
org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
Dec 20 03:09:03 at
org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
Dec 20 03:09:03 at
org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
Dec 20 03:09:03 at
org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
Dec 20 03:09:03 at
org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
Dec 20 03:09:03 at java.lang.reflect.Method.invoke(Method.java:498) {code}
was:
>From
>[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd]
```
Dec 20 03:09:03 03:09:03.411 [ERROR]
org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
-- Time elapsed: 0.879 s <<< ERROR!
Dec 20 03:09:03 java.lang.IllegalStateException: Illegal thread detected. This
method must be called from inside the mailbox thread!
Dec 20 03:09:03 at
org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
Dec 20 03:09:03 at
org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
Dec 20 03:09:03 at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
Dec 20 03:09:03 at
org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
Dec 20 03:09:03 at
org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
Dec 20 03:09:03 at
org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
Dec 20 03:09:03 at
org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
Dec 20 03:09:03 at
org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
Dec 20 03:09:03 at java.lang.reflect.Method.invoke(Method.java:498)
```
> Fix AsyncSinkWriterThrottlingTest test failure
> --
>
> Key: FLINK-33918
> URL: https://issues.apache.org/jira/browse/FLINK-33918
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Jim Hughes
>Priority: Major
> Labels: test-stability
>
> From
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd]
> {code:java}
> Dec 20 03:09:03 03:09:03.411 [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> – Time elapsed: 0.879 s <<< ERROR!
> Dec 20 03:09:03 java.lang.IllegalStateException: Illegal thread detected.
> This method must be called from inside the mailbox thread!
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
>
> Dec 20 03:09:03 at
> org.apache.flink.connector.base.sink.w
[jira] [Commented] (FLINK-33918) Fix AsyncSinkWriterThrottlingTest test failure
[
https://issues.apache.org/jira/browse/FLINK-33918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799139#comment-17799139
]
Jim Hughes commented on FLINK-33918:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55701&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd]
Same test; slightly different stack trace:
{code:java}
Dec 20 03:28:41 java.lang.IllegalStateException: Illegal thread detected. This
method must be called from inside the mailbox thread!
Dec 20 03:28:41 at
org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
Dec 20 03:28:41 at
org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
Dec 20 03:28:41 at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
Dec 20 03:28:41 at
org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
Dec 20 03:28:41 at
org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
Dec 20 03:28:41 at
org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
Dec 20 03:28:41 at
org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
Dec 20 03:28:41 at
org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
Dec 20 03:28:41 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
{code}
> Fix AsyncSinkWriterThrottlingTest test failure
> --
>
> Key: FLINK-33918
> URL: https://issues.apache.org/jira/browse/FLINK-33918
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Jim Hughes
>Priority: Major
> Labels: test-stability
>
> From
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd]
>
> ```
> Dec 20 03:09:03 03:09:03.411 [ERROR]
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
> -- Time elapsed: 0.879 s <<< ERROR!
> Dec 20 03:09:03 java.lang.IllegalStateException: Illegal thread detected.
> This method must be called from inside the mailbox thread!
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
>
> Dec 20 03:09:03 at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
>
> Dec 20 03:09:03 at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
>
> Dec 20 03:09:03 at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
>
> Dec 20 03:09:03 at
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
>
> Dec 20 03:09:03 at java.lang.reflect.Method.invoke(Method.java:498)
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33918) Fix AsyncSinkWriterThrottlingTest test failure
Jim Hughes created FLINK-33918: -- Summary: Fix AsyncSinkWriterThrottlingTest test failure Key: FLINK-33918 URL: https://issues.apache.org/jira/browse/FLINK-33918 Project: Flink Issue Type: Bug Affects Versions: 1.19.0 Reporter: Jim Hughes >From >[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55700&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd] ``` Dec 20 03:09:03 03:09:03.411 [ERROR] org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize -- Time elapsed: 0.879 s <<< ERROR! Dec 20 03:09:03 java.lang.IllegalStateException: Illegal thread detected. This method must be called from inside the mailbox thread! Dec 20 03:09:03 at org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262) Dec 20 03:09:03 at org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137) Dec 20 03:09:03 at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84) Dec 20 03:09:03 at org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367) Dec 20 03:09:03 at org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315) Dec 20 03:09:03 at org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199) Dec 20 03:09:03 at org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76) Dec 20 03:09:03 at org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64) Dec 20 03:09:03 at java.lang.reflect.Method.invoke(Method.java:498) ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33917) IllegalArgumentException: hostname can't be null
[
https://issues.apache.org/jira/browse/FLINK-33917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom updated FLINK-33917:
Description:
In certain scenarios, if the hostname contains certain characters it will throw
an exception when it tries to initialize the `InetSocketAddress`
{code:java}
java.lang.IllegalArgumentException: hostname can't be null at
java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
at java.base/java.net.InetSocketAddress.(InetSocketAddress.java:216)
{code}
[https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
{code:java}
@Override
public boolean isJobManagerPortReady(Configuration config) {
final URI uri;
try (var clusterClient = getClusterClient(config)) {
uri = URI.create(clusterClient.getWebInterfaceURL());
} catch (Exception ex) {
throw new FlinkRuntimeException(ex);
}
SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
uri.getPort());
Socket socket = new Socket();
try {
socket.connect(socketAddress, 1000);
socket.close();
return true;
} catch (IOException e) {
return false;
}
}
{code}
was:
In certain scenarios, if the hostname contains certain characters it will throw
an exception when it tries to initialize the InetSocketAddress
[https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
{code:java}
@Override
public boolean isJobManagerPortReady(Configuration config) {
final URI uri;
try (var clusterClient = getClusterClient(config)) {
uri = URI.create(clusterClient.getWebInterfaceURL());
} catch (Exception ex) {
throw new FlinkRuntimeException(ex);
}
SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
uri.getPort());
Socket socket = new Socket();
try {
socket.connect(socketAddress, 1000);
socket.close();
return true;
} catch (IOException e) {
return false;
}
}
{code}
> IllegalArgumentException: hostname can't be null
>
>
> Key: FLINK-33917
> URL: https://issues.apache.org/jira/browse/FLINK-33917
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Tom
>Priority: Major
>
> In certain scenarios, if the hostname contains certain characters it will
> throw an exception when it tries to initialize the `InetSocketAddress`
>
> {code:java}
> java.lang.IllegalArgumentException: hostname can't be null at
> java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
> at
> java.base/java.net.InetSocketAddress.(InetSocketAddress.java:216) {code}
>
> [https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
>
> {code:java}
> @Override
> public boolean isJobManagerPortReady(Configuration config) {
> final URI uri;
> try (var clusterClient = getClusterClient(config)) {
> uri = URI.create(clusterClient.getWebInterfaceURL());
> } catch (Exception ex) {
> throw new FlinkRuntimeException(ex);
> }
> SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
> uri.getPort());
> Socket socket = new Socket();
> try {
> socket.connect(socketAddress, 1000);
> socket.close();
> return true;
> } catch (IOException e) {
> return false;
> }
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33917) IllegalArgumentException: hostname can't be null
[
https://issues.apache.org/jira/browse/FLINK-33917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom updated FLINK-33917:
Description:
In certain scenarios, if the hostname contains certain characters it will throw
an exception when it tries to initialize the InetSocketAddress
[https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
{code:java}
@Override
public boolean isJobManagerPortReady(Configuration config) {
final URI uri;
try (var clusterClient = getClusterClient(config)) {
uri = URI.create(clusterClient.getWebInterfaceURL());
} catch (Exception ex) {
throw new FlinkRuntimeException(ex);
}
SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
uri.getPort());
Socket socket = new Socket();
try {
socket.connect(socketAddress, 1000);
socket.close();
return true;
} catch (IOException e) {
return false;
}
}
{code}
was:
In certain scenarios, if the hostname contains certain characters it will throw
an exception when it tries to initialize the InetSocketAddress
[https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
{code:java}
@Override
public boolean isJobManagerPortReady(Configuration config) {
final URI uri;
try (var clusterClient = getClusterClient(config))
{ uri = URI.create(clusterClient.getWebInterfaceURL()); }
catch (Exception ex)
{ throw new FlinkRuntimeException(ex); }
SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
uri.getPort());
Socket socket = new Socket();
try
{ socket.connect(socketAddress, 1000); socket.close();
return true; }
catch (IOException e)
{ return false; }
}
{code}
> IllegalArgumentException: hostname can't be null
>
>
> Key: FLINK-33917
> URL: https://issues.apache.org/jira/browse/FLINK-33917
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Tom
>Priority: Major
>
> In certain scenarios, if the hostname contains certain characters it will
> throw an exception when it tries to initialize the InetSocketAddress
> [https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
>
> {code:java}
> @Override
> public boolean isJobManagerPortReady(Configuration config) {
> final URI uri;
> try (var clusterClient = getClusterClient(config)) {
> uri = URI.create(clusterClient.getWebInterfaceURL());
> } catch (Exception ex) {
> throw new FlinkRuntimeException(ex);
> }
> SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
> uri.getPort());
> Socket socket = new Socket();
> try {
> socket.connect(socketAddress, 1000);
> socket.close();
> return true;
> } catch (IOException e) {
> return false;
> }
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33917) IllegalArgumentException: hostname can't be null
[
https://issues.apache.org/jira/browse/FLINK-33917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom updated FLINK-33917:
Description:
In certain scenarios, if the hostname contains certain characters it will throw
an exception when it tries to initialize the InetSocketAddress
[https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
{code:java}
@Override
public boolean isJobManagerPortReady(Configuration config) {
final URI uri;
try (var clusterClient = getClusterClient(config))
{ uri = URI.create(clusterClient.getWebInterfaceURL()); }
catch (Exception ex)
{ throw new FlinkRuntimeException(ex); }
SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
uri.getPort());
Socket socket = new Socket();
try
{ socket.connect(socketAddress, 1000); socket.close();
return true; }
catch (IOException e)
{ return false; }
}
{code}
was:
In certain scenarios, if the hostname contains certain characters it will throw
an exception when it tries to initialize the InetSocketAddress
@Override
public boolean isJobManagerPortReady(Configuration config) {
final URI uri;
try (var clusterClient = getClusterClient(config)) {
uri = URI.create(clusterClient.getWebInterfaceURL());
} catch (Exception ex) {
throw new FlinkRuntimeException(ex);
}
SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
uri.getPort());
Socket socket = new Socket();
try {
socket.connect(socketAddress, 1000);
socket.close();
return true;
} catch (IOException e) {
return false;
}
}
> IllegalArgumentException: hostname can't be null
>
>
> Key: FLINK-33917
> URL: https://issues.apache.org/jira/browse/FLINK-33917
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Tom
>Priority: Major
>
> In certain scenarios, if the hostname contains certain characters it will
> throw an exception when it tries to initialize the InetSocketAddress
> [https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
>
> {code:java}
>
> @Override
> public boolean isJobManagerPortReady(Configuration config) {
> final URI uri;
> try (var clusterClient = getClusterClient(config))
> { uri = URI.create(clusterClient.getWebInterfaceURL()); }
> catch (Exception ex)
> { throw new FlinkRuntimeException(ex); }
> SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
> uri.getPort());
> Socket socket = new Socket();
> try
> { socket.connect(socketAddress, 1000);
> socket.close(); return true; }
> catch (IOException e)
> { return false; }
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33917) IllegalArgumentException: hostname can't be null
Tom created FLINK-33917:
---
Summary: IllegalArgumentException: hostname can't be null
Key: FLINK-33917
URL: https://issues.apache.org/jira/browse/FLINK-33917
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: Tom
In certain scenarios, if the hostname contains certain characters it will throw
an exception when it tries to initialize the InetSocketAddress
@Override
public boolean isJobManagerPortReady(Configuration config) {
final URI uri;
try (var clusterClient = getClusterClient(config)) {
uri = URI.create(clusterClient.getWebInterfaceURL());
} catch (Exception ex) {
throw new FlinkRuntimeException(ex);
}
SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
uri.getPort());
Socket socket = new Socket();
try {
socket.connect(socketAddress, 1000);
socket.close();
return true;
} catch (IOException e) {
return false;
}
}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33531) Nightly Python fails with NPE at metadataHandlerProvider on AZP (StreamDependencyTests.test_add_python_archive)
[
https://issues.apache.org/jira/browse/FLINK-33531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799119#comment-17799119
]
Matthias Pohl commented on FLINK-33531:
---
It looks like this fix is not good enough? The following [1.18 nightly
run|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55653&view=logs&j=b53e1644-5cb4-5a3b-5d48-f523f39bcf06&t=b68c9f5c-04c9-5c75-3862-a3a27aabbce3&l=25428]
included the fixed
[0d6ab1db|https://github.com/apache/flink/commit/0d6ab1db6c04cd88f646d545075bea539bac9fcf]
mentioned above but failed with the same error if I'm not mistaken?
> Nightly Python fails with NPE at metadataHandlerProvider on AZP
> (StreamDependencyTests.test_add_python_archive)
> ---
>
> Key: FLINK-33531
> URL: https://issues.apache.org/jira/browse/FLINK-33531
> Project: Flink
> Issue Type: Technical Debt
> Components: API / Python
>Affects Versions: 1.18.0, 1.19.0
>Reporter: Sergey Nuyanzin
>Assignee: Xingbo Huang
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0, 1.18.1
>
>
> It seems starting 02.11.2023 every master nightly fails with this (that's why
> it is a blocker)
> for instance
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54512&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=c67e71ed-6451-5d26-8920-5a8cf9651901]
> {noformat}
> 2023-11-12T02:10:24.5082784Z Nov 12 02:10:24 if is_error(answer)[0]:
> 2023-11-12T02:10:24.5083620Z Nov 12 02:10:24 if len(answer) > 1:
> 2023-11-12T02:10:24.5084326Z Nov 12 02:10:24 type = answer[1]
> 2023-11-12T02:10:24.5085164Z Nov 12 02:10:24 value =
> OUTPUT_CONVERTER[type](answer[2:], gateway_client)
> 2023-11-12T02:10:24.5086061Z Nov 12 02:10:24 if answer[1] ==
> REFERENCE_TYPE:
> 2023-11-12T02:10:24.5086850Z Nov 12 02:10:24 > raise
> Py4JJavaError(
> 2023-11-12T02:10:24.5087677Z Nov 12 02:10:24 "An
> error occurred while calling {0}{1}{2}.\n".
> 2023-11-12T02:10:24.5088538Z Nov 12 02:10:24
> format(target_id, ".", name), value)
> 2023-11-12T02:10:24.5089551Z Nov 12 02:10:24 E
> py4j.protocol.Py4JJavaError: An error occurred while calling
> o3371.executeInsert.
> 2023-11-12T02:10:24.5090832Z Nov 12 02:10:24 E :
> java.lang.NullPointerException: metadataHandlerProvider
> 2023-11-12T02:10:24.5091832Z Nov 12 02:10:24 Eat
> java.util.Objects.requireNonNull(Objects.java:228)
> 2023-11-12T02:10:24.5093399Z Nov 12 02:10:24 Eat
> org.apache.calcite.rel.metadata.RelMetadataQueryBase.getMetadataHandlerProvider(RelMetadataQueryBase.java:122)
> 2023-11-12T02:10:24.5094480Z Nov 12 02:10:24 Eat
> org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java:118)
> 2023-11-12T02:10:24.5095365Z Nov 12 02:10:24 Eat
> org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:844)
> 2023-11-12T02:10:24.5096306Z Nov 12 02:10:24 Eat
> org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:307)
> 2023-11-12T02:10:24.5097238Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:337)
> 2023-11-12T02:10:24.5098014Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556)
> 2023-11-12T02:10:24.5098753Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:420)
> 2023-11-12T02:10:24.5099517Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan.hep.HepPlanner.executeRuleInstance(HepPlanner.java:243)
> 2023-11-12T02:10:24.5100373Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance$State.execute(HepInstruction.java:178)
> 2023-11-12T02:10:24.5101313Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan.hep.HepPlanner.lambda$executeProgram$0(HepPlanner.java:211)
> 2023-11-12T02:10:24.5102410Z Nov 12 02:10:24 Eat
> org.apache.flink.calcite.shaded.com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
> 2023-11-12T02:10:24.5103343Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:210)
> 2023-11-12T02:10:24.5104105Z Nov 12 02:10:24 Eat
> org.apache.calcite.plan
[jira] [Resolved] (FLINK-33641) JUnit5 fails to delete a directory on AZP for various table-planner tests
[
https://issues.apache.org/jira/browse/FLINK-33641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl resolved FLINK-33641.
---
Resolution: Fixed
> JUnit5 fails to delete a directory on AZP for various table-planner tests
> -
>
> Key: FLINK-33641
> URL: https://issues.apache.org/jira/browse/FLINK-33641
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Assignee: Jiabao Sun
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
>
> this build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54856&view=logs&j=a9db68b9-a7e0-54b6-0f98-010e0aff39e2&t=cdd32e0b-6047-565b-c58f-14054472f1be&l=11289
> fails with
> {noformat}
> Nov 24 02:21:53 Suppressed: java.nio.file.DirectoryNotEmptyException:
> /tmp/junit1727687356898183357/junit4798298549994985259/1ac07a5866d81240870d5a2982531508
> Nov 24 02:21:53 at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
> Nov 24 02:21:53 at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> Nov 24 02:21:53 at java.nio.file.Files.delete(Files.java:1126)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.deleteAndContinue(TempDirectory.java:293)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.postVisitDirectory(TempDirectory.java:288)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.postVisitDirectory(TempDirectory.java:264)
> Nov 24 02:21:53 at
> java.nio.file.Files.walkFileTree(Files.java:2688)
> Nov 24 02:21:53 at
> java.nio.file.Files.walkFileTree(Files.java:2742)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.deleteAllFilesAndDirectories(TempDirectory.java:264)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:249)
> Nov 24 02:21:53 ... 92 more
> {noformat}
> not sure however this might be related to recent JUnit4 => JUnit5 upgrade
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33641) JUnit5 fails to delete a directory on AZP for various table-planner tests
[
https://issues.apache.org/jira/browse/FLINK-33641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799113#comment-17799113
]
Matthias Pohl edited comment on FLINK-33641 at 12/20/23 5:42 PM:
-
Thanks for the point, [~jiabao.sun] . I will update my GHA test branch to
include ac88acfbb1b4ebf7336e9a20e0b6d0b0fe32be51. (y)
I'm gonna close the issue again in the meantime.
was (Author: mapohl):
Thanks for the point, [~jiabao.sun] . I will update my GHA test branch to
include ac88acfbb1b4ebf7336e9a20e0b6d0b0fe32be51. (y)
> JUnit5 fails to delete a directory on AZP for various table-planner tests
> -
>
> Key: FLINK-33641
> URL: https://issues.apache.org/jira/browse/FLINK-33641
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Assignee: Jiabao Sun
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
>
> this build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54856&view=logs&j=a9db68b9-a7e0-54b6-0f98-010e0aff39e2&t=cdd32e0b-6047-565b-c58f-14054472f1be&l=11289
> fails with
> {noformat}
> Nov 24 02:21:53 Suppressed: java.nio.file.DirectoryNotEmptyException:
> /tmp/junit1727687356898183357/junit4798298549994985259/1ac07a5866d81240870d5a2982531508
> Nov 24 02:21:53 at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
> Nov 24 02:21:53 at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> Nov 24 02:21:53 at java.nio.file.Files.delete(Files.java:1126)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.deleteAndContinue(TempDirectory.java:293)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.postVisitDirectory(TempDirectory.java:288)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.postVisitDirectory(TempDirectory.java:264)
> Nov 24 02:21:53 at
> java.nio.file.Files.walkFileTree(Files.java:2688)
> Nov 24 02:21:53 at
> java.nio.file.Files.walkFileTree(Files.java:2742)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.deleteAllFilesAndDirectories(TempDirectory.java:264)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:249)
> Nov 24 02:21:53 ... 92 more
> {noformat}
> not sure however this might be related to recent JUnit4 => JUnit5 upgrade
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33641) JUnit5 fails to delete a directory on AZP for various table-planner tests
[
https://issues.apache.org/jira/browse/FLINK-33641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799113#comment-17799113
]
Matthias Pohl commented on FLINK-33641:
---
Thanks for the point, [~jiabao.sun] . I will update my GHA test branch to
include ac88acfbb1b4ebf7336e9a20e0b6d0b0fe32be51. (y)
> JUnit5 fails to delete a directory on AZP for various table-planner tests
> -
>
> Key: FLINK-33641
> URL: https://issues.apache.org/jira/browse/FLINK-33641
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Assignee: Jiabao Sun
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
>
> this build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54856&view=logs&j=a9db68b9-a7e0-54b6-0f98-010e0aff39e2&t=cdd32e0b-6047-565b-c58f-14054472f1be&l=11289
> fails with
> {noformat}
> Nov 24 02:21:53 Suppressed: java.nio.file.DirectoryNotEmptyException:
> /tmp/junit1727687356898183357/junit4798298549994985259/1ac07a5866d81240870d5a2982531508
> Nov 24 02:21:53 at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
> Nov 24 02:21:53 at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> Nov 24 02:21:53 at java.nio.file.Files.delete(Files.java:1126)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.deleteAndContinue(TempDirectory.java:293)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.postVisitDirectory(TempDirectory.java:288)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.postVisitDirectory(TempDirectory.java:264)
> Nov 24 02:21:53 at
> java.nio.file.Files.walkFileTree(Files.java:2688)
> Nov 24 02:21:53 at
> java.nio.file.Files.walkFileTree(Files.java:2742)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.deleteAllFilesAndDirectories(TempDirectory.java:264)
> Nov 24 02:21:53 at
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:249)
> Nov 24 02:21:53 ... 92 more
> {noformat}
> not sure however this might be related to recent JUnit4 => JUnit5 upgrade
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
mxm commented on PR #728: URL: https://github.com/apache/flink-kubernetes-operator/pull/728#issuecomment-186482 > @gyfora @mxm by the way, I'd like to discuss the logic of scaling effectiveness evaluation with you. > > 1. Now it's controlled by two config `scaling.effectiveness.detection.enabled` and `scaling.effectiveness.threshold` and we evaluate the effectiveness under the condition `last scaling is scale up` and only refer to the last scaling effectiveness. > 2. Image the following scenario: scale up double parallism first, then scale down to 0.8 parallism, then scale up double scale down 0.8, the effectiveness detection will be invalid in this scenario, even scale up is ineffecive, we'll continue scale up > 3. Maybe we can add a new config like `scaling.effectiveness.history.reference.num` and set a default value, then we can evaluate based on the last `scaling.effectiveness.history.reference.num` scale up summaries. > > Looking forward to your reply. What yo you describe is an edge case we hadn't considered. We were more concerned about a continuous increase in parallelism. If there is any scale down we are currently assuming that the algorithm is not completely broken. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
mxm commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1432942758
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##
@@ -264,4 +262,24 @@ private static Map
getVertexParallelismOverrides(
});
return overrides;
}
+
+private boolean blockScalingExecution(
+Context context,
+Map scalingSummaries,
+Configuration conf,
+Instant now) {
+var scaleEnabled = conf.get(SCALING_ENABLED);
+var isExcluded = CalendarUtils.inExcludedPeriods(conf, now);
+autoScalerEventHandler.handleScalingEvent(
+context,
+scalingSummaries,
+scaleEnabled,
+isExcluded,
+conf.get(SCALING_EVENT_INTERVAL));
+
+if (!scaleEnabled || isExcluded) {
+return true;
+}
+return false;
Review Comment:
```suggestion
return !scaleEnabled || isExcluded;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-33900) Multiple failures in WindowRankITCase due to NoResourceAvailableException
[
https://issues.apache.org/jira/browse/FLINK-33900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799095#comment-17799095
]
Jiabao Sun commented on FLINK-33900:
I noticed there is information about JUnit4 in the stack trace, but we have
already completed the migration to JUnit5.
Regarding the NoResourceAvailableException issue, should it be outdated as we
are using the MiniClusterExtension, which releases resources after each test
method?
https://github.com/XComp/flink/actions/runs/7244405295/job/19733011527#step:12:12258
{noformat}
Dec 18 09:19:32 Suppressed: java.lang.AssertionError
Dec 18 09:19:32 at org.junit.Assert.fail(Assert.java:87)
Dec 18 09:19:32 at org.junit.Assert.assertTrue(Assert.java:42)
Dec 18 09:19:32 at org.junit.Assert.assertTrue(Assert.java:53)
Dec 18 09:19:32 at
org.apache.flink.table.planner.runtime.utils.StreamingWithStateTestBase.after(StreamingWithStateTestBase.scala:83)
{noformat}
> Multiple failures in WindowRankITCase due to NoResourceAvailableException
> -
>
> Key: FLINK-33900
> URL: https://issues.apache.org/jira/browse/FLINK-33900
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Priority: Major
> Labels: github-actions, test-stability
>
> [https://github.com/XComp/flink/actions/runs/7244405295/job/19733011527#step:12:14989]
> There are multiple tests in {{WindowRankITCase}} that fail due to a
> {{NoResourceAvailableException}} supposedly:
> {code:java}
> [...]
> Error: 09:19:33 09:19:32.966 [ERROR]
> WindowRankITCase.testTumbleWindowTVFWithOffset Time elapsed: 300.072 s <<<
> FAILURE!
> 14558Dec 18 09:19:33 org.opentest4j.MultipleFailuresError:
> 14559Dec 18 09:19:33 Multiple Failures (2 failures)
> 14560Dec 18 09:19:33 org.apache.flink.runtime.client.JobExecutionException:
> Job execution failed.
> 14561Dec 18 09:19:33 java.lang.AssertionError:
> 14562Dec 18 09:19:33 at
> org.junit.vintage.engine.execution.TestRun.getStoredResultOrSuccessful(TestRun.java:200)
> 14563Dec 18 09:19:33 at
> org.junit.vintage.engine.execution.RunListenerAdapter.fireExecutionFinished(RunListenerAdapter.java:248)
> 14564Dec 18 09:19:33 at
> org.junit.vintage.engine.execution.RunListenerAdapter.testFinished(RunListenerAdapter.java:214)
> 14565Dec 18 09:19:33 at
> org.junit.vintage.engine.execution.RunListenerAdapter.testFinished(RunListenerAdapter.java:88)
> 14566Dec 18 09:19:33 at
> org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87)
> 14567Dec 18 09:19:33 at
> org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225)
> 14568Dec 18 09:19:33 at
> org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
> 14569Dec 18 09:19:33 at
> org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222)
> 14570Dec 18 09:19:33 at
> org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38)
> 14571Dec 18 09:19:33 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372)
> 14572Dec 18 09:19:33 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 14573Dec 18 09:19:33 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 14574Dec 18 09:19:33 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 14575Dec 18 09:19:33 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 14576Dec 18 09:19:33 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 14577Dec 18 09:19:33 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 14578Dec 18 09:19:33 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 14579Dec 18 09:19:33 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 14580Dec 18 09:19:33 at org.junit.runners.Suite.runChild(Suite.java:128)
> 14581Dec 18 09:19:33 at org.junit.runners.Suite.runChild(Suite.java:27)
> 14582Dec 18 09:19:33 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 14583Dec 18 09:19:33 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 14584Dec 18 09:19:33 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 14585Dec 18 09:19:33 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 14586Dec 18 09:19:33 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 14587Dec 18 09:19:33 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 14588Dec 18 09:19:33 at
> org.junit.rules.ExternalResource$1.evaluate(Ext
Re: [PR] [FLINK-33888] Add release flink-1.18.1 [flink-web]
GOODBOY008 commented on code in PR #706: URL: https://github.com/apache/flink-web/pull/706#discussion_r1432936588 ## docs/content/posts/2023-12-20-release-1.18.1.md: ## @@ -0,0 +1,142 @@ +--- +title: "Apache Flink 1.18.1 Release Announcement" +date: "2023-12-20T00:00:00.000Z" +aliases: +- /news/2023/12/20/release-1.18.1.html + authors: Review Comment: ```suggestion authors: ``` Remove indentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33888] Add release flink-1.18.1 [flink-web]
GOODBOY008 commented on code in PR #706: URL: https://github.com/apache/flink-web/pull/706#discussion_r1432926072 ## docs/content/posts/2023-12-20-release-1.18.1.md: ## @@ -0,0 +1,142 @@ +--- +title: "Apache Flink 1.18.1 Release Announcement" +date: "2023-12-20T00:00:00.000Z" +aliases: +- /news/2023/12/20/release-1.18.1.html + authors: Review Comment: ```suggestion authors: ``` Remove indentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33888] Add release flink-1.18.1 [flink-web]
GOODBOY008 commented on code in PR #706: URL: https://github.com/apache/flink-web/pull/706#discussion_r1432926072 ## docs/content/posts/2023-12-20-release-1.18.1.md: ## @@ -0,0 +1,142 @@ +--- +title: "Apache Flink 1.18.1 Release Announcement" +date: "2023-12-20T00:00:00.000Z" +aliases: +- /news/2023/12/20/release-1.18.1.html + authors: Review Comment: ```suggestion authors: ``` Remove indentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32849][runtime][JUnit5 Migration] The resourcemanager package of flink-runtime module [flink]
1996fanrui commented on code in PR #23973:
URL: https://github.com/apache/flink/pull/23973#discussion_r1432906607
##
flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/ResourceManagerJobMasterTest.java:
##
@@ -191,12 +184,12 @@ public void
testRegisterJobMasterWithUnmatchedLeaderSessionId2() throws Exceptio
jobMasterGateway.getAddress(),
jobId,
TIMEOUT);
-assertTrue(unMatchedLeaderFuture.get() instanceof
RegistrationResponse.Failure);
+
assertThat(unMatchedLeaderFuture.get()).isInstanceOf(RegistrationResponse.Failure.class);
Review Comment:
```suggestion
assertThatFuture(unMatchedLeaderFuture)
.withThrowableOfType(RegistrationResponse.Failure.class)
```
##
flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/ResourceManagerJobMasterTest.java:
##
@@ -169,18 +166,14 @@ public void
testRegisterJobMasterWithUnmatchedLeaderSessionId1() throws Exceptio
jobMasterGateway.getAddress(),
jobId,
TIMEOUT);
-
-try {
-unMatchedLeaderFuture.get(5L, TimeUnit.SECONDS);
-fail("Should fail because we are using the wrong fencing token.");
-} catch (ExecutionException e) {
-assertTrue(ExceptionUtils.stripExecutionException(e) instanceof
FencingTokenException);
-}
+assertThatThrownBy(() -> unMatchedLeaderFuture.get(5L,
TimeUnit.SECONDS))
+.withFailMessage("Should fail because we are using the wrong
fencing token.")
+.hasCauseInstanceOf(FencingTokenException.class);
Review Comment:
How about using `assertThatFuture(invalidAddressFuture)`?
##
flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/ResourceManagerServiceImplTest.java:
##
@@ -309,12 +305,11 @@ public void revokeLeadership_stopExistLeader() throws
Exception {
leaderElection.notLeader();
// should terminate RM
-assertThat(terminateRmFuture.get(), is(leaderSessionId));
+assertThat(terminateRmFuture.get()).isSameAs(leaderSessionId);
Review Comment:
How about assertThatFuture?
##
flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/ResourceManagerJobMasterTest.java:
##
@@ -207,17 +200,16 @@ public void testRegisterJobMasterFromInvalidAddress()
throws Exception {
invalidAddress,
jobId,
TIMEOUT);
-assertTrue(
-invalidAddressFuture.get(5, TimeUnit.SECONDS)
-instanceof RegistrationResponse.Failure);
+assertThat(invalidAddressFuture.get(5, TimeUnit.SECONDS))
+.isInstanceOf(RegistrationResponse.Failure.class);
Review Comment:
```suggestion
assertThatFuture(invalidAddressFuture)
.failsWithin(5, TimeUnit.SECONDS)
.withThrowableOfType(RegistrationResponse.Failure.class)
```
##
flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManagerTest.java:
##
@@ -871,32 +868,34 @@ public void testRecoverWorkerFromPreviousAttempt() throws
Exception {
CompletableFuture
registerTaskExecutorFuture =
registerTaskExecutor(tmResourceId);
assertThat(
-
registerTaskExecutorFuture.get(TIMEOUT_SEC, TimeUnit.SECONDS),
-
instanceOf(RegistrationResponse.Success.class));
+registerTaskExecutorFuture.get(
+TIMEOUT_SEC,
TimeUnit.SECONDS))
+
.isInstanceOf(RegistrationResponse.Success.class);
});
}
};
}
/** Tests decline unknown worker registration. */
@Test
-public void testRegisterUnknownWorker() throws Exception {
+void testRegisterUnknownWorker() throws Exception {
new Context() {
{
runTest(
() -> {
CompletableFuture
registerTaskExecutorFuture =
registerTaskExecutor(ResourceID.generate());
assertThat(
-
registerTaskExecutorFuture.get(TIMEOUT_SEC, TimeUnit.SECONDS),
-
instanceOf(RegistrationResponse.Rejection.class));
+registerTaskExecutorFuture.get(
Review Comment:
Is the get needed?
##
flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/active/ActiveResourceManagerTest.java:
##
@@ -940,1
Re: [PR] [FLINK-33328] Allow TwoPhaseCommittingSink WithPreCommitTopology to alter the type of the Committable - mixin approach [flink]
pvary commented on PR #23912: URL: https://github.com/apache/flink/pull/23912#issuecomment-1864763831 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32850][flink-runtime][JUnit5 Migration] The io.network.partition package of flink-runtime module [flink]
Jiabao-Sun commented on PR #23974: URL: https://github.com/apache/flink/pull/23974#issuecomment-1864760862 Hi @reswqa, please help review it when you have time. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32850][flink-runtime][JUnit5 Migration] The io.network.partition package of flink-runtime module [flink]
flinkbot commented on PR #23974: URL: https://github.com/apache/flink/pull/23974#issuecomment-1864723222 ## CI report: * 1cf28b696a2555d7933944ce559fc90250be87b3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-33860) Implement restore tests for WindowTableFunction node
[ https://issues.apache.org/jira/browse/FLINK-33860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz closed FLINK-33860. Fix Version/s: 1.19.0 Resolution: Implemented Implemented in 1a8b8d512c213ba330f28eca663bc77e2369b61b..cabb28d25c4c58af3ee23fc4a63f9564aefd6146 > Implement restore tests for WindowTableFunction node > > > Key: FLINK-33860 > URL: https://issues.apache.org/jira/browse/FLINK-33860 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Bonnie Varghese >Assignee: Bonnie Varghese >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33860] Implement restore tests for WindowTableFunction node [flink]
dawidwys merged PR #23936: URL: https://github.com/apache/flink/pull/23936 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-32850][flink-runtime][JUnit5 Migration] The io.network.partition package of flink-runtime module [flink]
Jiabao-Sun opened a new pull request, #23974: URL: https://github.com/apache/flink/pull/23974 ## What is the purpose of the change [FLINK-32850][flink-runtime][JUnit5 Migration] The io.network.partition package of flink-runtime module ## Brief change log [FLINK-32850][flink-runtime][JUnit5 Migration] The io.network.partition package of flink-runtime module ## Verifying this change This change is already covered by existing tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32849][runtime][JUnit5 Migration] The resourcemanager package of flink-runtime module [flink]
flinkbot commented on PR #23973: URL: https://github.com/apache/flink/pull/23973#issuecomment-1864686150 ## CI report: * 0aa76f5a80cc057681f6d2407a2c0774a78a9b0e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33823] Make PlannerQueryOperation SQL serializable [flink]
dawidwys merged PR #23948: URL: https://github.com/apache/flink/pull/23948 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-33644) FLIP-393: Make QueryOperations SQL serializable
[ https://issues.apache.org/jira/browse/FLINK-33644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz closed FLINK-33644. Resolution: Implemented > FLIP-393: Make QueryOperations SQL serializable > --- > > Key: FLINK-33644 > URL: https://issues.apache.org/jira/browse/FLINK-33644 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Fix For: 1.19.0 > > > https://cwiki.apache.org/confluence/x/4guZE -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33823) Serialize PlannerQueryOperation into SQL
[ https://issues.apache.org/jira/browse/FLINK-33823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz closed FLINK-33823. Resolution: Implemented Implemented in 5919251d7a94264a6a72c31de0716b3f72d65437 > Serialize PlannerQueryOperation into SQL > > > Key: FLINK-33823 > URL: https://issues.apache.org/jira/browse/FLINK-33823 > Project: Flink > Issue Type: Sub-task >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32849) [JUnit5 Migration] The resourcemanager package of flink-runtime module
[ https://issues.apache.org/jira/browse/FLINK-32849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-32849: --- Labels: pull-request-available (was: ) > [JUnit5 Migration] The resourcemanager package of flink-runtime module > -- > > Key: FLINK-32849 > URL: https://issues.apache.org/jira/browse/FLINK-32849 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Rui Fan >Assignee: RocMarshal >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-32849][runtime][JUnit5 Migration] The resourcemanager package of flink-runtime module [flink]
RocMarshal opened a new pull request, #23973: URL: https://github.com/apache/flink/pull/23973 ## What is the purpose of the change [runtime][JUnit5 Migration] The resourcemanager package of flink-runtime module ## Brief change log [runtime][JUnit5 Migration] The resourcemanager package of flink-runtime module ## Verifying this change This change is already covered by existing tests, such as *(please describe tests)*. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33916][ci] Adds nightly workflow for most-recently published Flink version [flink]
flinkbot commented on PR #23972: URL: https://github.com/apache/flink/pull/23972#issuecomment-1864564943 ## CI report: * 1e973fb1650c919a738aa9fe69a0e6f9d7b9bdfb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33915][ci] Adds nightly workflow for the master branch [flink]
flinkbot commented on PR #23971: URL: https://github.com/apache/flink/pull/23971#issuecomment-1864563284 ## CI report: * 4cd00e3a6426196125c1ed5e5b03f77d7df6fba8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-33916) Workflow: Add nightly build for release-1.18
[
https://issues.apache.org/jira/browse/FLINK-33916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl reassigned FLINK-33916:
-
Assignee: Matthias Pohl
> Workflow: Add nightly build for release-1.18
>
>
> Key: FLINK-33916
> URL: https://issues.apache.org/jira/browse/FLINK-33916
> Project: Flink
> Issue Type: Sub-task
> Components: Build System / CI
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: github-actions, pull-request-available
>
> Add nightly workflow for {{{}release-1.18{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33914][ci] Adds basic Flink CI workflow [flink]
flinkbot commented on PR #23970: URL: https://github.com/apache/flink/pull/23970#issuecomment-1864549461 ## CI report: * ff8cb87a2880efbf3c210b31fab2d79a6c57ddee UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33916) Workflow: Add nightly build for release-1.18
[
https://issues.apache.org/jira/browse/FLINK-33916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-33916:
---
Labels: github-actions pull-request-available (was: github-actions)
> Workflow: Add nightly build for release-1.18
>
>
> Key: FLINK-33916
> URL: https://issues.apache.org/jira/browse/FLINK-33916
> Project: Flink
> Issue Type: Sub-task
> Components: Build System / CI
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Priority: Major
> Labels: github-actions, pull-request-available
>
> Add nightly workflow for {{{}release-1.18{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-33915) Workflow: Add nightly build for the dev version (currently called "master")
[
https://issues.apache.org/jira/browse/FLINK-33915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl reassigned FLINK-33915:
-
Assignee: Matthias Pohl
> Workflow: Add nightly build for the dev version (currently called "master")
> ---
>
> Key: FLINK-33915
> URL: https://issues.apache.org/jira/browse/FLINK-33915
> Project: Flink
> Issue Type: Sub-task
> Components: Build System / CI
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: github-actions, pull-request-available
>
> The nightly builds run on master and the two most-recently released versions
> of Flink as those are the supported versions. This logic is currently
> captured in
> [flink-ci/git-repo-sync:sync_repo.sh|https://github.com/flink-ci/git-repo-sync/blob/master/sync_repo.sh#L28].
> In
> [FLIP-396|https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink%27s+current+Azure+CI+infrastructure]
> we decided to go ahead and provide nightly builds for {{master}} and
> {{{}release-1.18{}}}. This issue is about providing the nightly workflow for
> {{master}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-33916][ci] Adds nightly workflow for most-recently published Flink version [flink]
XComp opened a new pull request, #23972: URL: https://github.com/apache/flink/pull/23972 Based on the following PR(s): * https://github.com/apache/flink/pull/23961 * https://github.com/apache/flink/pull/23962 * https://github.com/apache/flink/pull/23963 * https://github.com/apache/flink/pull/23964 * https://github.com/apache/flink/pull/23965 * https://github.com/apache/flink/pull/23970 ## What is the purpose of the change Creates an extended CI workflow that runs on master ## Brief change log * Adds release-1.18 workflow configuration * Adds release-1.18 workflow ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-33915][ci] Adds nightly workflow for the master branch [flink]
XComp opened a new pull request, #23971: URL: https://github.com/apache/flink/pull/23971 Based on the following PR(s): * https://github.com/apache/flink/pull/23961 * https://github.com/apache/flink/pull/23962 * https://github.com/apache/flink/pull/23963 * https://github.com/apache/flink/pull/23964 * https://github.com/apache/flink/pull/23965 * https://github.com/apache/flink/pull/23970 ## What is the purpose of the change Creates an extended CI workflow that runs on master ## Brief change log * Adds custom actions for selecting the release branch and commit hash * Adds custom action for selecting the workflow configuration * Adds fallback workflow configuration * Adds workflow for utilizing the workflow configuration ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33915) Workflow: Add nightly build for the dev version (currently called "master")
[
https://issues.apache.org/jira/browse/FLINK-33915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-33915:
---
Labels: github-actions pull-request-available (was: github-actions)
> Workflow: Add nightly build for the dev version (currently called "master")
> ---
>
> Key: FLINK-33915
> URL: https://issues.apache.org/jira/browse/FLINK-33915
> Project: Flink
> Issue Type: Sub-task
> Components: Build System / CI
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Priority: Major
> Labels: github-actions, pull-request-available
>
> The nightly builds run on master and the two most-recently released versions
> of Flink as those are the supported versions. This logic is currently
> captured in
> [flink-ci/git-repo-sync:sync_repo.sh|https://github.com/flink-ci/git-repo-sync/blob/master/sync_repo.sh#L28].
> In
> [FLIP-396|https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink%27s+current+Azure+CI+infrastructure]
> we decided to go ahead and provide nightly builds for {{master}} and
> {{{}release-1.18{}}}. This issue is about providing the nightly workflow for
> {{master}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-33912) Template: Add CI template for pre-compile steps
[ https://issues.apache.org/jira/browse/FLINK-33912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl reassigned FLINK-33912: - Assignee: Matthias Pohl > Template: Add CI template for pre-compile steps > --- > > Key: FLINK-33912 > URL: https://issues.apache.org/jira/browse/FLINK-33912 > Project: Flink > Issue Type: Sub-task > Components: Build System / CI >Affects Versions: 1.19.0 >Reporter: Matthias Pohl >Assignee: Matthias Pohl >Priority: Major > Labels: github-actions > > We want to have a template that triggers all checks that do not require > compilation. Those quick checks (e.g. code format) can run without waiting > for the compilation step to succeed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33913) Template: Add CI template for running Flink's test suite
[ https://issues.apache.org/jira/browse/FLINK-33913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl reassigned FLINK-33913: - Assignee: Matthias Pohl > Template: Add CI template for running Flink's test suite > > > Key: FLINK-33913 > URL: https://issues.apache.org/jira/browse/FLINK-33913 > Project: Flink > Issue Type: Sub-task > Components: Build System / CI >Affects Versions: 1.19.0 >Reporter: Matthias Pohl >Assignee: Matthias Pohl >Priority: Major > Labels: github-actions > > We want to have a template that runs the entire Flink test suite. -- This message was sent by Atlassian Jira (v8.20.10#820010)
