Re: [PR] [FLINK-32513][core] Add predecessor caching [flink]
zhuzhurk commented on code in PR #24475: URL: https://github.com/apache/flink/pull/24475#discussion_r1533318682 ## flink-streaming-java/src/test/java/org/apache/flink/streaming/api/transformations/GetTransitivePredecessorsTest.java: ## @@ -0,0 +1,147 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.streaming.api.transformations; + +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.dag.Transformation; +import org.apache.flink.api.java.typeutils.GenericTypeInfo; +import org.apache.flink.streaming.api.operators.AbstractStreamOperator; +import org.apache.flink.streaming.api.operators.OneInputStreamOperator; +import org.apache.flink.streaming.api.operators.TwoInputStreamOperator; +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord; +import org.apache.flink.util.TestLogger; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +/** Tests for {@code getTransitivePredecessors} method of {@link Transformation}. */ +class GetTransitivePredecessorsTest extends TestLogger { Review Comment: ` extends TestLogger` is no longer needed since JUnit5. The logging will be enabled automatically. ## flink-streaming-java/src/test/java/org/apache/flink/streaming/api/transformations/GetTransitivePredecessorsTest.java: ## @@ -0,0 +1,147 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.streaming.api.transformations; + +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.dag.Transformation; +import org.apache.flink.api.java.typeutils.GenericTypeInfo; +import org.apache.flink.streaming.api.operators.AbstractStreamOperator; +import org.apache.flink.streaming.api.operators.OneInputStreamOperator; +import org.apache.flink.streaming.api.operators.TwoInputStreamOperator; +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord; +import org.apache.flink.util.TestLogger; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +/** Tests for {@code getTransitivePredecessors} method of {@link Transformation}. */ +class GetTransitivePredecessorsTest extends TestLogger { + +private Transformation commonNode; +private Transformation midNode; + +@BeforeEach +void setup() { +commonNode = new TestTransformation<>("commonNode", new MockIntegerTypeInfo(), 1); +midNode = +new OneInputTransformation<>( +commonNode, +"midNode", +new DummyOneInputOperator(), +new MockIntegerTypeInfo(), +1); +} + +@Test +void testTwoInputTransformation() { +Transformation topNode = +new TwoInputTransformation<>( +commonNode, +midNode, +"topNode", +new DummyTwoInputOperator<>(), +midNode.getOutputType(), +1); +List> predecessors = topNode.getTransitivePredece
[jira] [Commented] (FLINK-34898) Cannot create named STRUCT with a single field
[ https://issues.apache.org/jira/browse/FLINK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829398#comment-17829398 ] Chloe He commented on FLINK-34898: -- [~liyubin117] Yes this does work for me, but this generates a named STRUCT with two fields, `a` and `b`. I want to construct a named STRUCT with a single field, i.e., something like {code:java} SELECT CAST((1) as ROW) AS row1; {code} This is not supported. I am using 1.19. > Cannot create named STRUCT with a single field > -- > > Key: FLINK-34898 > URL: https://issues.apache.org/jira/browse/FLINK-34898 > Project: Flink > Issue Type: Bug >Reporter: Chloe He >Priority: Major > Attachments: image-2024-03-21-12-00-00-183.png > > > I'm trying to create named structs using Flink SQL and I found a previous > ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use > of the following syntax: > {code:java} > SELECT CAST(('a', 1) as ROW) AS row1; > {code} > However, my named struct has a single field and effectively it should look > something like `\{"a": 1}`. I can't seem to be able to find a way to > construct this. I have experimented with a few different syntax and it either > throws parsing error or casting error: > {code:java} > Cast function cannot convert value of type INTEGER to type > RecordType(VARCHAR(2147483647) a) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34901) Improve the performance of the update Postgres database
ouyangwulin created FLINK-34901: --- Summary: Improve the performance of the update Postgres database Key: FLINK-34901 URL: https://issues.apache.org/jira/browse/FLINK-34901 Project: Flink Issue Type: Improvement Components: Connectors / JDBC Reporter: ouyangwulin When writing Postgres data using flinker-connector-jdbc, using upsert mode, we need only update no uniqueColumns to improve performance. Whether you can modify the code is as follows: String updateClause = {code:java} Arrays.stream(fieldNames) .filter(fieldName -> !Arrays.asList(uniqueKeyFields).contains(fieldName)) .map(f -> quoteIdentifier(f) + "=EXCLUDED." + quoteIdentifier(f)) .collect(Collectors.joining(", "));{code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34537] Autoscaler JDBC Support HikariPool [flink-kubernetes-operator]
czy006 commented on code in PR #785: URL: https://github.com/apache/flink-kubernetes-operator/pull/785#discussion_r1533316957 ## flink-autoscaler-standalone/src/main/java/org/apache/flink/autoscaler/standalone/utils/HikariJDBCUtil.java: ## @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.autoscaler.standalone.utils; + +import org.apache.flink.configuration.Configuration; + +import com.zaxxer.hikari.HikariConfig; +import com.zaxxer.hikari.HikariDataSource; + +import static org.apache.flink.autoscaler.standalone.config.AutoscalerStandaloneOptions.JDBC_PASSWORD_ENV_VARIABLE; +import static org.apache.flink.autoscaler.standalone.config.AutoscalerStandaloneOptions.JDBC_URL; +import static org.apache.flink.autoscaler.standalone.config.AutoscalerStandaloneOptions.JDBC_USERNAME; +import static org.apache.flink.util.Preconditions.checkArgument; + +public class HikariJDBCUtil { + +public static HikariDataSource getHikariDataSource(Configuration conf, String errMsg) { Review Comment: now include this part of the test in unit tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34537] Autoscaler JDBC Support HikariPool [flink-kubernetes-operator]
czy006 commented on code in PR #785: URL: https://github.com/apache/flink-kubernetes-operator/pull/785#discussion_r1533316345 ## flink-autoscaler-standalone/src/main/java/org/apache/flink/autoscaler/standalone/utils/HikariJDBCUtil.java: ## @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.autoscaler.standalone.utils; + +import org.apache.flink.configuration.Configuration; + +import com.zaxxer.hikari.HikariConfig; +import com.zaxxer.hikari.HikariDataSource; + +import static org.apache.flink.autoscaler.standalone.config.AutoscalerStandaloneOptions.JDBC_PASSWORD_ENV_VARIABLE; +import static org.apache.flink.autoscaler.standalone.config.AutoscalerStandaloneOptions.JDBC_URL; +import static org.apache.flink.autoscaler.standalone.config.AutoscalerStandaloneOptions.JDBC_USERNAME; +import static org.apache.flink.util.Preconditions.checkArgument; + +public class HikariJDBCUtil { + +public static HikariDataSource getHikariDataSource(Configuration conf, String errMsg) { +final var jdbcUrl = conf.get(JDBC_URL); +checkArgument(jdbcUrl != null, errMsg, JDBC_URL.key()); +var user = conf.get(JDBC_USERNAME); +var password = System.getenv().get(conf.get(JDBC_PASSWORD_ENV_VARIABLE)); +HikariConfig hikariConfig = new HikariConfig(); +hikariConfig.setJdbcUrl(jdbcUrl); +hikariConfig.setUsername(user); +hikariConfig.setPassword(password); Review Comment: Now that I've added it to my tests, Derby works fine with its empty user name and password, and I didn't test the other databases -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34725) Dockerfiles for release publishing has incorrect config.yaml path
[ https://issues.apache.org/jira/browse/FLINK-34725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829394#comment-17829394 ] lincoln lee commented on FLINK-34725: - Thanks for fixing this! > Dockerfiles for release publishing has incorrect config.yaml path > - > > Key: FLINK-34725 > URL: https://issues.apache.org/jira/browse/FLINK-34725 > Project: Flink > Issue Type: Bug > Components: flink-docker >Affects Versions: 1.19.0 >Reporter: lincoln lee >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0, 1.19.1 > > > An issue found when do docker image publishing, unexpected error msg: > {code:java} > sed: can't read /config.yaml: No such file or directory{code} > > also found in flink-docker/master daily Publish SNAPSHOTs action: > [https://github.com/apache/flink-docker/actions/runs/8210534289/job/22458150514#step:8:588] > [https://github.com/apache/flink-docker/actions/runs/8210534289/job/22458150322#step:8:549] > > This related to changes by https://issues.apache.org/jira/browse/FLINK-34205 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
lincoln-lil commented on PR #24517: URL: https://github.com/apache/flink/pull/24517#issuecomment-2011312055 @masteryhx thank you! Another input: there was no snapshot file generated for this case related data directory in previous releases, so I'm a bit skeptical that it's due to other changes. ![image](https://github.com/apache/flink/assets/3712895/b7e913aa-3f0a-437e-9e1c-7b016304fb10) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-34725) Dockerfiles for release publishing has incorrect config.yaml path
[ https://issues.apache.org/jira/browse/FLINK-34725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu closed FLINK-34725. --- Resolution: Fixed master: 3f4a80989fe7243983926f09fac2283f6fa63693 release-1.19: f53c5628e43777b4b924ec81224acc3df938800a > Dockerfiles for release publishing has incorrect config.yaml path > - > > Key: FLINK-34725 > URL: https://issues.apache.org/jira/browse/FLINK-34725 > Project: Flink > Issue Type: Bug > Components: flink-docker >Affects Versions: 1.19.0 >Reporter: lincoln lee >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0, 1.19.1 > > > An issue found when do docker image publishing, unexpected error msg: > {code:java} > sed: can't read /config.yaml: No such file or directory{code} > > also found in flink-docker/master daily Publish SNAPSHOTs action: > [https://github.com/apache/flink-docker/actions/runs/8210534289/job/22458150514#step:8:588] > [https://github.com/apache/flink-docker/actions/runs/8210534289/job/22458150322#step:8:549] > > This related to changes by https://issues.apache.org/jira/browse/FLINK-34205 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34725][dist] Fix wrong config file dir when `config-parser-utils.sh` script attempt to retrieve the Java Home. [flink]
zhuzhurk closed pull request #24527: [FLINK-34725][dist] Fix wrong config file dir when `config-parser-utils.sh` script attempt to retrieve the Java Home. URL: https://github.com/apache/flink/pull/24527 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-34731) Remove SpeculativeScheduler and incorporate its features into AdaptiveBatchScheduler
[ https://issues.apache.org/jira/browse/FLINK-34731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu reassigned FLINK-34731: --- Assignee: Junrui Li > Remove SpeculativeScheduler and incorporate its features into > AdaptiveBatchScheduler > > > Key: FLINK-34731 > URL: https://issues.apache.org/jira/browse/FLINK-34731 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Fix For: 1.20.0 > > > Presently, speculative execution is exposed to users as a feature of the > AdaptiveBatchScheduler. > To streamline our codebase and reduce maintenance overhead, this ticket will > consolidate the SpeculativeScheduler into the AdaptiveBatchScheduler, > eliminating the need for a separate SpeculativeScheduler class. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34516] Move CheckpointingMode to flink-core [flink]
masteryhx commented on code in PR #24381: URL: https://github.com/apache/flink/pull/24381#discussion_r1533247878 ## flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/CheckpointConfig.java: ## @@ -177,20 +177,44 @@ public boolean isCheckpointingEnabled() { * Gets the checkpointing mode (exactly-once vs. at-least-once). * * @return The checkpointing mode. + * @deprecated Use {@link #getConsistencyMode} instead. */ -public CheckpointingMode getCheckpointingMode() { +@Deprecated +public org.apache.flink.streaming.api.CheckpointingMode getCheckpointingMode() { return configuration.get(ExecutionCheckpointingOptions.CHECKPOINTING_MODE); } /** * Sets the checkpointing mode (exactly-once vs. at-least-once). * * @param checkpointingMode The checkpointing mode. + * @deprecated Use {@link #setConsistencyMode} instead. */ -public void setCheckpointingMode(CheckpointingMode checkpointingMode) { +@Deprecated +public void setCheckpointingMode( +org.apache.flink.streaming.api.CheckpointingMode checkpointingMode) { configuration.set(ExecutionCheckpointingOptions.CHECKPOINTING_MODE, checkpointingMode); } +/** + * Gets the checkpointing consistency mode (exactly-once vs. at-least-once). + * + * @return The checkpointing mode. + */ +public CheckpointingMode getConsistencyMode() { Review Comment: This looks good to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34516] Move CheckpointingMode to flink-core [flink]
Zakelly commented on code in PR #24381: URL: https://github.com/apache/flink/pull/24381#discussion_r1533244572 ## flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/CheckpointConfig.java: ## @@ -177,20 +177,44 @@ public boolean isCheckpointingEnabled() { * Gets the checkpointing mode (exactly-once vs. at-least-once). * * @return The checkpointing mode. + * @deprecated Use {@link #getConsistencyMode} instead. */ -public CheckpointingMode getCheckpointingMode() { +@Deprecated +public org.apache.flink.streaming.api.CheckpointingMode getCheckpointingMode() { return configuration.get(ExecutionCheckpointingOptions.CHECKPOINTING_MODE); } /** * Sets the checkpointing mode (exactly-once vs. at-least-once). * * @param checkpointingMode The checkpointing mode. + * @deprecated Use {@link #setConsistencyMode} instead. */ -public void setCheckpointingMode(CheckpointingMode checkpointingMode) { +@Deprecated +public void setCheckpointingMode( +org.apache.flink.streaming.api.CheckpointingMode checkpointingMode) { configuration.set(ExecutionCheckpointingOptions.CHECKPOINTING_MODE, checkpointingMode); } +/** + * Gets the checkpointing consistency mode (exactly-once vs. at-least-once). + * + * @return The checkpointing mode. + */ +public CheckpointingMode getConsistencyMode() { Review Comment: I'd change the API name `get/setConsistencyMode` to `get/setCheckpointingConsistencyMode` and add some description showing the relationship between the name `CheckpointingConsistencyMode` and `CheckpointingMode`. But I'd suggest keep `CheckpointingMode` as it is since it is a first class citizen of Flink and widely accepted by users. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-34647) Path normalization is allocation intensive
[ https://issues.apache.org/jira/browse/FLINK-34647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hangxiang Yu resolved FLINK-34647. -- Fix Version/s: 1.20.0 Assignee: David Schlosnagle Resolution: Fixed Merged 32144689 into master > Path normalization is allocation intensive > -- > > Key: FLINK-34647 > URL: https://issues.apache.org/jira/browse/FLINK-34647 > Project: Flink > Issue Type: Improvement > Components: API / Core, Runtime / Checkpointing >Reporter: David Schlosnagle >Assignee: David Schlosnagle >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > Attachments: image-2024-03-11-20-38-12-573.png > > > While investigating allocation stalls and GC pressure of a Flink streaming > pipeline, I noticed significant allocations in JFR from Flink path > normalization: > !image-2024-03-11-20-38-12-573.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34647][core] Optimize Path normalization [flink]
masteryhx closed pull request #24473: [FLINK-34647][core] Optimize Path normalization URL: https://github.com/apache/flink/pull/24473 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34516] Move CheckpointingMode to flink-core [flink]
masteryhx commented on code in PR #24381: URL: https://github.com/apache/flink/pull/24381#discussion_r1533238343 ## flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/CheckpointConfig.java: ## @@ -177,20 +177,44 @@ public boolean isCheckpointingEnabled() { * Gets the checkpointing mode (exactly-once vs. at-least-once). * * @return The checkpointing mode. + * @deprecated Use {@link #getConsistencyMode} instead. */ -public CheckpointingMode getCheckpointingMode() { +@Deprecated +public org.apache.flink.streaming.api.CheckpointingMode getCheckpointingMode() { return configuration.get(ExecutionCheckpointingOptions.CHECKPOINTING_MODE); } /** * Sets the checkpointing mode (exactly-once vs. at-least-once). * * @param checkpointingMode The checkpointing mode. + * @deprecated Use {@link #setConsistencyMode} instead. */ -public void setCheckpointingMode(CheckpointingMode checkpointingMode) { +@Deprecated +public void setCheckpointingMode( +org.apache.flink.streaming.api.CheckpointingMode checkpointingMode) { configuration.set(ExecutionCheckpointingOptions.CHECKPOINTING_MODE, checkpointingMode); } +/** + * Gets the checkpointing consistency mode (exactly-once vs. at-least-once). + * + * @return The checkpointing mode. + */ +public CheckpointingMode getConsistencyMode() { Review Comment: I just have a minor concern that users may lost about `CheckpointingMode` and `ConsistencyMode`. How about renaming this to `CheckpointingConsistencyMode` and adding some descriptions about the relationship between them ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-34702) Rank should not convert to StreamExecDuplicate when the input is not insert only
[ https://issues.apache.org/jira/browse/FLINK-34702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829374#comment-17829374 ] Jacky Lau edited comment on FLINK-34702 at 3/21/24 4:32 AM: When adopting Solution 3 it was found to be unviable for two main reasons: # The rule cannot be placed within the physical rewrite phase. This approach is somewhat tricky, as noted in my poc. The reason is that the MiniBatchAssigner operator is whether the proctime / eventime is related to the downstream operators. The current Rank operator does not require watermarks, whereas the Deduplicate operator for rowtime does. This can be observed in the return value of requireWatermark. See the plan change in the test {{{}testMiniBatchInferFirstRowOnRowtime{}}}. # The RelTraitSet produced by the Deduplicate operator differs from that of the Rank operator. Although we can modify the RelTrait of the deduplication operator using the code below, there is no way to change the RelTraits of other downstream operators belonging to the deduplication operator, unless we run the logic of the FlinkChangelogModeInferenceProgram again. The local-global two-phase rule can be dealt with within the physical rewrite phase because the RelTraitSet of the local aggregate is same with the upstream node. See the plan change in the test testUpdatableRankWithDeduplicate {code:java} ModifyKindSetTrait modifyKindSetTrait; UpdateKindTrait updateKindTrait; if (!isLastRow && !isRowtime) { // only proctime first row deduplicate does not produce UPDATE changes modifyKindSetTrait = ModifyKindSetTrait.INSERT_ONLY(); updateKindTrait = UpdateKindTrait.NONE(); } else { // other deduplicate produce update changes modifyKindSetTrait = ModifyKindSetTrait.ALL_CHANGES(); updateKindTrait = rank.getTraitSet() .getTrait(UpdateKindTraitDef.INSTANCE()); } RelTraitSet duplicateTraitSet = rank.getTraitSet() .replace(modifyKindSetTrait) .replace(updateKindTrait); {code} was (Author: jackylau): When adopting Solution 3 it was found to be unviable for two main reasons: # The rule cannot be placed within the physical rewrite phase. This approach is somewhat tricky, as noted in my poc. The reason is that the MiniBatchAssigner operator is whether the proctime / eventime is related to the downstream operators. The current Rank operator does not require watermarks, whereas the Deduplicate operator for rowtime does. This can be observed in the return value of requireWatermark. See the plan change in the test {{{}testMiniBatchInferFirstRowOnRowtime{}}}. # The RelTraitSet produced by the Deduplicate operator differs from that of the Rank operator. Although we can modify the RelTrait of the deduplication operator using the code below, there is no way to change the RelTraits of other downstream operators belonging to the deduplication operator, unless we un the logic of the FlinkChangelogModeInferenceProgram again. The local-global two-phase rule can be dealt with within the physical rewrite phase because the RelTraitSet of the local aggregate is same with the upstream node. See the plan change in the test testUpdatableRankWithDeduplicate {code:java} ModifyKindSetTrait modifyKindSetTrait; UpdateKindTrait updateKindTrait; if (!isLastRow && !isRowtime) { // only proctime first row deduplicate does not produce UPDATE changes modifyKindSetTrait = ModifyKindSetTrait.INSERT_ONLY(); updateKindTrait = UpdateKindTrait.NONE(); } else { // other deduplicate produce update changes modifyKindSetTrait = ModifyKindSetTrait.ALL_CHANGES(); updateKindTrait = rank.getTraitSet() .getTrait(UpdateKindTraitDef.INSTANCE()); } RelTraitSet duplicateTraitSet = rank.getTraitSet() .replace(modifyKindSetTrait) .replace(updateKindTrait); {code} > Rank should not convert to StreamExecDuplicate when the input is not insert > only > > > Key: FLINK-34702 > URL: https://issues.apache.org/jira/browse/FLINK-34702 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.20.0 >Reporter: Jacky Lau >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > > {code:java} > @Test > def testSimpleFirstRowOnBuiltinProctime1(): Unit = { > val sqlQuery = > """ > |SELECT * > |FROM ( > | SELECT *, > |ROW_NUMBER() OVER (PARTITION BY a ORDER BY PROCTIME() ASC) as > rowNum > | FROM (select a, count(b) as b from MyTable group by a) > |) > |WHERE rowNum = 1 > """.stripMargin > util.verifyExecPlan(sqlQuery) > } {code} > Exception: > org.apache.flink.table.api.TableException: StreamPhysicalDeduplic
[jira] [Comment Edited] (FLINK-34702) Rank should not convert to StreamExecDuplicate when the input is not insert only
[ https://issues.apache.org/jira/browse/FLINK-34702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829374#comment-17829374 ] Jacky Lau edited comment on FLINK-34702 at 3/21/24 4:31 AM: When adopting Solution 3 it was found to be unviable for two main reasons: # The rule cannot be placed within the physical rewrite phase. This approach is somewhat tricky, as noted in my poc. The reason is that the MiniBatchAssigner operator is whether the proctime / eventime is related to the downstream operators. The current Rank operator does not require watermarks, whereas the Deduplicate operator for rowtime does. This can be observed in the return value of requireWatermark. See the plan change in the test {{{}testMiniBatchInferFirstRowOnRowtime{}}}. # The RelTraitSet produced by the Deduplicate operator differs from that of the Rank operator. Although we can modify the RelTrait of the deduplication operator using the code below, there is no way to change the RelTraits of other downstream operators belonging to the deduplication operator, unless we un the logic of the FlinkChangelogModeInferenceProgram again. The local-global two-phase rule can be dealt with within the physical rewrite phase because the RelTraitSet of the local aggregate is same with the upstream node. See the plan change in the test testUpdatableRankWithDeduplicate {code:java} ModifyKindSetTrait modifyKindSetTrait; UpdateKindTrait updateKindTrait; if (!isLastRow && !isRowtime) { // only proctime first row deduplicate does not produce UPDATE changes modifyKindSetTrait = ModifyKindSetTrait.INSERT_ONLY(); updateKindTrait = UpdateKindTrait.NONE(); } else { // other deduplicate produce update changes modifyKindSetTrait = ModifyKindSetTrait.ALL_CHANGES(); updateKindTrait = rank.getTraitSet() .getTrait(UpdateKindTraitDef.INSTANCE()); } RelTraitSet duplicateTraitSet = rank.getTraitSet() .replace(modifyKindSetTrait) .replace(updateKindTrait); {code} was (Author: jackylau): When opting for Solution 3, it was found to be unviable for two main reasons: # The rule cannot be placed within the physical rewrite phase. This approach is somewhat tricky, as noted in my poc. The reason is that the MiniBatchAssigner operator is whether the proctime / eventime is related to the downstream operators. The current Rank operator does not require watermarks, whereas the Deduplicate operator for rowtime does. This can be observed in the return value of requireWatermark. See the plan change in the test {{{}testMiniBatchInferFirstRowOnRowtime{}}}. # The RelTraitSet produced by the Deduplicate operator differs from that of the Rank operator. Although we can modify the RelTrait of the deduplication operator using the code below, there is no way to change the RelTraits of other downstream operators belonging to the deduplication operator, unless we un the logic of the FlinkChangelogModeInferenceProgram again. The local-global two-phase rule can be dealt with within the physical rewrite phase because the RelTraitSet of the local aggregate is same with the upstream node. See the plan change in the test testUpdatableRankWithDeduplicate {code:java} ModifyKindSetTrait modifyKindSetTrait; UpdateKindTrait updateKindTrait; if (!isLastRow && !isRowtime) { // only proctime first row deduplicate does not produce UPDATE changes modifyKindSetTrait = ModifyKindSetTrait.INSERT_ONLY(); updateKindTrait = UpdateKindTrait.NONE(); } else { // other deduplicate produce update changes modifyKindSetTrait = ModifyKindSetTrait.ALL_CHANGES(); updateKindTrait = rank.getTraitSet() .getTrait(UpdateKindTraitDef.INSTANCE()); } RelTraitSet duplicateTraitSet = rank.getTraitSet() .replace(modifyKindSetTrait) .replace(updateKindTrait); {code} > Rank should not convert to StreamExecDuplicate when the input is not insert > only > > > Key: FLINK-34702 > URL: https://issues.apache.org/jira/browse/FLINK-34702 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.20.0 >Reporter: Jacky Lau >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > > {code:java} > @Test > def testSimpleFirstRowOnBuiltinProctime1(): Unit = { > val sqlQuery = > """ > |SELECT * > |FROM ( > | SELECT *, > |ROW_NUMBER() OVER (PARTITION BY a ORDER BY PROCTIME() ASC) as > rowNum > | FROM (select a, count(b) as b from MyTable group by a) > |) > |WHERE rowNum = 1 > """.stripMargin > util.verifyExecPlan(sqlQuery) > } {code} > Exception: > org.apache.flink.table.api.TableException: StreamPhysicalDedupl
[jira] [Commented] (FLINK-34702) Rank should not convert to StreamExecDuplicate when the input is not insert only
[ https://issues.apache.org/jira/browse/FLINK-34702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829374#comment-17829374 ] Jacky Lau commented on FLINK-34702: --- When opting for Solution 3, it was found to be unviable for two main reasons: # The rule cannot be placed within the physical rewrite phase. This approach is somewhat tricky, as noted in my poc. The reason is that the MiniBatchAssigner operator is whether the proctime / eventime is related to the downstream operators. The current Rank operator does not require watermarks, whereas the Deduplicate operator for rowtime does. This can be observed in the return value of requireWatermark. See the plan change in the test {{{}testMiniBatchInferFirstRowOnRowtime{}}}. # The RelTraitSet produced by the Deduplicate operator differs from that of the Rank operator. Although we can modify the RelTrait of the deduplication operator using the code below, there is no way to change the RelTraits of other downstream operators belonging to the deduplication operator, unless we un the logic of the FlinkChangelogModeInferenceProgram again. The local-global two-phase rule can be dealt with within the physical rewrite phase because the RelTraitSet of the local aggregate is same with the upstream node. See the plan change in the test testUpdatableRankWithDeduplicate {code:java} ModifyKindSetTrait modifyKindSetTrait; UpdateKindTrait updateKindTrait; if (!isLastRow && !isRowtime) { // only proctime first row deduplicate does not produce UPDATE changes modifyKindSetTrait = ModifyKindSetTrait.INSERT_ONLY(); updateKindTrait = UpdateKindTrait.NONE(); } else { // other deduplicate produce update changes modifyKindSetTrait = ModifyKindSetTrait.ALL_CHANGES(); updateKindTrait = rank.getTraitSet() .getTrait(UpdateKindTraitDef.INSTANCE()); } RelTraitSet duplicateTraitSet = rank.getTraitSet() .replace(modifyKindSetTrait) .replace(updateKindTrait); {code} > Rank should not convert to StreamExecDuplicate when the input is not insert > only > > > Key: FLINK-34702 > URL: https://issues.apache.org/jira/browse/FLINK-34702 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.20.0 >Reporter: Jacky Lau >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > > {code:java} > @Test > def testSimpleFirstRowOnBuiltinProctime1(): Unit = { > val sqlQuery = > """ > |SELECT * > |FROM ( > | SELECT *, > |ROW_NUMBER() OVER (PARTITION BY a ORDER BY PROCTIME() ASC) as > rowNum > | FROM (select a, count(b) as b from MyTable group by a) > |) > |WHERE rowNum = 1 > """.stripMargin > util.verifyExecPlan(sqlQuery) > } {code} > Exception: > org.apache.flink.table.api.TableException: StreamPhysicalDeduplicate doesn't > support consuming update changes which is produced by node > GroupAggregate(groupBy=[a], select=[a, COUNT(b) AS b]) > because the StreamPhysicalDeduplicate can not consuming update changes now > while StreamExecRank can. > so we should not convert the FlinkLogicalRank to StreamPhysicalDeduplicate in > this case. and we can defer whether input contains update change in the > "optimize the physical plan" phase. > so we can add an option to solve it. and when the StreamPhysicalDeduplicate > can support consuming update changes , we can deprecate it -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
masteryhx commented on PR #24517: URL: https://github.com/apache/flink/pull/24517#issuecomment-2011189235 > @masteryhx I've tried manually regenerate the specified test `mvn clean package -Pgenerate-migration-test-data -Dgenerate.classes=StatefulJobSnapshotMigrationITCase -Dgenerate.version=1.19 -nsu -Dfast -DskipTests` > > a strange log says the test class does not exist: > > ``` > main: > [java] 743 [main] INFO org.apache.flink.test.migration.MigrationTestsSnapshotGenerator [] - Start test data generating for module /Users/lilin/work/git/flink/flink-tests and version 1.19 > [java] 749 [main] WARN org.apache.flink.test.migration.MigrationTestsSnapshotGenerator [] - Class StatefulJobSnapshotMigrationITCase does not exist. > [INFO] Executed tasks > [INFO] > [INFO] BUILD SUCCESS > [INFO] > [INFO] Total time: 01:42 min > [INFO] Finished at: 2024-03-20T23:35:58+08:00 > ``` > > Do you know what's wrong here? (If you know this, I haven't had time to dive into it yet) 1. The generate.classes should be configured as its complete path, e.g. org.apache.flink.test.checkpointing.StatefulJobSnapshotMigrationITCase 2. I just also tried to generate it but seems no result. I took a quick look and haven't found the reason. Maybe there are some issues for the generator logic. You could also debug this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34898) Cannot create named STRUCT with a single field
[ https://issues.apache.org/jira/browse/FLINK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829371#comment-17829371 ] Yubin Li commented on FLINK-34898: -- I use the latest version to run the syntax and work fine, maybe you could try 1.19. !image-2024-03-21-12-00-00-183.png|width=688,height=161! > Cannot create named STRUCT with a single field > -- > > Key: FLINK-34898 > URL: https://issues.apache.org/jira/browse/FLINK-34898 > Project: Flink > Issue Type: Bug >Reporter: Chloe He >Priority: Major > Attachments: image-2024-03-21-12-00-00-183.png > > > I'm trying to create named structs using Flink SQL and I found a previous > ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use > of the following syntax: > {code:java} > SELECT CAST(('a', 1) as ROW) AS row1; > {code} > However, my named struct has a single field and effectively it should look > something like `\{"a": 1}`. I can't seem to be able to find a way to > construct this. I have experimented with a few different syntax and it either > throws parsing error or casting error: > {code:java} > Cast function cannot convert value of type INTEGER to type > RecordType(VARCHAR(2147483647) a) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34900) Check compatibility for classes in flink-core-api that skip japicmp
[ https://issues.apache.org/jira/browse/FLINK-34900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-34900: --- Description: As we said in FLINK-34899, some classes are temporarily ignored by japicmp. Before the next release, we need to manually check if any of these classes have been changed in an incompatible way then. > Check compatibility for classes in flink-core-api that skip japicmp > --- > > Key: FLINK-34900 > URL: https://issues.apache.org/jira/browse/FLINK-34900 > Project: Flink > Issue Type: Sub-task > Components: API / Core >Reporter: Weijie Guo >Assignee: Weijie Guo >Priority: Major > > As we said in FLINK-34899, some classes are temporarily ignored by japicmp. > Before the next release, we need to manually check if any of these classes > have been changed in an incompatible way then. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34898) Cannot create named STRUCT with a single field
[ https://issues.apache.org/jira/browse/FLINK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yubin Li updated FLINK-34898: - Attachment: image-2024-03-21-12-00-00-183.png > Cannot create named STRUCT with a single field > -- > > Key: FLINK-34898 > URL: https://issues.apache.org/jira/browse/FLINK-34898 > Project: Flink > Issue Type: Bug >Reporter: Chloe He >Priority: Major > Attachments: image-2024-03-21-12-00-00-183.png > > > I'm trying to create named structs using Flink SQL and I found a previous > ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use > of the following syntax: > {code:java} > SELECT CAST(('a', 1) as ROW) AS row1; > {code} > However, my named struct has a single field and effectively it should look > something like `\{"a": 1}`. I can't seem to be able to find a way to > construct this. I have experimented with a few different syntax and it either > throws parsing error or casting error: > {code:java} > Cast function cannot convert value of type INTEGER to type > RecordType(VARCHAR(2147483647) a) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34899) Include all classes in flink-core-api that skip the japicmp check back
[ https://issues.apache.org/jira/browse/FLINK-34899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo reassigned FLINK-34899: -- Assignee: Weijie Guo > Include all classes in flink-core-api that skip the japicmp check back > -- > > Key: FLINK-34899 > URL: https://issues.apache.org/jira/browse/FLINK-34899 > Project: Flink > Issue Type: Sub-task > Components: API / Core >Reporter: Weijie Guo >Assignee: Weijie Guo >Priority: Major > > We have moved some classes from flink-core to flink-core-api without changing > the package name, but the japicmp check will also consider this incompatible > changes. As a result, we excluded them for now. But we'll have to include > them back in the next release. > The list of these classes: > org.apache.flink.api.common.functions.Function > org.apache.flink.api.java.functions.KeySelector -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34900) Check compatibility for classes in flink-core-api that skip japicmp
Weijie Guo created FLINK-34900: -- Summary: Check compatibility for classes in flink-core-api that skip japicmp Key: FLINK-34900 URL: https://issues.apache.org/jira/browse/FLINK-34900 Project: Flink Issue Type: Sub-task Components: API / Core Reporter: Weijie Guo Assignee: Weijie Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-26088) Add Elasticsearch 8.0 support
[ https://issues.apache.org/jira/browse/FLINK-26088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo closed FLINK-26088. -- Fix Version/s: elasticsearch-3.1.0 Resolution: Done main via 963c5058c9821682d76d9033a3417dfdb5f30ff7. > Add Elasticsearch 8.0 support > - > > Key: FLINK-26088 > URL: https://issues.apache.org/jira/browse/FLINK-26088 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Reporter: Yuhao Bi >Assignee: Matheus Felisberto >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: elasticsearch-3.1.0 > > > Since Elasticsearch 8.0 is officially released, I think it's time to consider > adding es8 connector support. > The High Level REST Client we used for connection [is marked deprecated in es > 7.15.0|https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high.html]. > Maybe we can migrate to use the new [Java API > Client|https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/8.0/index.html] > at this time. > Elasticsearch8.0 release note: > [https://www.elastic.co/guide/en/elasticsearch/reference/8.0/release-notes-8.0.0.html] > release highlights: > [https://www.elastic.co/guide/en/elasticsearch/reference/8.0/release-highlights.html] > REST API compatibility: > https://www.elastic.co/guide/en/elasticsearch/reference/8.0/rest-api-compatibility.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-26088][Connectors/ElasticSearch] Add Elasticsearch 8.0 support [flink-connector-elasticsearch]
boring-cyborg[bot] commented on PR #53: URL: https://github.com/apache/flink-connector-elasticsearch/pull/53#issuecomment-2011167239 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-26088][Connectors/ElasticSearch] Add Elasticsearch 8.0 support [flink-connector-elasticsearch]
reswqa merged PR #53: URL: https://github.com/apache/flink-connector-elasticsearch/pull/53 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34899) Include all classes in flink-core-api that skip the japicmp check back
[ https://issues.apache.org/jira/browse/FLINK-34899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-34899: --- Description: We have moved some classes from flink-core to flink-core-api without changing the package name, but the japicmp check will also consider this incompatible changes. As a result, we excluded them for now. But we'll have to include them back in the next release. The list of these classes: org.apache.flink.api.common.functions.Function org.apache.flink.api.java.functions.KeySelector was: We have moved some classes from flink-core to flink-core-api without changing the package name, but the japicmp check will also consider this incompatible changes. As a result, we excluded them for now. But we'll have to include them back in the next release. > Include all classes in flink-core-api that skip the japicmp check back > -- > > Key: FLINK-34899 > URL: https://issues.apache.org/jira/browse/FLINK-34899 > Project: Flink > Issue Type: Sub-task > Components: API / Core >Reporter: Weijie Guo >Priority: Major > > We have moved some classes from flink-core to flink-core-api without changing > the package name, but the japicmp check will also consider this incompatible > changes. As a result, we excluded them for now. But we'll have to include > them back in the next release. > The list of these classes: > org.apache.flink.api.common.functions.Function > org.apache.flink.api.java.functions.KeySelector -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34899) Include all classes in flink-core-api that skip the japicmp check back
[ https://issues.apache.org/jira/browse/FLINK-34899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-34899: --- Summary: Include all classes in flink-core-api that skip the japicmp check back (was: Remove all classes that skip the japicmp check for flink-core-api) > Include all classes in flink-core-api that skip the japicmp check back > -- > > Key: FLINK-34899 > URL: https://issues.apache.org/jira/browse/FLINK-34899 > Project: Flink > Issue Type: Sub-task > Components: API / Core >Reporter: Weijie Guo >Priority: Major > > We have moved some classes from flink-core to flink-core-api without changing > the package name, but the japicmp check will also consider this incompatible > changes. As a result, we excluded them for now. But we'll have to include > them back in the next release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34899) Remove all classes that skip the japicmp check for flink-core-api
[ https://issues.apache.org/jira/browse/FLINK-34899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-34899: --- Description: We have moved some classes from flink-core to flink-core-api without changing the package name, but the japicmp check will also consider this incompatible changes. As a result, we excluded them for now. But we'll have to include them back in the next release. was: We have moved some classes from flink-core to flink-core-api without changing the package name, but the japicmp check will also consider this incompatible changes. As a result, we excluded them for now. > Remove all classes that skip the japicmp check for flink-core-api > - > > Key: FLINK-34899 > URL: https://issues.apache.org/jira/browse/FLINK-34899 > Project: Flink > Issue Type: Sub-task > Components: API / Core >Reporter: Weijie Guo >Priority: Major > > We have moved some classes from flink-core to flink-core-api without changing > the package name, but the japicmp check will also consider this incompatible > changes. As a result, we excluded them for now. But we'll have to include > them back in the next release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34899) Remove all classes that skip the japicmp check for flink-core-api
[ https://issues.apache.org/jira/browse/FLINK-34899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-34899: --- Description: We have moved some classes from flink-core to flink-core-api without changing the package name, but the japicmp check will also consider this incompatible changes. As a result, we excluded them for now. > Remove all classes that skip the japicmp check for flink-core-api > - > > Key: FLINK-34899 > URL: https://issues.apache.org/jira/browse/FLINK-34899 > Project: Flink > Issue Type: Sub-task > Components: API / Core >Reporter: Weijie Guo >Priority: Major > > We have moved some classes from flink-core to flink-core-api without changing > the package name, but the japicmp check will also consider this incompatible > changes. As a result, we excluded them for now. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34899) Remove all classes that skip the japicmp check for flink-core-api
Weijie Guo created FLINK-34899: -- Summary: Remove all classes that skip the japicmp check for flink-core-api Key: FLINK-34899 URL: https://issues.apache.org/jira/browse/FLINK-34899 Project: Flink Issue Type: Sub-task Components: API / Core Reporter: Weijie Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34668][checkpoint] Report state handle of file merging directory to JM [flink]
Zakelly commented on code in PR #24513: URL: https://github.com/apache/flink/pull/24513#discussion_r1533201932 ## flink-runtime/src/main/java/org/apache/flink/runtime/state/filemerging/SegmentFileStateHandle.java: ## @@ -62,19 +77,58 @@ public class SegmentFileStateHandle implements StreamStateHandle { * @param scope The state's scope, whether it is exclusive or shared. */ public SegmentFileStateHandle( -Path filePath, long startPos, long stateSize, CheckpointedStateScope scope) { +Path directoryPath, +Path filePath, +long startPos, +long stateSize, +CheckpointedStateScope scope) { this.filePath = filePath; this.stateSize = stateSize; this.startPos = startPos; this.scope = scope; +this.directoryStateHandle = +DirectoryStreamStateHandle.forPathWithSize( +new File(directoryPath.getPath()).toPath()); Review Comment: Will `forPathWithSize` be inefficiency? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34689][MySQL][Feature] check binlog_row_value_optoins [flink-cdc]
SML0127 commented on PR #3148: URL: https://github.com/apache/flink-cdc/pull/3148#issuecomment-2011101317 @ruanhang1993 PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] (FLINK-34898) Cannot create named STRUCT with a single field
[ https://issues.apache.org/jira/browse/FLINK-34898 ] Chloe He deleted comment on FLINK-34898: -- was (Author: JIRAUSER304541): I also can't wrap this in an ARRAY like so {code:java} SELECT * FROM (VALUES ARRAY[CAST((1, 2) as ROW)] ); {code} This raises {code:java} java.lang.UnsupportedOperationException: class org.apache.calcite.sql.SqlBasicCall: ROW('a', 1) {code} > Cannot create named STRUCT with a single field > -- > > Key: FLINK-34898 > URL: https://issues.apache.org/jira/browse/FLINK-34898 > Project: Flink > Issue Type: Bug >Reporter: Chloe He >Priority: Major > > I'm trying to create named structs using Flink SQL and I found a previous > ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use > of the following syntax: > {code:java} > SELECT CAST(('a', 1) as ROW) AS row1; > {code} > However, my named struct has a single field and effectively it should look > something like `\{"a": 1}`. I can't seem to be able to find a way to > construct this. I have experimented with a few different syntax and it either > throws parsing error or casting error: > {code:java} > Cast function cannot convert value of type INTEGER to type > RecordType(VARCHAR(2147483647) a) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34898) Cannot create named STRUCT with a single field
[ https://issues.apache.org/jira/browse/FLINK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829349#comment-17829349 ] Chloe He commented on FLINK-34898: -- I also can't wrap this in an ARRAY like so {code:java} SELECT * FROM (VALUES ARRAY[CAST((1, 2) as ROW)] ); {code} This raises {code:java} java.lang.UnsupportedOperationException: class org.apache.calcite.sql.SqlBasicCall: ROW('a', 1) {code} > Cannot create named STRUCT with a single field > -- > > Key: FLINK-34898 > URL: https://issues.apache.org/jira/browse/FLINK-34898 > Project: Flink > Issue Type: Bug >Reporter: Chloe He >Priority: Major > > I'm trying to create named structs using Flink SQL and I found a previous > ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use > of the following syntax: > {code:java} > SELECT CAST(('a', 1) as ROW) AS row1; > {code} > However, my named struct has a single field and effectively it should look > something like `\{"a": 1}`. I can't seem to be able to find a way to > construct this. I have experimented with a few different syntax and it either > throws parsing error or casting error: > {code:java} > Cast function cannot convert value of type INTEGER to type > RecordType(VARCHAR(2147483647) a) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34526][runtime] Actively disconnect the TM in RM to reduce restart time [flink]
qinf commented on PR #24539: URL: https://github.com/apache/flink/pull/24539#issuecomment-2011045063 @1996fanrui @RocMarshal Could you help review this PR in your free time? Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Flink 34770 [flink-cdc]
lvyanquan closed pull request #3183: Flink 34770 URL: https://github.com/apache/flink-cdc/pull/3183 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34898) Cannot create named STRUCT with a single field
[ https://issues.apache.org/jira/browse/FLINK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chloe He updated FLINK-34898: - Description: I'm trying to create named structs using Flink SQL and I found a previous ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use of the following syntax: {code:java} SELECT CAST(('a', 1) as ROW) AS row1; {code} However, my named struct has a single field and effectively it should look something like `\{"a": 1}`. I can't seem to be able to find a way to construct this. I have experimented with a few different syntax and it either throws parsing error or casting error: {code:java} Cast function cannot convert value of type INTEGER to type RecordType(VARCHAR(2147483647) a) {code} was: I'm trying to create named structs using Flink SQL and I found a previous ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use of the following syntax: ```sql SELECT CAST(('a', 1) as ROW) AS row1; ``` However, my named struct has a single field and effectively it should look something like `\{"a": 1}`. I can't seem to be able to find a way to construct this. I have experimented with a few different syntax and it either throws parsing error or casting error: ``` Cast function cannot convert value of type INTEGER to type RecordType(VARCHAR(2147483647) a) ``` > Cannot create named STRUCT with a single field > -- > > Key: FLINK-34898 > URL: https://issues.apache.org/jira/browse/FLINK-34898 > Project: Flink > Issue Type: Bug >Reporter: Chloe He >Priority: Major > > I'm trying to create named structs using Flink SQL and I found a previous > ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use > of the following syntax: > > {code:java} > SELECT CAST(('a', 1) as ROW) AS row1; > {code} > However, my named struct has a single field and effectively it should look > something like `\{"a": 1}`. I can't seem to be able to find a way to > construct this. I have experimented with a few different syntax and it either > throws parsing error or casting error: > {code:java} > Cast function cannot convert value of type INTEGER to type > RecordType(VARCHAR(2147483647) a) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34898) Cannot create named STRUCT with a single field
Chloe He created FLINK-34898: Summary: Cannot create named STRUCT with a single field Key: FLINK-34898 URL: https://issues.apache.org/jira/browse/FLINK-34898 Project: Flink Issue Type: Bug Reporter: Chloe He I'm trying to create named structs using Flink SQL and I found a previous ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use of the following syntax: ```sql SELECT CAST(('a', 1) as ROW) AS row1; ``` However, my named struct has a single field and effectively it should look something like `\{"a": 1}`. I can't seem to be able to find a way to construct this. I have experimented with a few different syntax and it either throws parsing error or casting error: ``` Cast function cannot convert value of type INTEGER to type RecordType(VARCHAR(2147483647) a) ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34898) Cannot create named STRUCT with a single field
[ https://issues.apache.org/jira/browse/FLINK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chloe He updated FLINK-34898: - Description: I'm trying to create named structs using Flink SQL and I found a previous ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use of the following syntax: {code:java} SELECT CAST(('a', 1) as ROW) AS row1; {code} However, my named struct has a single field and effectively it should look something like `\{"a": 1}`. I can't seem to be able to find a way to construct this. I have experimented with a few different syntax and it either throws parsing error or casting error: {code:java} Cast function cannot convert value of type INTEGER to type RecordType(VARCHAR(2147483647) a) {code} was: I'm trying to create named structs using Flink SQL and I found a previous ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use of the following syntax: {code:java} SELECT CAST(('a', 1) as ROW) AS row1; {code} However, my named struct has a single field and effectively it should look something like `\{"a": 1}`. I can't seem to be able to find a way to construct this. I have experimented with a few different syntax and it either throws parsing error or casting error: {code:java} Cast function cannot convert value of type INTEGER to type RecordType(VARCHAR(2147483647) a) {code} > Cannot create named STRUCT with a single field > -- > > Key: FLINK-34898 > URL: https://issues.apache.org/jira/browse/FLINK-34898 > Project: Flink > Issue Type: Bug >Reporter: Chloe He >Priority: Major > > I'm trying to create named structs using Flink SQL and I found a previous > ticket https://issues.apache.org/jira/browse/FLINK-9161 that mentions the use > of the following syntax: > {code:java} > SELECT CAST(('a', 1) as ROW) AS row1; > {code} > However, my named struct has a single field and effectively it should look > something like `\{"a": 1}`. I can't seem to be able to find a way to > construct this. I have experimented with a few different syntax and it either > throws parsing error or casting error: > {code:java} > Cast function cannot convert value of type INTEGER to type > RecordType(VARCHAR(2147483647) a) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-31664][table]Add ARRAY_INTERSECT function [flink]
liuyongvs commented on PR #23171: URL: https://github.com/apache/flink/pull/23171#issuecomment-2011038228 hi @dawidwys will you help review this https://github.com/apache/flink/pull/24526 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34732][cdc][ci] Add document dead link check for Flink CDC Documentation [flink-cdc]
GOODBOY008 commented on PR #3168: URL: https://github.com/apache/flink-cdc/pull/3168#issuecomment-2011028273 @PatrickRen Rebased to master , PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33220) PyFlink support for Datagen connector
[ https://issues.apache.org/jira/browse/FLINK-33220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829346#comment-17829346 ] Dian Fu commented on FLINK-33220: - [~liu.chong] I missed this ticket. Feel free to submit the PR and ping me~ > PyFlink support for Datagen connector > - > > Key: FLINK-33220 > URL: https://issues.apache.org/jira/browse/FLINK-33220 > Project: Flink > Issue Type: New Feature > Components: API / Python >Reporter: Liu Chong >Priority: Minor > > This is a simple Jira to propose the support of Datagen in PyFlink datastream > API as a built-in source connector -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33220) PyFlink support for Datagen connector
[ https://issues.apache.org/jira/browse/FLINK-33220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829346#comment-17829346 ] Dian Fu edited comment on FLINK-33220 at 3/21/24 1:24 AM: -- [~liu.chong] Sorry, I missed this ticket. Feel free to submit the PR and ping me~ was (Author: dianfu): [~liu.chong] I missed this ticket. Feel free to submit the PR and ping me~ > PyFlink support for Datagen connector > - > > Key: FLINK-33220 > URL: https://issues.apache.org/jira/browse/FLINK-33220 > Project: Flink > Issue Type: New Feature > Components: API / Python >Reporter: Liu Chong >Priority: Minor > > This is a simple Jira to propose the support of Datagen in PyFlink datastream > API as a built-in source connector -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [Improve][pipeline]Support set checkpoint. [flink-cdc]
lightzhao commented on PR #3101: URL: https://github.com/apache/flink-cdc/pull/3101#issuecomment-2011007376 @leonardBang PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Bug][pipeline]Set default values for the parallelism. [flink-cdc]
lightzhao commented on PR #3100: URL: https://github.com/apache/flink-cdc/pull/3100#issuecomment-2011007083 @leonardBang PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-34701) Publish the Dockerfiles for the new release
[ https://issues.apache.org/jira/browse/FLINK-34701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee closed FLINK-34701. --- Resolution: Fixed > Publish the Dockerfiles for the new release > --- > > Key: FLINK-34701 > URL: https://issues.apache.org/jira/browse/FLINK-34701 > Project: Flink > Issue Type: Sub-task >Reporter: lincoln lee >Assignee: lincoln lee >Priority: Major > Labels: pull-request-available > > Note: the official Dockerfiles fetch the binary distribution of the target > Flink version from an Apache mirror. After publishing the binary release > artifacts, mirrors can take some hours to start serving the new artifacts, so > you may want to wait to do this step until you are ready to continue with the > "Promote the release" steps in the follow-up Jira. > Follow the [release instructions in the flink-docker > repo|https://github.com/apache/flink-docker#release-workflow] to build the > new Dockerfiles and send an updated manifest to Docker Hub so the new images > are built and published. > > > h3. Expectations > * Dockerfiles in [flink-docker|https://github.com/apache/flink-docker] > updated for the new Flink release and pull request opened on the Docker > official-images with an updated manifest -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34701) Publish the Dockerfiles for the new release
[ https://issues.apache.org/jira/browse/FLINK-34701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829340#comment-17829340 ] lincoln lee commented on FLINK-34701: - merged in [{{official-images}}|https://github.com/docker-library/official-images] master: fe836e1e82ee9a807d9e9825d98a9a47b8c5 > Publish the Dockerfiles for the new release > --- > > Key: FLINK-34701 > URL: https://issues.apache.org/jira/browse/FLINK-34701 > Project: Flink > Issue Type: Sub-task >Reporter: lincoln lee >Assignee: lincoln lee >Priority: Major > Labels: pull-request-available > > Note: the official Dockerfiles fetch the binary distribution of the target > Flink version from an Apache mirror. After publishing the binary release > artifacts, mirrors can take some hours to start serving the new artifacts, so > you may want to wait to do this step until you are ready to continue with the > "Promote the release" steps in the follow-up Jira. > Follow the [release instructions in the flink-docker > repo|https://github.com/apache/flink-docker#release-workflow] to build the > new Dockerfiles and send an updated manifest to Docker Hub so the new images > are built and published. > > > h3. Expectations > * Dockerfiles in [flink-docker|https://github.com/apache/flink-docker] > updated for the new Flink release and pull request opened on the Docker > official-images with an updated manifest -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34892][ci] Fix python test failure due to config file change [flink-connector-aws]
dannycranmer commented on PR #133: URL: https://github.com/apache/flink-connector-aws/pull/133#issuecomment-2010729129 Thanks @z3d1k. Waiting for actions to complete before merge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32706][table] Add built-in SPLIT_STRING function [flink]
hanyuzheng7 commented on code in PR #24365: URL: https://github.com/apache/flink/pull/24365#discussion_r1532815010 ## flink-python/pyflink/table/expression.py: ## @@ -1609,6 +1609,17 @@ def array_min(self) -> 'Expression': """ return _unary_op("arrayMin")(self) +def split(self, delimiter) -> 'Expression': Review Comment: I just support Java one, so I think we don't need add python test(s) for `split`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-27146] [Filesystem] Migrate to Junit5 [flink]
kottmann commented on PR #22789: URL: https://github.com/apache/flink/pull/22789#issuecomment-2010479115 @ferenc-csaky Thank you for your insightful feedback and for directing me towards the assertj decision. I updated my branch by rebasing it onto master and incorporating the changes related to assertj. Additionally, I will make another commit to address the remaining feedback that isn't connected to assertj. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-26570][statefun] Remote module configuration interpolation [flink-statefun]
nicklester commented on PR #309: URL: https://github.com/apache/flink-statefun/pull/309#issuecomment-2010471575 I still see value in this PR. When deploying into a managed environment such as Kinesis Data Analytics on AWS it is not possible to use something like 'jar -uf' to modify the jar at deploy time. We see value in being able to pass a hostname for API Gateway as an environment variable, for instance, to simplify multi environment deployment in an AWS CDK stack. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34696) GSRecoverableWriterCommitter is generating excessive data blobs
[ https://issues.apache.org/jira/browse/FLINK-34696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829246#comment-17829246 ] Galen Warren commented on FLINK-34696: -- I'm not aware of any special considerations in RecoverableWriter for batch mode, but to be honest I'm not very knowledgeable about batch mode in general. Maybe someone else knows more about that ... {quote}The code above also avoids inserting the intermediate compose blob identifier at index 0. That looks expensive if the list is very big (and as far as I see not a LinkedList) {quote} I don't think it's inserting anything, really, as the blobs are immutable. I think that just controls the order of the bytes in the newly created blob. > GSRecoverableWriterCommitter is generating excessive data blobs > --- > > Key: FLINK-34696 > URL: https://issues.apache.org/jira/browse/FLINK-34696 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Simon-Shlomo Poil >Priority: Major > > The `composeBlobs` method in > `org.apache.flink.fs.gs.writer.GSRecoverableWriterCommitter` is designed to > merge multiple small blobs into a single large blob using Google Cloud > Storage's compose method. This process is iterative, combining the result > from the previous iteration with 31 new blobs until all blobs are merged. > Upon completion of the composition, the method proceeds to remove the > temporary blobs. > *Issue:* > This methodology results in significant, unnecessary data storage consumption > during the blob composition process, incurring considerable costs due to > Google Cloud Storage pricing models. > *Example to Illustrate the Problem:* > - Initial state: 64 blobs, each 1 GB in size (totaling 64 GB). > - After 1st step: 32 blobs are merged into a single blob, increasing total > storage to 96 GB (64 original + 32 GB new). > - After 2nd step: The newly created 32 GB blob is merged with 31 more blobs, > raising the total to 159 GB. > - After 3rd step: The final blob is merged, culminating in a total of 223 GB > to combine the original 64 GB of data. This results in an overhead of 159 GB. > *Impact:* > This inefficiency has a profound impact, especially at scale, where terabytes > of data can incur overheads in the petabyte range, leading to unexpectedly > high costs. Additionally, we have observed an increase in storage exceptions > thrown by the Google Storage library, potentially linked to this issue. > *Suggested Solution:* > To mitigate this problem, we propose modifying the `composeBlobs` method to > immediately delete source blobs once they have been successfully combined. > This change could significantly reduce data duplication and associated costs. > However, the implications for data recovery and integrity need careful > consideration to ensure that this optimization does not compromise the > ability to recover data in case of a failure during the composition process. > *Steps to Reproduce:* > 1. Initiate the blob composition process in an environment with a significant > number of blobs (e.g., 64 blobs of 1 GB each). > 2. Observe the temporary increase in data storage as blobs are iteratively > combined. > 3. Note the final amount of data storage used compared to the initial total > size of the blobs. > *Expected Behavior:* > The blob composition process should minimize unnecessary data storage use, > efficiently managing resources to combine blobs without generating excessive > temporary data overhead. > *Actual Behavior:* > The current implementation results in significant temporary increases in data > storage, leading to high costs and potential system instability due to > frequent storage exceptions. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34706) Promote release 1.19
[ https://issues.apache.org/jira/browse/FLINK-34706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829244#comment-17829244 ] lincoln lee commented on FLINK-34706: - # (/) Website pull request to [list the release|http://flink.apache.org/downloads.html] merged # (/) Release announced on the user@ mailing list: [[announcement link|https://lists.apache.org/thread/sofmxytbh6y20nwot1gywqqc2lqxn4hm]|https://lists.apache.org/thread/72nmfwsgs7sqkw7mykz4h36hgb7wo04d] # (/) Blog post published, if applicable:[ blog post|https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/] # (/) Release recorded in [reporter.apache.org: https://reporter.apache.org/addrelease.html?flink|https://reporter.apache.org/addrelease.html?flink] # (/) Release announced on social media: [Twitter|https://twitter.com/ApacheFlink/status/1638839542403981312?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1638839542403981312%7Ctwgr%5E7f3046f67668cf3ebbd929ef126a32473db2a1b5%7Ctwcon%5Es1_c10&ref_url=https%3A%2F%2Fpublish.twitter.com%2F%3Fquery%3Dhttps3A2F2Ftwitter.com2FApacheFlink2Fstatus2F1638839542403981312widget%3DTweet] # (/) Completion declared on the dev@ [mailing list |https://lists.apache.org/thread/z8sfwlppsodcyng62c584n76b69b16fc] # (/) Update Homebrew: [https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request] (seems to be done automatically - at least for minor releases for both minor and major releases): [https://formulae.brew.sh/formula/apache-flink#default] # (/) No need to update quickstart scripts in {{{}flink-web{}}}, under the {{q/}} directory (alread use global version variables) # Updated the japicmp configuration: Done in https://issues.apache.org/jira/browse/FLINK-34707 # Update the list of previous version in {{docs/config.toml}} on the master branch: Done in # Set {{show_outdated_warning: true}} in {{docs/config.toml}} in the branch of the _previous_ Flink version: (for 1.17) # (/) Update stable and master alias in [https://github.com/apache/flink/blob/master/.github/workflows/docs.yml] Done in [a6a4667|https://github.com/apache/flink/commit/a6a4667202a0f89fe63ff4f2e476c0200ec66e63] > Promote release 1.19 > > > Key: FLINK-34706 > URL: https://issues.apache.org/jira/browse/FLINK-34706 > Project: Flink > Issue Type: New Feature >Affects Versions: 1.18.0 >Reporter: Lincoln Lee >Assignee: lincoln lee >Priority: Major > Labels: pull-request-available > > Once the release has been finalized (FLINK-32920), the last step of the > process is to promote the release within the project and beyond. Please wait > for 24h after finalizing the release in accordance with the [ASF release > policy|http://www.apache.org/legal/release-policy.html#release-announcements]. > *Final checklist to declare this issue resolved:* > # Website pull request to [list the > release|http://flink.apache.org/downloads.html] merged > # Release announced on the user@ mailing list. > # Blog post published, if applicable. > # Release recorded in > [reporter.apache.org|https://reporter.apache.org/addrelease.html?flink]. > # Release announced on social media. > # Completion declared on the dev@ mailing list. > # Update Homebrew: > [https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request] (seems to be done > automatically - at least for minor releases for both minor and major > releases) > # Updated the japicmp configuration > ** corresponding SNAPSHOT branch japicmp reference version set to the just > released version, and API compatibiltity checks for {{@PublicEvolving}} was > enabled > ** (minor version release only) master branch japicmp reference version set > to the just released version > ** (minor version release only) master branch japicmp exclusions have been > cleared > # Update the list of previous version in {{docs/config.toml}} on the master > branch. > # Set {{show_outdated_warning: true}} in {{docs/config.toml}} in the branch > of the _now deprecated_ Flink version (i.e. 1.16 if 1.18.0 is released) > # Update stable and master alias in > [https://github.com/apache/flink/blob/master/.github/workflows/docs.yml] > # Open discussion thread for End of Life for Unsupported version (i.e. 1.16) -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34706][docs] Add 1.19 to PreviousDocs list. [flink]
flinkbot commented on PR #24548: URL: https://github.com/apache/flink/pull/24548#issuecomment-2010028945 ## CI report: * d73fe76e42235b2ddc1937b70c1f1b4eaf5aa38c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34706][docs] Deprecates 1.17 docs. [flink]
flinkbot commented on PR #24547: URL: https://github.com/apache/flink/pull/24547#issuecomment-2010015809 ## CI report: * aea991f5e66edb1f26c7ef16e71c564083b58a73 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-34696) GSRecoverableWriterCommitter is generating excessive data blobs
[ https://issues.apache.org/jira/browse/FLINK-34696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829228#comment-17829228 ] Simon-Shlomo Poil edited comment on FLINK-34696 at 3/20/24 4:22 PM: I checked using the CLI gcloud storage objects compose blobA blobB blobA that worked. But yes indeed the documentation is a bit unclear. How the actual compose works on the GCS side is a good question. If I understand correctly - this committer is called regardless of the execution mode: ie. BATCH / STREAMING? In BATCH mode checkpointing is not supported, so the code should not depend on frequent checkpointing to work efficiently. The code above also avoids inserting the intermediate compose blob identifier at index 0. That looks expensive if the list is very big (and as far as I see not a LinkedList) was (Author: sisp): I checked using the CLI gcloud storage objects compose blobA blobB blobA that worked. But yes indeed the documentation is a bit unclear. How the actual compose works on the GCS side is a good question. If I understand correctly - this committer is called regardless of the execution mode: ie. BATCH / STREAMING? In BATCH mode checkpointing is not supported, so the code should depend on frequent checkpointing to work efficiently. The code above also avoids inserting the intermediate compose blob identifier at index 0. That looks expensive if the list is very big (and as far as I see not a LinkedList) > GSRecoverableWriterCommitter is generating excessive data blobs > --- > > Key: FLINK-34696 > URL: https://issues.apache.org/jira/browse/FLINK-34696 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Simon-Shlomo Poil >Priority: Major > > The `composeBlobs` method in > `org.apache.flink.fs.gs.writer.GSRecoverableWriterCommitter` is designed to > merge multiple small blobs into a single large blob using Google Cloud > Storage's compose method. This process is iterative, combining the result > from the previous iteration with 31 new blobs until all blobs are merged. > Upon completion of the composition, the method proceeds to remove the > temporary blobs. > *Issue:* > This methodology results in significant, unnecessary data storage consumption > during the blob composition process, incurring considerable costs due to > Google Cloud Storage pricing models. > *Example to Illustrate the Problem:* > - Initial state: 64 blobs, each 1 GB in size (totaling 64 GB). > - After 1st step: 32 blobs are merged into a single blob, increasing total > storage to 96 GB (64 original + 32 GB new). > - After 2nd step: The newly created 32 GB blob is merged with 31 more blobs, > raising the total to 159 GB. > - After 3rd step: The final blob is merged, culminating in a total of 223 GB > to combine the original 64 GB of data. This results in an overhead of 159 GB. > *Impact:* > This inefficiency has a profound impact, especially at scale, where terabytes > of data can incur overheads in the petabyte range, leading to unexpectedly > high costs. Additionally, we have observed an increase in storage exceptions > thrown by the Google Storage library, potentially linked to this issue. > *Suggested Solution:* > To mitigate this problem, we propose modifying the `composeBlobs` method to > immediately delete source blobs once they have been successfully combined. > This change could significantly reduce data duplication and associated costs. > However, the implications for data recovery and integrity need careful > consideration to ensure that this optimization does not compromise the > ability to recover data in case of a failure during the composition process. > *Steps to Reproduce:* > 1. Initiate the blob composition process in an environment with a significant > number of blobs (e.g., 64 blobs of 1 GB each). > 2. Observe the temporary increase in data storage as blobs are iteratively > combined. > 3. Note the final amount of data storage used compared to the initial total > size of the blobs. > *Expected Behavior:* > The blob composition process should minimize unnecessary data storage use, > efficiently managing resources to combine blobs without generating excessive > temporary data overhead. > *Actual Behavior:* > The current implementation results in significant temporary increases in data > storage, leading to high costs and potential system instability due to > frequent storage exceptions. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34696) GSRecoverableWriterCommitter is generating excessive data blobs
[ https://issues.apache.org/jira/browse/FLINK-34696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829228#comment-17829228 ] Simon-Shlomo Poil commented on FLINK-34696: --- I checked using the CLI gcloud storage objects compose blobA blobB blobA that worked. But yes indeed the documentation is a bit unclear. How the actual compose works on the GCS side is a good question. If I understand correctly - this committer is called regardless of the execution mode: ie. BATCH / STREAMING? In BATCH mode checkpointing is not supported, so the code should depend on frequent checkpointing to work efficiently. The code above also avoids inserting the intermediate compose blob identifier at index 0. That looks expensive if the list is very big (and as far as I see not a LinkedList) > GSRecoverableWriterCommitter is generating excessive data blobs > --- > > Key: FLINK-34696 > URL: https://issues.apache.org/jira/browse/FLINK-34696 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Simon-Shlomo Poil >Priority: Major > > The `composeBlobs` method in > `org.apache.flink.fs.gs.writer.GSRecoverableWriterCommitter` is designed to > merge multiple small blobs into a single large blob using Google Cloud > Storage's compose method. This process is iterative, combining the result > from the previous iteration with 31 new blobs until all blobs are merged. > Upon completion of the composition, the method proceeds to remove the > temporary blobs. > *Issue:* > This methodology results in significant, unnecessary data storage consumption > during the blob composition process, incurring considerable costs due to > Google Cloud Storage pricing models. > *Example to Illustrate the Problem:* > - Initial state: 64 blobs, each 1 GB in size (totaling 64 GB). > - After 1st step: 32 blobs are merged into a single blob, increasing total > storage to 96 GB (64 original + 32 GB new). > - After 2nd step: The newly created 32 GB blob is merged with 31 more blobs, > raising the total to 159 GB. > - After 3rd step: The final blob is merged, culminating in a total of 223 GB > to combine the original 64 GB of data. This results in an overhead of 159 GB. > *Impact:* > This inefficiency has a profound impact, especially at scale, where terabytes > of data can incur overheads in the petabyte range, leading to unexpectedly > high costs. Additionally, we have observed an increase in storage exceptions > thrown by the Google Storage library, potentially linked to this issue. > *Suggested Solution:* > To mitigate this problem, we propose modifying the `composeBlobs` method to > immediately delete source blobs once they have been successfully combined. > This change could significantly reduce data duplication and associated costs. > However, the implications for data recovery and integrity need careful > consideration to ensure that this optimization does not compromise the > ability to recover data in case of a failure during the composition process. > *Steps to Reproduce:* > 1. Initiate the blob composition process in an environment with a significant > number of blobs (e.g., 64 blobs of 1 GB each). > 2. Observe the temporary increase in data storage as blobs are iteratively > combined. > 3. Note the final amount of data storage used compared to the initial total > size of the blobs. > *Expected Behavior:* > The blob composition process should minimize unnecessary data storage use, > efficiently managing resources to combine blobs without generating excessive > temporary data overhead. > *Actual Behavior:* > The current implementation results in significant temporary increases in data > storage, leading to high costs and potential system instability due to > frequent storage exceptions. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-34713) Updates the docs stable version
[ https://issues.apache.org/jira/browse/FLINK-34713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee closed FLINK-34713. --- Resolution: Fixed Already fixed by 3d67a2b9b208ed662f81c1ef43d76d10a82a7d88 on release-1.19 branch > Updates the docs stable version > --- > > Key: FLINK-34713 > URL: https://issues.apache.org/jira/browse/FLINK-34713 > Project: Flink > Issue Type: Sub-task >Reporter: Lincoln Lee >Assignee: lincoln lee >Priority: Major > > Update docs to "stable" in {{docs/config.toml}} in the branch of the > _just-released_ version: > * Change V{{{}ersion{}}} from {{{}x.y-SNAPSHOT }}to \{{{}x.y.z{}}}, i.e. > {{1.6-SNAPSHOT}} to {{1.6.0}} > * Change V{{{}ersionTitle{}}} from {{x.y-SNAPSHOT}} to {{{}x.y{}}}, i.e. > {{1.6-SNAPSHOT}} to {{1.6}} > * Change Branch from {{master}} to {{{}release-x.y{}}}, i.e. {{master}} to > {{release-1.6}} > * Change {{baseURL}} from > {{//[ci.apache.org/projects/flink/flink-docs-master|http://ci.apache.org/projects/flink/flink-docs-master]}} > to > {{//[ci.apache.org/projects/flink/flink-docs-release-x.y|http://ci.apache.org/projects/flink/flink-docs-release-x.y]}} > * Change {{javadocs_baseurl}} from > {{//[ci.apache.org/projects/flink/flink-docs-master|http://ci.apache.org/projects/flink/flink-docs-master]}} > to > {{//[ci.apache.org/projects/flink/flink-docs-release-x.y|http://ci.apache.org/projects/flink/flink-docs-release-x.y]}} > * Change {{IsStable}} to {{true}} -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
lincoln-lil commented on PR #24517: URL: https://github.com/apache/flink/pull/24517#issuecomment-2009918611 @masteryhx I've tried manually regenerate the specified test `mvn clean package -Pgenerate-migration-test-data -Dgenerate.classes=StatefulJobSnapshotMigrationITCase -Dgenerate.version=1.19 -nsu -Dfast -DskipTests` a strange log says the test class does not exist: ``` main: [java] 743 [main] INFO org.apache.flink.test.migration.MigrationTestsSnapshotGenerator [] - Start test data generating for module /Users/lilin/work/git/flink/flink-tests and version 1.19 [java] 749 [main] WARN org.apache.flink.test.migration.MigrationTestsSnapshotGenerator [] - Class StatefulJobSnapshotMigrationITCase does not exist. [INFO] Executed tasks [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 01:42 min [INFO] Finished at: 2024-03-20T23:35:58+08:00 ``` Do you know what's wrong here? (If you know this, I haven't had time to dive into it yet) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34893] Bump checkstyle to 9.3 [flink]
snuyanzin commented on code in PR #24540: URL: https://github.com/apache/flink/pull/24540#discussion_r1532320054 ## tools/maven/checkstyle.xml: ## @@ -238,16 +238,9 @@ This file is based on the checkstyle file of Apache Beam. Review Comment: Thanks, there is another link in the same file, I updated it as well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34897][test] Enables JobMasterServiceLeadershipRunnerTest#testJobMasterServiceLeadershipRunnerCloseWhenElectionServiceGrantLeaderShip again. [flink]
flinkbot commented on PR #24546: URL: https://github.com/apache/flink/pull/24546#issuecomment-2009806089 ## CI report: * 234bd00f873cb44f04131539ecae98ef7d2d9b9d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34897) JobMasterServiceLeadershipRunnerTest#testJobMasterServiceLeadershipRunnerCloseWhenElectionServiceGrantLeaderShip needs to be enabled again
[ https://issues.apache.org/jira/browse/FLINK-34897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34897: --- Labels: pull-request-available (was: ) > JobMasterServiceLeadershipRunnerTest#testJobMasterServiceLeadershipRunnerCloseWhenElectionServiceGrantLeaderShip > needs to be enabled again > -- > > Key: FLINK-34897 > URL: https://issues.apache.org/jira/browse/FLINK-34897 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Affects Versions: 1.17.2, 1.19.0, 1.18.1, 1.20.0 >Reporter: Matthias Pohl >Assignee: Matthias Pohl >Priority: Major > Labels: pull-request-available > > While working on FLINK-34672 I noticed that > {{JobMasterServiceLeadershipRunnerTest#testJobMasterServiceLeadershipRunnerCloseWhenElectionServiceGrantLeaderShip}} > is disabled without a reason. > It looks like I disabled it accidentally as part of FLINK-31783. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-34897][test] Enables JobMasterServiceLeadershipRunnerTest#test… [flink]
XComp opened a new pull request, #24546: URL: https://github.com/apache/flink/pull/24546 … ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-34897) JobMasterServiceLeadershipRunnerTest#testJobMasterServiceLeadershipRunnerCloseWhenElectionServiceGrantLeaderShip needs to be enabled again
Matthias Pohl created FLINK-34897: - Summary: JobMasterServiceLeadershipRunnerTest#testJobMasterServiceLeadershipRunnerCloseWhenElectionServiceGrantLeaderShip needs to be enabled again Key: FLINK-34897 URL: https://issues.apache.org/jira/browse/FLINK-34897 Project: Flink Issue Type: Technical Debt Components: Runtime / Coordination Affects Versions: 1.18.1, 1.19.0, 1.17.2, 1.20.0 Reporter: Matthias Pohl While working on FLINK-34672 I noticed that {{JobMasterServiceLeadershipRunnerTest#testJobMasterServiceLeadershipRunnerCloseWhenElectionServiceGrantLeaderShip}} is disabled without a reason. It looks like I disabled it accidentally as part of FLINK-31783. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34897) JobMasterServiceLeadershipRunnerTest#testJobMasterServiceLeadershipRunnerCloseWhenElectionServiceGrantLeaderShip needs to be enabled again
[ https://issues.apache.org/jira/browse/FLINK-34897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl reassigned FLINK-34897: - Assignee: Matthias Pohl > JobMasterServiceLeadershipRunnerTest#testJobMasterServiceLeadershipRunnerCloseWhenElectionServiceGrantLeaderShip > needs to be enabled again > -- > > Key: FLINK-34897 > URL: https://issues.apache.org/jira/browse/FLINK-34897 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Affects Versions: 1.17.2, 1.19.0, 1.18.1, 1.20.0 >Reporter: Matthias Pohl >Assignee: Matthias Pohl >Priority: Major > > While working on FLINK-34672 I noticed that > {{JobMasterServiceLeadershipRunnerTest#testJobMasterServiceLeadershipRunnerCloseWhenElectionServiceGrantLeaderShip}} > is disabled without a reason. > It looks like I disabled it accidentally as part of FLINK-31783. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34893] Bump checkstyle to 9.3 [flink]
RyanSkraba commented on code in PR #24540: URL: https://github.com/apache/flink/pull/24540#discussion_r1532074768 ## tools/maven/checkstyle.xml: ## @@ -238,16 +238,9 @@ This file is based on the checkstyle file of Apache Beam. - Review Comment: ```suggestion ``` The `scope` property was renamed, but it still exists. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34893] Bump checkstyle to 9.3 [flink]
RyanSkraba commented on code in PR #24540: URL: https://github.com/apache/flink/pull/24540#discussion_r1532031293 ## tools/maven/checkstyle.xml: ## @@ -238,16 +238,9 @@ This file is based on the checkstyle file of Apache Beam. Review Comment: While you're here! ```suggestion ``` ## tools/maven/checkstyle.xml: ## @@ -238,16 +238,9 @@ This file is based on the checkstyle file of Apache Beam. - Review Comment: ```suggestion ``` The `scope` property was renamed, but it still exists. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-34745) Parsing temporal table join throws cryptic exceptions
[ https://issues.apache.org/jira/browse/FLINK-34745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz closed FLINK-34745. Resolution: Fixed Fixed in 4142c4386a92f1ec5016583f4832f8869782765e > Parsing temporal table join throws cryptic exceptions > - > > Key: FLINK-34745 > URL: https://issues.apache.org/jira/browse/FLINK-34745 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.19.0 >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > > 1. Wrong expression type in {{AS OF}}: > {code} > SELECT * " + > "FROM Orders AS o JOIN " + > "RatesHistoryWithPK FOR SYSTEM_TIME AS OF 'o.rowtime' AS r " + > "ON o.currency = r.currency > {code} > throws: > {code} > java.lang.AssertionError: cannot convert CHAR literal to class > org.apache.calcite.util.TimestampString > {code} > 2. Not a simple table reference in {{AS OF}} > {code} > SELECT * " + > "FROM Orders AS o JOIN " + > "RatesHistoryWithPK FOR SYSTEM_TIME AS OF o.rowtime + INTERVAL '1' > SECOND AS r " + > "ON o.currency = r.currency > {code} > throws: > {code} > java.lang.AssertionError: no unique expression found for {id: o.rowtime, > prefix: 1}; count is 0 > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34745] Improve validations for a period in Time Travel [flink]
dawidwys merged PR #24534: URL: https://github.com/apache/flink/pull/24534 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34896][table] Migrate CorrelateSortToRankRule to java [flink]
flinkbot commented on PR #24545: URL: https://github.com/apache/flink/pull/24545#issuecomment-2009564801 ## CI report: * 83bedd5de8ddc74d3095feb937b1728b611db4a1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34896) Migrate CorrelateSortToRankRule
[ https://issues.apache.org/jira/browse/FLINK-34896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34896: --- Labels: pull-request-available (was: ) > Migrate CorrelateSortToRankRule > --- > > Key: FLINK-34896 > URL: https://issues.apache.org/jira/browse/FLINK-34896 > Project: Flink > Issue Type: Sub-task >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-34896][table] Migrate CorrelateSortToRankRule to java [flink]
snuyanzin opened a new pull request, #24545: URL: https://github.com/apache/flink/pull/24545 ## What is the purpose of the change A bit reasoning for the change >The reason is that since 1.28.0 ( [CALCITE-4787](https://issues.apache.org/jira/browse/CALCITE-4787) - Move core to use Immutables instead of ImmutableBeans ) Calcite started to use Immutables (https://immutables.github.io/) and since 1.29.0 removed ImmutableBeans ( [CALCITE-4839](https://issues.apache.org/jira/browse/CALCITE-4839) - Remove remnants of ImmutableBeans post 1.28 release ). All rule configuration related api which is not Immutables based is marked as deprecated. Since Immutables implies code generation while java compilation it is seems impossible to use for rules in Scala code. The PR migrates `CorrelateSortToRankRule` to java it doesn't touch `CorrelateSortToRankRuleTest` to be sure that java version continues passing it ## Verifying this change This change is already covered by existing tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no ) - The runtime per-record code paths (performance sensitive): (no ) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no ) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-34896) Mi
Sergey Nuyanzin created FLINK-34896: --- Summary: Mi Key: FLINK-34896 URL: https://issues.apache.org/jira/browse/FLINK-34896 Project: Flink Issue Type: Sub-task Reporter: Sergey Nuyanzin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34896) Migrate CorrelateSortToRankRule
[ https://issues.apache.org/jira/browse/FLINK-34896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Nuyanzin reassigned FLINK-34896: --- Assignee: Sergey Nuyanzin > Migrate CorrelateSortToRankRule > --- > > Key: FLINK-34896 > URL: https://issues.apache.org/jira/browse/FLINK-34896 > Project: Flink > Issue Type: Sub-task >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34896) Migrate CorrelateSortToRankRule
[ https://issues.apache.org/jira/browse/FLINK-34896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Nuyanzin updated FLINK-34896: Summary: Migrate CorrelateSortToRankRule (was: Mi) > Migrate CorrelateSortToRankRule > --- > > Key: FLINK-34896 > URL: https://issues.apache.org/jira/browse/FLINK-34896 > Project: Flink > Issue Type: Sub-task >Reporter: Sergey Nuyanzin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34239][core] Add copy() method for SerializerConfig [flink]
flinkbot commented on PR #24544: URL: https://github.com/apache/flink/pull/24544#issuecomment-2009531400 ## CI report: * 3d3ca469055b60fe00261751c3cb86c988c7c7bf UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34239) Introduce a deep copy method of SerializerConfig for merging with Table configs in org.apache.flink.table.catalog.DataTypeFactoryImpl
[ https://issues.apache.org/jira/browse/FLINK-34239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34239: --- Labels: pull-request-available (was: ) > Introduce a deep copy method of SerializerConfig for merging with Table > configs in org.apache.flink.table.catalog.DataTypeFactoryImpl > -- > > Key: FLINK-34239 > URL: https://issues.apache.org/jira/browse/FLINK-34239 > Project: Flink > Issue Type: Sub-task > Components: API / Core >Affects Versions: 1.19.0 >Reporter: Zhanghao Chen >Assignee: Kumar Mallikarjuna >Priority: Major > Labels: pull-request-available > > *Problem* > Currently, > org.apache.flink.table.catalog.DataTypeFactoryImpl#createSerializerExecutionConfig > will create a deep-copy of the SerializerConfig and merge Table config into > it. However, the deep copy is done by manully calling the getter and setter > methods of SerializerConfig, and is prone to human errors, e.g. missing > copying a newly added field in SerializerConfig. > *Proposal* > Introduce a deep copy method for SerializerConfig and replace the curr impl > in > org.apache.flink.table.catalog.DataTypeFactoryImpl#createSerializerExecutionConfig. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-34239][core] Add copy() method for SerializerConfig [flink]
kumar-mallikarjuna opened a new pull request, #24544: URL: https://github.com/apache/flink/pull/24544 ## What is the purpose of the change `org.apache.flink.table.catalog.DataTypeFactoryImpl#createSerializerConfig` is currently manually making a copy of a `SerializerConfig` object. This is cumbersome and error-prone. This change moves this copying logic inside a copy method in `SerializerConfig(Impl)`. ## Brief change log - A copy method is introduced in `SerializerConfig` and `SerializerConfigImpl`. - `DataTypeFactoryImpl#createSerializerExecutionConfig` reuses this method. ## Verifying this change This change added tests and can be verified as follows: - Added a unit test for `SerializerConfigImply.copy()` ## Does this pull request potentially affect one of the following parts: - Dependencies (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (don't know - changes the underlying implementation but no functional change to the serializers) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34895][table] Migrate FlinkRewriteSubQueryRule to java [flink]
flinkbot commented on PR #24543: URL: https://github.com/apache/flink/pull/24543#issuecomment-2009485635 ## CI report: * 7dc3990422cb330fb1aa8fcdd50872f74dcb283e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-26088][Connectors/ElasticSearch] Add Elasticsearch 8.0 support [flink-connector-elasticsearch]
mtfelisb commented on PR #53: URL: https://github.com/apache/flink-connector-elasticsearch/pull/53#issuecomment-2009477539 > Thanks @mtfelisb, looks good to me. Could you please rebase and squash all commits? Thank you for the review, @reswqa! Just did it :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-34895][table] Migrate FlinkRewriteSubQueryRule to java [flink]
snuyanzin opened a new pull request, #24543: URL: https://github.com/apache/flink/pull/24543 ## What is the purpose of the change A bit reasoning for the change >The reason is that since 1.28.0 ( [CALCITE-4787](https://issues.apache.org/jira/browse/CALCITE-4787) - Move core to use Immutables instead of ImmutableBeans ) Calcite started to use Immutables (https://immutables.github.io/) and since 1.29.0 removed ImmutableBeans ( [CALCITE-4839](https://issues.apache.org/jira/browse/CALCITE-4839) - Remove remnants of ImmutableBeans post 1.28 release ). All rule configuration related api which is not Immutables based is marked as deprecated. Since Immutables implies code generation while java compilation it is seems impossible to use for rules in Scala code. The PR migrates `FlinkRewriteSubQueryRule` to java it doesn't touch `FlinkRewriteSubQueryRuleTest` to be sure that java version continues passing it ## Verifying this change This change is already covered by existing tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no ) - The runtime per-record code paths (performance sensitive): (no ) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no ) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-34895) Migrate FlinkRewriteSubQueryRule
Sergey Nuyanzin created FLINK-34895: --- Summary: Migrate FlinkRewriteSubQueryRule Key: FLINK-34895 URL: https://issues.apache.org/jira/browse/FLINK-34895 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Sergey Nuyanzin Assignee: Sergey Nuyanzin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34895) Migrate FlinkRewriteSubQueryRule
[ https://issues.apache.org/jira/browse/FLINK-34895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34895: --- Labels: pull-request-available (was: ) > Migrate FlinkRewriteSubQueryRule > > > Key: FLINK-34895 > URL: https://issues.apache.org/jira/browse/FLINK-34895 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-34549][API] Introduce config, context and processingTimerService for DataStream API V2 [flink]
reswqa opened a new pull request, #24541: URL: https://github.com/apache/flink/pull/24541 ## What is the purpose of the change *Implementation of FLIP-410. This PR will Introduce config, context and processingTimerService for DataStream API V2.* ## Brief change log - *Introduce ProcessConfigurable to configurate processing of stream. * - *Introduce RuntimeContext* - *Support process timer service* ## Verifying this change Can be verified by unit tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34894) Migrate JoinDependentConditionDerivationRule
[ https://issues.apache.org/jira/browse/FLINK-34894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34894: --- Labels: pull-request-available (was: ) > Migrate JoinDependentConditionDerivationRule > > > Key: FLINK-34894 > URL: https://issues.apache.org/jira/browse/FLINK-34894 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34672) HA deadlock between JobMasterServiceLeadershipRunner and DefaultLeaderElectionService
[ https://issues.apache.org/jira/browse/FLINK-34672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl updated FLINK-34672: -- Affects Version/s: 1.19.0 1.17.2 1.20.0 > HA deadlock between JobMasterServiceLeadershipRunner and > DefaultLeaderElectionService > - > > Key: FLINK-34672 > URL: https://issues.apache.org/jira/browse/FLINK-34672 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.17.2, 1.19.0, 1.18.1, 1.20.0 >Reporter: Chesnay Schepler >Assignee: Matthias Pohl >Priority: Major > Fix For: 1.18.2, 1.20.0, 1.19.1 > > > We recently observed a deadlock in the JM within the HA system. > (see below for the thread dump) > [~mapohl] and I looked a bit into it and there appears to be a race condition > when leadership is revoked while a JobMaster is being started. > It appears to be caused by > {{JobMasterServiceLeadershipRunner#createNewJobMasterServiceProcess}} > forwarding futures while holding a lock; depending on whether the forwarded > future is already complete the next stage may or may not run while holding > that same lock. > We haven't determined yet whether we should be holding that lock or not. > {code} > "DefaultLeaderElectionService-leadershipOperationExecutor-thread-1" #131 > daemon prio=5 os_prio=0 cpu=157.44ms elapsed=78749.65s tid=0x7f531f43d000 > nid=0x19d waiting for monitor entry [0x7f53084fd000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.runIfStateRunning(JobMasterServiceLeadershipRunner.java:462) > - waiting to lock <0xf1c0e088> (a java.lang.Object) > at > org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.revokeLeadership(JobMasterServiceLeadershipRunner.java:397) > at > org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.notifyLeaderContenderOfLeadershipLoss(DefaultLeaderElectionService.java:484) > at > org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService$$Lambda$1252/0x000840ddec40.accept(Unknown > Source) > at java.util.HashMap.forEach(java.base@11.0.22/HashMap.java:1337) > at > org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadershipInternal(DefaultLeaderElectionService.java:452) > at > org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService$$Lambda$1251/0x000840dcf840.run(Unknown > Source) > at > org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.lambda$runInLeaderEventThread$3(DefaultLeaderElectionService.java:549) > - locked <0xf0e3f4d8> (a java.lang.Object) > at > org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService$$Lambda$1075/0x000840c23040.run(Unknown > Source) > at > java.util.concurrent.CompletableFuture$AsyncRun.run(java.base@11.0.22/CompletableFuture.java:1736) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.22/ThreadPoolExecutor.java:1128) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.22/ThreadPoolExecutor.java:628) > at java.lang.Thread.run(java.base@11.0.22/Thread.java:829) > {code} > {code} > "jobmanager-io-thread-1" #636 daemon prio=5 os_prio=0 cpu=125.56ms > elapsed=78699.01s tid=0x7f5321c6e800 nid=0x396 waiting for monitor entry > [0x7f530567d000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.hasLeadership(DefaultLeaderElectionService.java:366) > - waiting to lock <0xf0e3f4d8> (a java.lang.Object) > at > org.apache.flink.runtime.leaderelection.DefaultLeaderElection.hasLeadership(DefaultLeaderElection.java:52) > at > org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.isValidLeader(JobMasterServiceLeadershipRunner.java:509) > at > org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.lambda$forwardIfValidLeader$15(JobMasterServiceLeadershipRunner.java:520) > - locked <0xf1c0e088> (a java.lang.Object) > at > org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner$$Lambda$1320/0x000840e1a840.accept(Unknown > Source) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@11.0.22/CompletableFuture.java:859) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@11.0.22/CompletableFuture.java:837) > at > java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.22/CompletableFutu
Re: [PR] [FLINK-34894][table] Migrate JoinDependentConditionDerivationRule to java [flink]
flinkbot commented on PR #24542: URL: https://github.com/apache/flink/pull/24542#issuecomment-2009455915 ## CI report: * 564b0ac7360a5212a2a22bce9968c2dbe71bcc1b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34549][API] Introduce config, context and processingTimerService for DataStream API V2 [flink]
flinkbot commented on PR #24541: URL: https://github.com/apache/flink/pull/24541#issuecomment-2009426855 ## CI report: * 08547b7fa9baa619ec7b62bc893ccc6abcc86bb3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34643) JobIDLoggingITCase failed
[ https://issues.apache.org/jira/browse/FLINK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17828930#comment-17828930 ] Ryan Skraba commented on FLINK-34643: - * [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58429&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=8243] Here are the older build logs that should have been added over the weekend. * [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58320&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8349] * [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58320&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=8244] * [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352&view=logs&j=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3&t=712ade8c-ca16-5b76-3acd-14df33bc1cb1&l=8210] * [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58352&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7904] * [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58359&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8354] * [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58359&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=8249] * [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58366&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8354] * [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58366&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7903] * [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58398&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=8249] > JobIDLoggingITCase failed > - > > Key: FLINK-34643 > URL: https://issues.apache.org/jira/browse/FLINK-34643 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.20.0 >Reporter: Matthias Pohl >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available, test-stability > Fix For: 1.20.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=7897 > {code} > Mar 09 01:24:23 01:24:23.498 [ERROR] Tests run: 1, Failures: 0, Errors: 1, > Skipped: 0, Time elapsed: 4.209 s <<< FAILURE! -- in > org.apache.flink.test.misc.JobIDLoggingITCase > Mar 09 01:24:23 01:24:23.498 [ERROR] > org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(ClusterClient) > -- Time elapsed: 1.459 s <<< ERROR! > Mar 09 01:24:23 java.lang.IllegalStateException: Too few log events recorded > for org.apache.flink.runtime.jobmaster.JobMaster (12) - this must be a bug in > the test code > Mar 09 01:24:23 at > org.apache.flink.util.Preconditions.checkState(Preconditions.java:215) > Mar 09 01:24:23 at > org.apache.flink.test.misc.JobIDLoggingITCase.assertJobIDPresent(JobIDLoggingITCase.java:148) > Mar 09 01:24:23 at > org.apache.flink.test.misc.JobIDLoggingITCase.testJobIDLogging(JobIDLoggingITCase.java:132) > Mar 09 01:24:23 at java.lang.reflect.Method.invoke(Method.java:498) > Mar 09 01:24:23 at > java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189) > Mar 09 01:24:23 at > java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) > Mar 09 01:24:23 at > java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) > Mar 09 01:24:23 at > java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) > Mar 09 01:24:23 at > java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) > Mar 09 01:24:23 > {code} > The other test failures of this build were also caused by the same test: > * > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8349 > * > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58187&view=logs&j=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3&t=712ade8c-ca16-5b76-3acd-14df33bc1cb1&l=8209 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34770) Unify download links of legacy connector to fat jar
[ https://issues.apache.org/jira/browse/FLINK-34770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34770: --- Labels: pull-request-available (was: ) > Unify download links of legacy connector to fat jar > --- > > Key: FLINK-34770 > URL: https://issues.apache.org/jira/browse/FLINK-34770 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Priority: Minor > Labels: pull-request-available > Fix For: cdc-3.1.0 > > > Some links will lead to download fat jar, but some links will lead to > download thin jar, and some are invalid. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-34893] Bump checkstyle to 9.3 [flink]
snuyanzin opened a new pull request, #24540: URL: https://github.com/apache/flink/pull/24540 ## What is the purpose of the change The issue with current checkstyle is that there is checkstyle IntellijIdea plugin And recently it dropped checkstyle 8 support [1] At the same time we can not move to Checkstyle 10 since 10.x requires java 11+ Also it moves version to properties to allow build with different checkstyle versions for instance after this PR it would be possible to build with checkstyle 10 as well (leveraging jdk11+) like `mvn clean install -DskipTests -Dcheckstyle.version=10.14.2` [1] https://github.com/jshiell/checkstyle-idea/blob/main/CHANGELOG.md ## Verifying this change This change is a trivial rework ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes ) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: ( no) - The runtime per-record code paths (performance sensitive): ( no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no) - The S3 file system connector: ( no ) ## Documentation - Does this pull request introduce a new feature? ( no) - If yes, how is the feature documented? (not applicable ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Backport translated Chinese docs in master to release-3.0 branch [flink-cdc]
leonardBang merged PR #3179: URL: https://github.com/apache/flink-cdc/pull/3179 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-34893) Bump Checkstyle to 9+
Sergey Nuyanzin created FLINK-34893: --- Summary: Bump Checkstyle to 9+ Key: FLINK-34893 URL: https://issues.apache.org/jira/browse/FLINK-34893 Project: Flink Issue Type: Bug Components: Build System Reporter: Sergey Nuyanzin Assignee: Sergey Nuyanzin The issue with current checkstyle is that there is checkstyle IntellijIdea plugin And recently it dropped checkstyle 8 support [1] At the same time we can not move to Checkstyle 10 since 10.x requires java 11+ [1] https://github.com/jshiell/checkstyle-idea/blob/main/CHANGELOG.md -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34894) Migrate JoinDependentConditionDerivationRule
Sergey Nuyanzin created FLINK-34894: --- Summary: Migrate JoinDependentConditionDerivationRule Key: FLINK-34894 URL: https://issues.apache.org/jira/browse/FLINK-34894 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Sergey Nuyanzin Assignee: Sergey Nuyanzin -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34892][ci] Fix python test failure due to config file change [flink-connector-aws]
z3d1k commented on code in PR #133: URL: https://github.com/apache/flink-connector-aws/pull/133#discussion_r1531899935 ## flink-python/dev/integration_test.sh: ## @@ -41,10 +41,5 @@ FLINK_PYTHON_DIR=$(dirname "$CURRENT_DIR") # set the FLINK_TEST_LIB_DIR to "/flink-connector-aws/flink-python-connector-aws/target/dep..." export FLINK_TEST_LIBS="${FLINK_PYTHON_DIR}/target/test-dependencies/*" -# Temporarily update the installed 'pyflink_gateway_server.py' files with the new one -# Needed only until Flink 1.19 release -echo "Checking ${FLINK_PYTHON_DIR} for 'pyflink_gateway_server.py'" -find "${FLINK_PYTHON_DIR}/.tox" -name pyflink_gateway_server.py -exec cp "${FLINK_PYTHON_DIR}/pyflink/pyflink_gateway_server.py" {} \; Review Comment: Removed to use `pyflink_gateway_server.py` provided by pyflink instead of local copy. ## flink-python/tox.ini: ## @@ -21,7 +21,7 @@ # in multiple virtualenvs. This configuration file will run the # test suite on all supported python versions. # new environments will be excluded by default unless explicitly added to envlist. -envlist = {py37, py38, py39, py310}-cython Review Comment: The current fix does not work for python 3.7 because there is no 1.19 package released for this python version. Since dependency on `apache-flink` python package is set as `apache-flink>=1.17.0` - only the latest available Flink version is used during testing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-34892][ci] Fix python test failure due to config file change [flink-connector-aws]
z3d1k opened a new pull request, #133: URL: https://github.com/apache/flink-connector-aws/pull/133 ## Purpose of the change Removing `pyflink_gateway_server.py` override after Flink 1.19.0 release to support configuration file change. Override was initially introduced [here](https://github.com/apache/flink-connector-kafka/pull/69) and replicated for AWS connectors. ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing Local test run ## Significant changes *(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)* - [ ] Dependencies have been added or upgraded - [ ] Public API has been changed (Public API is any class annotated with `@Public(Evolving)`) - [ ] Serializers have been changed - [ ] New feature has been introduced - If yes, how is this documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34526][runtime] Actively disconnect the TM in RM to reduce restart time [flink]
flinkbot commented on PR #24539: URL: https://github.com/apache/flink/pull/24539#issuecomment-2009214558 ## CI report: * f4747e921f746298229adde25eefcb9f2b8e0cf4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34718) KeyedPartitionWindowedStream and NonPartitionWindowedStream IllegalStateException in AZP
[ https://issues.apache.org/jira/browse/FLINK-34718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17828931#comment-17828931 ] Ryan Skraba commented on FLINK-34718: - * [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58429&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=9664] (before the fix was committed to master) > KeyedPartitionWindowedStream and NonPartitionWindowedStream > IllegalStateException in AZP > > > Key: FLINK-34718 > URL: https://issues.apache.org/jira/browse/FLINK-34718 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: 1.20.0 >Reporter: Ryan Skraba >Assignee: Ryan Skraba >Priority: Critical > Labels: pull-request-available, test-stability > Fix For: 1.20.0 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58320&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=9646] > 18 of the KeyedPartitionWindowedStreamITCase and > NonKeyedPartitionWindowedStreamITCase unit tests introduced in FLINK-34543 > are failing in the adaptive scheduler profile, with errors similar to: > {code:java} > Mar 15 01:54:12 Caused by: java.lang.IllegalStateException: The adaptive > scheduler supports pipelined data exchanges (violated by MapPartition > (org.apache.flink.streaming.runtime.tasks.OneInputStreamTask) -> > ddb598ad156ed281023ba4eebbe487e3). > Mar 15 01:54:12 at > org.apache.flink.util.Preconditions.checkState(Preconditions.java:215) > Mar 15 01:54:12 at > org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.assertPreconditions(AdaptiveScheduler.java:438) > Mar 15 01:54:12 at > org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.(AdaptiveScheduler.java:356) > Mar 15 01:54:12 at > org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerFactory.createInstance(AdaptiveSchedulerFactory.java:124) > Mar 15 01:54:12 at > org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:121) > Mar 15 01:54:12 at > org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:384) > Mar 15 01:54:12 at > org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:361) > Mar 15 01:54:12 at > org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128) > Mar 15 01:54:12 at > org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100) > Mar 15 01:54:12 at > org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112) > Mar 15 01:54:12 at > java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) > Mar 15 01:54:12 ... 4 more > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)