[GitHub] [flink] KarmaGYZ commented on a change in pull request #16307: [FLINK-21925][api] Support configuring fine grained resource requirements via DataStream API
KarmaGYZ commented on a change in pull request #16307:
URL: https://github.com/apache/flink/pull/16307#discussion_r662754528
##
File path:
flink-core/src/main/java/org/apache/flink/api/common/operators/util/SlotSharingGroupUtils.java
##
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.operators.util;
+
+import org.apache.flink.api.common.operators.ResourceSpec;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
+import org.apache.flink.api.common.resources.ExternalResource;
+import org.apache.flink.util.Preconditions;
+
+import java.util.stream.Collectors;
+
+/** Utils for {@link SlotSharingGroup}. */
+public class SlotSharingGroupUtils {
+public static ResourceSpec extractResourceSpec(SlotSharingGroup
slotSharingGroup) {
+if (!slotSharingGroup.getCpuCores().isPresent()) {
+return ResourceSpec.UNKNOWN;
+}
+
+Preconditions.checkState(slotSharingGroup.getCpuCores().isPresent());
+
Preconditions.checkState(slotSharingGroup.getTaskHeapMemory().isPresent());
+
Preconditions.checkState(slotSharingGroup.getTaskOffHeapMemory().isPresent());
+
Preconditions.checkState(slotSharingGroup.getManagedMemory().isPresent());
+
+return ResourceSpec.newBuilder(slotSharingGroup.getCpuCores().get(), 0)
Review comment:
The unit of this param is `MB`. So, I think we cannot use
`slotSharingGroup.getTaskHeapMemory().get()` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16307: [FLINK-21925][api] Support configuring fine grained resource requirements via DataStream API
zhuzhurk commented on a change in pull request #16307:
URL: https://github.com/apache/flink/pull/16307#discussion_r662721409
##
File path:
flink-core/src/main/java/org/apache/flink/api/common/operators/util/SlotSharingGroupUtils.java
##
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.operators.util;
+
+import org.apache.flink.api.common.operators.ResourceSpec;
+import org.apache.flink.api.common.operators.SlotSharingGroup;
+import org.apache.flink.api.common.resources.ExternalResource;
+import org.apache.flink.util.Preconditions;
+
+import java.util.stream.Collectors;
+
+/** Utils for {@link SlotSharingGroup}. */
+public class SlotSharingGroupUtils {
+public static ResourceSpec extractResourceSpec(SlotSharingGroup
slotSharingGroup) {
+if (!slotSharingGroup.getCpuCores().isPresent()) {
+return ResourceSpec.UNKNOWN;
+}
+
+Preconditions.checkState(slotSharingGroup.getCpuCores().isPresent());
+
Preconditions.checkState(slotSharingGroup.getTaskHeapMemory().isPresent());
+
Preconditions.checkState(slotSharingGroup.getTaskOffHeapMemory().isPresent());
+
Preconditions.checkState(slotSharingGroup.getManagedMemory().isPresent());
+
+return ResourceSpec.newBuilder(slotSharingGroup.getCpuCores().get(), 0)
Review comment:
`0` -> `slotSharingGroup.getTaskHeapMemory().get()`
And the next line to `setTaskHeapMemory()` can be removed.
##
File path:
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/StreamGraphGeneratorTest.java
##
@@ -749,6 +750,60 @@ public void
testSettingSavepointRestoreSettingsSetterOverrides() {
equalTo(SavepointRestoreSettings.forPath("/tmp/savepoint1")));
}
+@Test
+public void testConfigureSlotSharingGroupResource() {
+final SlotSharingGroup ssg1 =
Review comment:
Given that `UNKNOWN` will not be included, let's also check cases that
1. operator without SSG set
2. operator with `UNKNOWN` resource SSG set
3. env with `UNKNOWN` resource SSG set
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-21504) State ownership: notify TMs about checkpoint subsumption
[ https://issues.apache.org/jira/browse/FLINK-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-21504: -- Description: Goal: enable TMs to discard state after subsumption (FLINK-23139). Proposed solution: include earliest non-subsumed checkpoint ID into checkpoint confirmation notification. Code-wise, it would require passing the checkpoint ID through many existing CheckpointListener implementations (e.g. Task). CheckpointListener is a public interface and should not be concerned with retained checkpoints (runtime detail). So it's better to remove "implements" from such classes and either call methods directly or introduce a new (runtime) interface if necessary. See [state ownership design doc|https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit?usp=sharing], in particular [subsumption notifications|https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit#heading=h.auqo5xe66sg5] was: Goal: enable TMs to discard state after subsumption (FLINK-23139). Proposed solution: include earliest non-subsumed checkpoint ID into checkpoint confirmation notification. See [state ownership design doc|https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit?usp=sharing], in particular [subsumption notifications|https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit#heading=h.auqo5xe66sg5] > State ownership: notify TMs about checkpoint subsumption > > > Key: FLINK-21504 > URL: https://issues.apache.org/jira/browse/FLINK-21504 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing >Reporter: Roman Khachatryan >Assignee: Yun Tang >Priority: Major > Labels: auto-unassigned > > Goal: enable TMs to discard state after subsumption (FLINK-23139). > Proposed solution: include earliest non-subsumed checkpoint ID into > checkpoint confirmation notification. > Code-wise, it would require passing the checkpoint ID through many existing > CheckpointListener implementations (e.g. Task). CheckpointListener is a > public interface and should not be concerned with retained checkpoints > (runtime detail). So it's better to remove "implements" from such classes and > either call methods directly or introduce a new (runtime) interface if > necessary. > > See [state ownership design > doc|https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit?usp=sharing], > in particular [subsumption > notifications|https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit#heading=h.auqo5xe66sg5] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16348: [FLINK-16093][docs-zh]Translate "System Functions" page of "Functions" into Chinese
flinkbot edited a comment on pull request #16348: URL: https://github.com/apache/flink/pull/16348#issuecomment-872690776 ## CI report: * 01f78c6f750f488bb1922e6aba615ade352cfd94 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19807) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16341: [FLINK-21804][state/changelog] Create and wire changelog storage with state backend
flinkbot edited a comment on pull request #16341: URL: https://github.com/apache/flink/pull/16341#issuecomment-872173720 ## CI report: * 01c17423b7f48b807c395ede07d2ee28e23292d3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19793) * a8089d2e0b879ce4b8f08db445be616e5ef1c167 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19809) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16307: [FLINK-21925][api] Support configuring fine grained resource requirements via DataStream API
flinkbot edited a comment on pull request #16307: URL: https://github.com/apache/flink/pull/16307#issuecomment-869431047 ## CI report: * 08b9c387ce32fc99185348a428b1d4ea7b042ae2 UNKNOWN * 02f9640278623b31942b4cfddfa66038dcad6bf4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19772) * 39494a1405c8ffafe8418156c1750b9962f637b4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19808) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #15055: [FLINK-21086][runtime][checkpoint] Support checkpoint alignment with finished input channels
gaoyunhaii commented on a change in pull request #15055:
URL: https://github.com/apache/flink/pull/15055#discussion_r662731089
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RecoveredInputChannel.java
##
@@ -205,6 +205,12 @@ public void resumeConsumption() {
throw new UnsupportedOperationException("RecoveredInputChannel should
never be blocked.");
}
+@Override
+public void acknowledgeAllRecordsProcessed() throws IOException {
+throw new UnsupportedOperationException(
+"RecoveredInputChannel should not need acknowledge all records
processed.");
+}
+
Review comment:
Hi, I think logically this should be supported:
1. Currently the downstream tasks in fact do not consider the
`EndOfUserRecordsEvent` for alignment, the `EndOfUserRecordsEvent` only used to
hold the upstream task till all the records are processed. Then if unaligned
checkpoints are triggered after `EndOfUserRecordsEvent` emitted, the downstream
task could still aligned, and during this process when persisting the channels,
with the current implementation all the events would be ignored, thus the
`EndOfUserRecordsEvent` would not affect the result of the unaligned
checkpoint.
2. Then after failover, with the current implementation the upstream task
would restart and may skip the execution of operators. Then when the new task
finishing, it would re-emitted the `EndOfUserRecordsEvent`. For the recovering
step, `EndOfUserRecordsEvent` should not have impaction since it would not be
included in 1.
Previously we have had ITCase on the PoC branch, but I'm still rebasing and
modifying the following commits, since we would enable the final checkpoint as
a whole at last, thus we might only be able to formally commit the ITCase at
then. Currently the 1 could be covered by the UT and 2 should not happen. I'll
also try to get the following commits done as soon as possible~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rovo98 commented on pull request #16347: [hotfix][javadocs] Fix broken inline link
rovo98 commented on pull request #16347: URL: https://github.com/apache/flink/pull/16347#issuecomment-872702281 Yes, I have seen the problem before. It seems the previous commit violates the code format, and I have updated (from 6d73e9b to ce116b0 ) the PR following the instruction given by CI results. (I have setup IDE according to https://ci.apache.org/projects/flink/flink-docs-master/docs/flinkdev/ide_setup/, already run mvn spotless:apply) What else should I do？Thanks for you review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16347: [hotfix][javadocs] Fix broken inline link
flinkbot edited a comment on pull request #16347: URL: https://github.com/apache/flink/pull/16347#issuecomment-872653761 ## CI report: * Unknown: [CANCELED](TBD) * ce116b0e54eb1e7735ebaaaf26914c72cd168687 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19806) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16348: [FLINK-16093][docs-zh]Translate "System Functions" page of "Functions" into Chinese
flinkbot edited a comment on pull request #16348: URL: https://github.com/apache/flink/pull/16348#issuecomment-872690776 ## CI report: * 01f78c6f750f488bb1922e6aba615ade352cfd94 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19807) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16341: [FLINK-21804][state/changelog] Create and wire changelog storage with state backend
flinkbot edited a comment on pull request #16341: URL: https://github.com/apache/flink/pull/16341#issuecomment-872173720 ## CI report: * 01c17423b7f48b807c395ede07d2ee28e23292d3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19793) * a8089d2e0b879ce4b8f08db445be616e5ef1c167 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16307: [FLINK-21925][api] Support configuring fine grained resource requirements via DataStream API
flinkbot edited a comment on pull request #16307: URL: https://github.com/apache/flink/pull/16307#issuecomment-869431047 ## CI report: * 08b9c387ce32fc99185348a428b1d4ea7b042ae2 UNKNOWN * 02f9640278623b31942b4cfddfa66038dcad6bf4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19772) * 39494a1405c8ffafe8418156c1750b9962f637b4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15972: Add common source and operator metrics.
flinkbot edited a comment on pull request #15972: URL: https://github.com/apache/flink/pull/15972#issuecomment-844955772 ## CI report: * 9223b2bfe148ae335891393f244dc2c29f39d2ee UNKNOWN * 1e6b0f37803e0fa5326184c17b3a1f3669667de2 UNKNOWN * 9bf9ec12f659236c7d63f0631c93dc760bbe05cb UNKNOWN * 93ae7ec71ee7958ef5ec295a6eba871b00c94574 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19800) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16307: [FLINK-21925][api] Support configuring fine grained resource requirements via DataStream API
zhuzhurk commented on a change in pull request #16307:
URL: https://github.com/apache/flink/pull/16307#discussion_r662720720
##
File path:
flink-core/src/main/java/org/apache/flink/api/common/operators/SlotSharingGroup.java
##
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.operators;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** Describe the name and the the different resource components of a slot
sharing group. */
+@PublicEvolving
+public class SlotSharingGroup implements Serializable {
+private static final long serialVersionUID = 1L;
+
+private final String name;
+
+/** How many cpu cores are needed. Can be null only if it is unknown. */
+@Nullable // can be null only for UNKNOWN
Review comment:
I see. Thanks for the explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-23213) Remove ProcessFunctionOperation
Dian Fu created FLINK-23213: --- Summary: Remove ProcessFunctionOperation Key: FLINK-23213 URL: https://issues.apache.org/jira/browse/FLINK-23213 Project: Flink Issue Type: Sub-task Components: API / Python Reporter: Dian Fu Assignee: Dian Fu Fix For: 1.14.0 Currently there are three operations to support Python DataStream API: ProcessFunctionOperation, DataStreamKeyedStatefulOperation DataStreamStatelessFunctionOperation. Actually we could refactor it a bit to merge ProcessFunctionOperation and DataStreamStatelessFunctionOperation into one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23212) Skip code-wise tests for pure documentation changing PRs
Xintong Song created FLINK-23212: Summary: Skip code-wise tests for pure documentation changing PRs Key: FLINK-23212 URL: https://issues.apache.org/jira/browse/FLINK-23212 Project: Flink Issue Type: Improvement Components: Build System / CI Reporter: Xintong Song It's brought up in [this thread|https://lists.apache.org/thread.html/r275493c169a585c9472beaf6796fb650b1493a9bea9d8daf2101c2df%40%3Cdev.flink.apache.org%3E] whether we should skip code-wise tests for pure documentation changing PRs, to avoid blocking documentation efforts on build instabilities and reduce the workload of CI workers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-22913) Support Python UDF chaining in Python DataStream API
[
https://issues.apache.org/jira/browse/FLINK-22913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu reassigned FLINK-22913:
---
Assignee: Dian Fu
> Support Python UDF chaining in Python DataStream API
>
>
> Key: FLINK-22913
> URL: https://issues.apache.org/jira/browse/FLINK-22913
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
>Reporter: Dian Fu
>Assignee: Dian Fu
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently, for the following job:
> {code}
> ds = ..
> ds.map(map_func1)
> .map(map_func2)
> {code}
> The Python function `map_func1` and `map_func2` will runs in separate Python
> workers and the result of `map_func1` will be transferred to JVM and then
> transferred to `map_func2` which may resides in another Python worker. This
> introduces redundant communication and serialization/deserialization overhead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #16348: [FLINK-16093][docs-zh]Translate "System Functions" page of "Functions" into Chinese
flinkbot commented on pull request #16348: URL: https://github.com/apache/flink/pull/16348#issuecomment-872690776 ## CI report: * 01f78c6f750f488bb1922e6aba615ade352cfd94 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16347: [hotfix][javadocs] Fix broken inline link
flinkbot edited a comment on pull request #16347: URL: https://github.com/apache/flink/pull/16347#issuecomment-872653761 ## CI report: * Unknown: [CANCELED](TBD) * ce116b0e54eb1e7735ebaaaf26914c72cd168687 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15322: [FLINK-21353][state] Add DFS-based StateChangelog (TM-owned state)
flinkbot edited a comment on pull request #15322: URL: https://github.com/apache/flink/pull/15322#issuecomment-804015738 ## CI report: * c6ae187f7949bb09b629fa51375dd9becd406b28 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19799) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] KarmaGYZ commented on pull request #16307: [FLINK-21925][api] Support configuring fine grained resource requirements via DataStream API
KarmaGYZ commented on pull request #16307: URL: https://github.com/apache/flink/pull/16307#issuecomment-872688266 @zhuzhurk Thanks for the review! Please take another look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16093) Translate "System Functions" page of "Functions" into Chinese
[
https://issues.apache.org/jira/browse/FLINK-16093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373190#comment-17373190
]
ZhiJie Yang commented on FLINK-16093:
-
and I found I can't change the status of this issue, does it is a bug for Jira?
!image-2021-07-02-11-26-04-216.png!
> Translate "System Functions" page of "Functions" into Chinese
> --
>
> Key: FLINK-16093
> URL: https://issues.apache.org/jira/browse/FLINK-16093
> Project: Flink
> Issue Type: Sub-task
> Components: chinese-translation, Documentation
>Reporter: Jark Wu
>Assignee: ZhiJie Yang
>Priority: Major
> Labels: auto-unassigned, pull-request-available
> Attachments: image-2021-07-02-11-26-04-216.png
>
>
> The page url is
> https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/functions/systemFunctions.html
> The markdown file is located in
> {{flink/docs/dev/table/functions/systemFunctions.zh.md}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #16348: [FLINK-16093][docs-zh]Translate "System Functions" page of "Functions" into Chinese
flinkbot commented on pull request #16348: URL: https://github.com/apache/flink/pull/16348#issuecomment-872682867 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 01f78c6f750f488bb1922e6aba615ade352cfd94 (Fri Jul 02 03:18:44 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-16093) Translate "System Functions" page of "Functions" into Chinese
[
https://issues.apache.org/jira/browse/FLINK-16093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhiJie Yang updated FLINK-16093:
Attachment: image-2021-07-02-11-26-04-216.png
> Translate "System Functions" page of "Functions" into Chinese
> --
>
> Key: FLINK-16093
> URL: https://issues.apache.org/jira/browse/FLINK-16093
> Project: Flink
> Issue Type: Sub-task
> Components: chinese-translation, Documentation
>Reporter: Jark Wu
>Assignee: ZhiJie Yang
>Priority: Major
> Labels: auto-unassigned, pull-request-available
> Attachments: image-2021-07-02-11-26-04-216.png
>
>
> The page url is
> https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/functions/systemFunctions.html
> The markdown file is located in
> {{flink/docs/dev/table/functions/systemFunctions.zh.md}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18464) ClassCastException during namespace serialization for checkpoint (Heap and RocksDB)
[
https://issues.apache.org/jira/browse/FLINK-18464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373186#comment-17373186
]
Yun Tang commented on FLINK-18464:
--
[~guxiangfly], in general, adding a performance test is not a good choice and
you could refer to
[flink-benchmarks|https://github.com/apache/flink-benchmarks] to see whether
[window-benchmarks|https://github.com/apache/flink-benchmarks/blob/master/src/main/java/org/apache/flink/benchmark/WindowBenchmarks.java]
could satisify your request.
> ClassCastException during namespace serialization for checkpoint (Heap and
> RocksDB)
> ---
>
> Key: FLINK-18464
> URL: https://issues.apache.org/jira/browse/FLINK-18464
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / State Backends
>Affects Versions: 1.9.3, 1.13.1
>Reporter: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-06-21-20-06-51-323.png,
> image-2021-06-21-20-07-30-281.png, image-2021-06-21-20-07-43-246.png,
> image-2021-06-21-20-33-39-295.png, image-2021-06-23-14-34-37-703.png,
> image-2021-06-24-16-41-54-425.png, image-2021-06-24-17-51-53-734.png
>
>
> (see FLINK-23036 for error details with RocksDB)
>
> From
> [thread|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Checkpoint-failed-because-of-TimeWindow-cannot-be-cast-to-VoidNamespace-td36310.html]
> {quote}I'm using flink 1.9 on Mesos and I try to use my own trigger and
> evictor. The state is stored to memory.
> {quote}
>
>
> {code:java}
> input.setParallelism(processParallelism)
> .assignTimestampsAndWatermarks(new UETimeAssigner)
> .keyBy(_.key)
> .window(TumblingEventTimeWindows.of(Time.minutes(20)))
> .trigger(new MyTrigger)
> .evictor(new MyEvictor)
> .process(new MyFunction).setParallelism(aggregateParallelism)
> .addSink(kafkaSink).setParallelism(sinkParallelism)
> .name("kafka-record-sink"){code}
>
>
> {code:java}
> java.lang.Exception: Could not materialize checkpoint 1 for operator
> Window(TumblingEventTimeWindows(120), JoinTrigger, JoinEvictor,
> ScalaProcessWindowFunctionWrapper) -> Sink: kafka-record-sink (2/5).
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:1100)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:1042)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.ClassCastException:
> org.apache.flink.streaming.api.windowing.windows.TimeWindow cannot be cast to
> org.apache.flink.runtime.state.VoidNamespace
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:450)
> at
> org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:47)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:1011)
>
> ... 3 more
> Caused by: java.lang.ClassCastException:
> org.apache.flink.streaming.api.windowing.windows.TimeWindow cannot be cast to
> org.apache.flink.runtime.state.VoidNamespace
> at
> org.apache.flink.runtime.state.VoidNamespaceSerializer.serialize(VoidNamespaceSerializer.java:32)
> at
> org.apache.flink.runtime.state.heap.CopyOnWriteStateMapSnapshot.writeState(CopyOnWriteStateMapSnapshot.java:114)
> at
> org.apache.flink.runtime.state.heap.AbstractStateTableSnapshot.writeStateInKeyGroup(AbstractStateTableSnapshot.java:121)
> at
> org.apache.flink.runtime.state.heap.CopyOnWriteStateTableSnapshot.writeStateInKeyGroup(CopyOnWriteStateTableSnapshot.java:37)
> at
> org.apache.flink.runtime.state.heap.HeapSnapshotStrategy$1.callInternal(HeapSnapshotStrategy.java:191)
> at
> org.apache.flink.runtime.state.heap.HeapSnapshotStrategy$1.callInternal(HeapSnapshotStrategy.java:158)
> at
> org.apache.flink.runtime.state.AsyncSnapshotCallable.call(AsyncSnapshotCallable.java:75)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:447)
>
> ... 5 more
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16093) Translate "System Functions" page of "Functions" into Chinese
[
https://issues.apache.org/jira/browse/FLINK-16093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373185#comment-17373185
]
ZhiJie Yang commented on FLINK-16093:
-
Hi [~jark], I just have done the work, but before the translation, I found some
wrong page will block building sites, so please first check the PR in
[https://github.com/apache/flink/pull/16320]
> Translate "System Functions" page of "Functions" into Chinese
> --
>
> Key: FLINK-16093
> URL: https://issues.apache.org/jira/browse/FLINK-16093
> Project: Flink
> Issue Type: Sub-task
> Components: chinese-translation, Documentation
>Reporter: Jark Wu
>Assignee: ZhiJie Yang
>Priority: Major
> Labels: auto-unassigned, pull-request-available
>
> The page url is
> https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/functions/systemFunctions.html
> The markdown file is located in
> {{flink/docs/dev/table/functions/systemFunctions.zh.md}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rovo98 commented on pull request #16347: [hotfix][javadocs] Fix broken inline link
rovo98 commented on pull request #16347: URL: https://github.com/apache/flink/pull/16347#issuecomment-872684520 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16192: [FLINK-22954][table-planner-blink] Rewrite Join on constant TableFunctionScan to Correlate
flinkbot edited a comment on pull request #16192: URL: https://github.com/apache/flink/pull/16192#issuecomment-863860019 ## CI report: * 5bca7943d5ca42ee8340f05b8102e7d3ecb41b75 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19783) * 7832a85bb50b435242438fee99a53c474e388355 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19805) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ZhijieYang commented on pull request #16320: [hotfix] [docs] Fix the bug that the document address does not match the actual web page
ZhijieYang commented on pull request #16320: URL: https://github.com/apache/flink/pull/16320#issuecomment-872683479 $ ./build_docs.sh -p Start building sites … hugo v0.84.1-4BD65E22+extended windows/amd64 BuildDate=2021-06-24T11:44:23Z VendorInfo=gohugoio WARN 2021/07/02 11:17:01 Expand shortcode is deprecated. Use 'details' instead. ERROR 2021/07/02 11:17:03 Alias "/zh/dev/custom_serializers." contains component with a trailing space or period, problematic on Windows WARN 2021/07/02 11:17:04 Expand shortcode is deprecated. Use 'details' instead. Built in 6422 ms Error: Error building site: logged 1 error(s) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23196) JobMasterITCase fail on azure due to BindException
[
https://issues.apache.org/jira/browse/FLINK-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373178#comment-17373178
]
Xintong Song commented on FLINK-23196:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19798=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=c2734c79-73b6-521c-e85a-67c7ecae9107=9340
> JobMasterITCase fail on azure due to BindException
> --
>
> Key: FLINK-23196
> URL: https://issues.apache.org/jira/browse/FLINK-23196
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19753=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56=4251
> {code}
> Jul 01 00:00:27 [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 4.272 s <<< FAILURE! - in
> org.apache.flink.runtime.jobmaster.JobMasterITCase
> Jul 01 00:00:27 [ERROR]
> testRejectionOfEmptyJobGraphs(org.apache.flink.runtime.jobmaster.JobMasterITCase)
> Time elapsed: 3.009 s <<< ERROR!
> Jul 01 00:00:27 org.apache.flink.util.FlinkException: Could not create the
> DispatcherResourceManagerComponent.
> Jul 01 00:00:27 at
> org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:275)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.createDispatcherResourceManagerComponents(MiniCluster.java:470)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.setupDispatcherResourceManagerComponents(MiniCluster.java:429)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:373)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.jobmaster.JobMasterITCase.testRejectionOfEmptyJobGraphs(JobMasterITCase.java:56)
> Jul 01 00:00:27 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jul 01 00:00:27 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jul 01 00:00:27 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 01 00:00:27 at java.lang.reflect.Method.invoke(Method.java:498)
> Jul 01 00:00:27 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jul 01 00:00:27 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jul 01 00:00:27 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Jul 01 00:00:27 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jul 01 00:00:27 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Jul 01 00:00:27 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> Jul 01 00:00:27 at
>
[jira] [Commented] (FLINK-23196) JobMasterITCase fail on azure due to BindException
[
https://issues.apache.org/jira/browse/FLINK-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373177#comment-17373177
]
Xintong Song commented on FLINK-23196:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19798=logs=59c257d0-c525-593b-261d-e96a86f1926b=b93980e3-753f-5433-6a19-13747adae66a=9340
> JobMasterITCase fail on azure due to BindException
> --
>
> Key: FLINK-23196
> URL: https://issues.apache.org/jira/browse/FLINK-23196
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19753=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56=4251
> {code}
> Jul 01 00:00:27 [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 4.272 s <<< FAILURE! - in
> org.apache.flink.runtime.jobmaster.JobMasterITCase
> Jul 01 00:00:27 [ERROR]
> testRejectionOfEmptyJobGraphs(org.apache.flink.runtime.jobmaster.JobMasterITCase)
> Time elapsed: 3.009 s <<< ERROR!
> Jul 01 00:00:27 org.apache.flink.util.FlinkException: Could not create the
> DispatcherResourceManagerComponent.
> Jul 01 00:00:27 at
> org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:275)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.createDispatcherResourceManagerComponents(MiniCluster.java:470)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.setupDispatcherResourceManagerComponents(MiniCluster.java:429)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:373)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.jobmaster.JobMasterITCase.testRejectionOfEmptyJobGraphs(JobMasterITCase.java:56)
> Jul 01 00:00:27 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jul 01 00:00:27 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jul 01 00:00:27 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 01 00:00:27 at java.lang.reflect.Method.invoke(Method.java:498)
> Jul 01 00:00:27 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jul 01 00:00:27 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jul 01 00:00:27 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Jul 01 00:00:27 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jul 01 00:00:27 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Jul 01 00:00:27 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> Jul 01 00:00:27 at
>
[jira] [Commented] (FLINK-23196) JobMasterITCase fail on azure due to BindException
[
https://issues.apache.org/jira/browse/FLINK-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373176#comment-17373176
]
Xintong Song commented on FLINK-23196:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19798=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56=4063
> JobMasterITCase fail on azure due to BindException
> --
>
> Key: FLINK-23196
> URL: https://issues.apache.org/jira/browse/FLINK-23196
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19753=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56=4251
> {code}
> Jul 01 00:00:27 [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 4.272 s <<< FAILURE! - in
> org.apache.flink.runtime.jobmaster.JobMasterITCase
> Jul 01 00:00:27 [ERROR]
> testRejectionOfEmptyJobGraphs(org.apache.flink.runtime.jobmaster.JobMasterITCase)
> Time elapsed: 3.009 s <<< ERROR!
> Jul 01 00:00:27 org.apache.flink.util.FlinkException: Could not create the
> DispatcherResourceManagerComponent.
> Jul 01 00:00:27 at
> org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:275)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.createDispatcherResourceManagerComponents(MiniCluster.java:470)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.setupDispatcherResourceManagerComponents(MiniCluster.java:429)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:373)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.jobmaster.JobMasterITCase.testRejectionOfEmptyJobGraphs(JobMasterITCase.java:56)
> Jul 01 00:00:27 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jul 01 00:00:27 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jul 01 00:00:27 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 01 00:00:27 at java.lang.reflect.Method.invoke(Method.java:498)
> Jul 01 00:00:27 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jul 01 00:00:27 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jul 01 00:00:27 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Jul 01 00:00:27 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jul 01 00:00:27 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Jul 01 00:00:27 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> Jul 01 00:00:27 at
>
[GitHub] [flink] rovo98 removed a comment on pull request #16347: [hotfix][javadocs] Fix broken inline link
rovo98 removed a comment on pull request #16347: URL: https://github.com/apache/flink/pull/16347#issuecomment-872681797 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-21504) State ownership: notify TMs about checkpoint subsumption
[ https://issues.apache.org/jira/browse/FLINK-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Tang reassigned FLINK-21504: Assignee: Yun Tang > State ownership: notify TMs about checkpoint subsumption > > > Key: FLINK-21504 > URL: https://issues.apache.org/jira/browse/FLINK-21504 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing >Reporter: Roman Khachatryan >Assignee: Yun Tang >Priority: Major > Labels: auto-unassigned > > Goal: enable TMs to discard state after subsumption (FLINK-23139). > Proposed solution: include earliest non-subsumed checkpoint ID into > checkpoint confirmation notification. > See [state ownership design > doc|https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit?usp=sharing], > in particular [subsumption > notifications|https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit#heading=h.auqo5xe66sg5] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-23137) State ownership: disable discarding and ref counting state of ChangelogStateBackendHandle on JM
[ https://issues.apache.org/jira/browse/FLINK-23137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Tang reassigned FLINK-23137: Assignee: Zakelly Lan > State ownership: disable discarding and ref counting state of > ChangelogStateBackendHandle on JM > --- > > Key: FLINK-23137 > URL: https://issues.apache.org/jira/browse/FLINK-23137 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Zakelly Lan >Priority: Major > Fix For: 1.14.0 > > > TM should own changelog backend state to prevent re-uploading state on > checkpoint abortion (or missing confirmation). A simpler solution to only own > aborted state is less maintanable in the long run. > For that, on JM side, ChangelogStateBackendHandle shouldn't discard and *nor > register any shared state*. > > Ticket for TM-side: FLINK-23139 > > See [state ownership design > doc|https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit?usp=sharing] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] KarmaGYZ commented on a change in pull request #16307: [FLINK-21925][api] Support configuring fine grained resource requirements via DataStream API
KarmaGYZ commented on a change in pull request #16307: URL: https://github.com/apache/flink/pull/16307#discussion_r662708965 ## File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java ## @@ -189,6 +195,8 @@ private final List jobListeners = new ArrayList<>(); +private final Map slotSharingGroupResources = new HashMap<>(); Review comment: `slotSharingGroupResources` should only contain specified resources, I'll claim this in the javadoc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rovo98 commented on pull request #16347: [hotfix][javadocs] Fix broken inline link
rovo98 commented on pull request #16347: URL: https://github.com/apache/flink/pull/16347#issuecomment-872681797 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-16093) Translate "System Functions" page of "Functions" into Chinese
[
https://issues.apache.org/jira/browse/FLINK-16093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-16093:
---
Labels: auto-unassigned pull-request-available (was: auto-unassigned)
> Translate "System Functions" page of "Functions" into Chinese
> --
>
> Key: FLINK-16093
> URL: https://issues.apache.org/jira/browse/FLINK-16093
> Project: Flink
> Issue Type: Sub-task
> Components: chinese-translation, Documentation
>Reporter: Jark Wu
>Assignee: ZhiJie Yang
>Priority: Major
> Labels: auto-unassigned, pull-request-available
>
> The page url is
> https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/functions/systemFunctions.html
> The markdown file is located in
> {{flink/docs/dev/table/functions/systemFunctions.zh.md}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] ZhijieYang opened a new pull request #16348: [FLINK-16093][docs-zh]Translate "System Functions" page of "Functions" into Chinese
ZhijieYang opened a new pull request #16348: URL: https://github.com/apache/flink/pull/16348 ## What is the purpose of the change Translate "System Functions" page of "Functions" into Chinese, and fix some wrong descriptions in the English version ## Brief change log - Translate the "System Functions" page of "Functions" into Chinese - Fix some wrong descriptions in the English version ## Verifying this change This change is a trivial rework/code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-23198) An old interface method is used in this section of [Passing Options Factory to RocksDB].
[ https://issues.apache.org/jira/browse/FLINK-23198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373160#comment-17373160 ] Yun Tang edited comment on FLINK-23198 at 7/2/21, 3:04 AM: --- [~yanchenyun] You don't need to create another ticket. In general, one ticket could follow one problem at different versions. BTW, I did not see any PR related to this ticket. If you create the PR with correct ticket number, the correct link would be appended in this ticket. was (Author: yunta): [~yanchenyun] You don't need to create another ticket. In general, one ticket could follow one problem at different version. BTW, I did not see any PR related to this ticket. If you create the PR with correct ticket number, the correct link would be appended in this ticket. > An old interface method is used in this section of [Passing Options Factory > to RocksDB]. > > > Key: FLINK-23198 > URL: https://issues.apache.org/jira/browse/FLINK-23198 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.12.0 >Reporter: Carl >Assignee: Carl >Priority: Minor > Attachments: image-2021-07-01-11-30-57-676.png, > image-2021-07-01-11-32-25-200.png > > > [https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/state/state_backends.html] > > !image-2021-07-01-11-30-57-676.png! > In version 1.12 of Flink, this method has been replaced by the following one： > !image-2021-07-01-11-32-25-200.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #11877: [FLINK-16641][network] Announce sender's backlog to solve the deadlock issue without exclusive buffers
flinkbot edited a comment on pull request #11877: URL: https://github.com/apache/flink/pull/11877#issuecomment-618273998 ## CI report: * c161aff32e5aed483ca0722e0504ec8493348947 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19767) * 159596e8f11586211a5f54e6aaae2c9fdee532a2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19804) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23198) An old interface method is used in this section of [Passing Options Factory to RocksDB].
[ https://issues.apache.org/jira/browse/FLINK-23198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373160#comment-17373160 ] Yun Tang commented on FLINK-23198: -- [~yanchenyun] You don't need to create another ticket. In general, one ticket could follow one problem at different version. BTW, I did not see any PR related to this ticket. If you create the PR with correct ticket number, the correct link would be appended in this ticket. > An old interface method is used in this section of [Passing Options Factory > to RocksDB]. > > > Key: FLINK-23198 > URL: https://issues.apache.org/jira/browse/FLINK-23198 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.12.0 >Reporter: Carl >Assignee: Carl >Priority: Minor > Attachments: image-2021-07-01-11-30-57-676.png, > image-2021-07-01-11-32-25-200.png > > > [https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/state/state_backends.html] > > !image-2021-07-01-11-30-57-676.png! > In version 1.12 of Flink, this method has been replaced by the following one： > !image-2021-07-01-11-32-25-200.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-23198) An old interface method is used in this section of [Passing Options Factory to RocksDB].
[ https://issues.apache.org/jira/browse/FLINK-23198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Tang reassigned FLINK-23198: Assignee: Carl > An old interface method is used in this section of [Passing Options Factory > to RocksDB]. > > > Key: FLINK-23198 > URL: https://issues.apache.org/jira/browse/FLINK-23198 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.12.0 >Reporter: Carl >Assignee: Carl >Priority: Minor > Attachments: image-2021-07-01-11-30-57-676.png, > image-2021-07-01-11-32-25-200.png > > > [https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/state/state_backends.html] > > !image-2021-07-01-11-30-57-676.png! > In version 1.12 of Flink, this method has been replaced by the following one： > !image-2021-07-01-11-32-25-200.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-23211) An old interface method is used in this section of [Passing Options Factory to RocksDB].
[ https://issues.apache.org/jira/browse/FLINK-23211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Tang closed FLINK-23211. Resolution: Duplicate > An old interface method is used in this section of [Passing Options Factory > to RocksDB]. > > > Key: FLINK-23211 > URL: https://issues.apache.org/jira/browse/FLINK-23211 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.13.0 >Reporter: Carl >Priority: Minor > Attachments: image-2021-07-02-09-19-30-547.png, > image-2021-07-02-09-20-03-344.png > > > [https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/state/state_backends/] > !image-2021-07-02-09-19-30-547.png! > > *In version 1.13 of Flink, this method has been replaced by the following > interface：* > > !image-2021-07-02-09-20-03-344.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired
[
https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Liao updated FLINK-23208:
---
Description:
The problem is from the codes below:
{code:java}
public static long getProcessingTimeDelay(long processingTimestamp, long
currentTimestamp) {
// delay the firing of the timer by 1 ms to align the semantics with
watermark. A watermark
// T says we won't see elements in the future with a timestamp smaller
or equal to T.
// With processing time, we therefore need to delay firing the timer by
one ms.
return Math.max(processingTimestamp - currentTimestamp, 0) + 1;
}
{code}
Assuming a Flink job creates 1 timer per millionseconds, and is able to consume
1 timer/ms. Here is what will happen:
* Timestmap1(1st ms): timer1 is registered and will be triggered on Timestamp2.
* Timestamp2(2nd ms): timer2 is registered and timer1 is triggered
* Timestamp3(3rd ms): timer3 is registered and timer1 is consumed, after this,
{{InternalTimerServiceImpl}} registers next timer, which is timer2, and timer2
will be triggered on Timestamp4(wait 1ms at least)
* Timestamp4(4th ms): timer4 is registered and timer2 is triggered
* Timestamp5(5th ms): timer5 is registered and timer2 is consumed, after this,
{{InternalTimerServiceImpl}} registers next timer, which is timer3, and timer3
will be triggered on Timestamp6(wait 1ms at least)
As we can see here, the ability of the Flink job is consuming 1 timer/ms, but
it's actually able to consume 0.5 timer/ms. And another problem is that we
cannot observe the delay from the lag metrics of the source(Kafka). Instead,
what we can tell is that the moment of output is much later than expected. I've
added a metrics in our inner version, we can see the lag of the timer
triggering keeps increasing:
!screenshot-1.png!
*In another word, we should never let the late processing timer wait 1ms, I
think a simple change would be as below:*
{code:java}
return Math.max(processingTimestamp - currentTimestamp, -1) + 1;
{code}
was:
The problem is from the codes below:
{code:java}
public static long getProcessingTimeDelay(long processingTimestamp, long
currentTimestamp) {
// delay the firing of the timer by 1 ms to align the semantics with
watermark. A watermark
// T says we won't see elements in the future with a timestamp smaller
or equal to T.
// With processing time, we therefore need to delay firing the timer by
one ms.
return Math.max(processingTimestamp - currentTimestamp, 0) + 1;
}
{code}
Assuming a Flink job creates 1 timer per millionseconds, and is able to consume
1 timer/ms. Here is what will happen:
* Timestmap1(1st ms): timer1 is registered and will be triggered on Timestamp2.
* Timestamp2(2nd ms): timer2 is registered and timer1 is triggered
* Timestamp3(3rd ms): timer3 is registered and timer1 is consumed, after this,
{{InternalTimerServiceImpl}} registers next timer, which is timer2, and timer2
will be triggered on Timestamp4(wait 1ms at least)
* Timestamp4(4th ms): timer4 is registered and timer2 is triggered
* Timestamp5(5th ms): timer5 is registered and timer2 is consumed, after this,
{{InternalTimerServiceImpl}} registers next timer, which is timer3, and timer3
will be triggered on Timestamp6(wait 1ms at least)
As we can see here, the ability of the Flink job is consuming 1 timer/ms, but
it's actually able to consume 0.5 timer/ms. And the biggest problem is that we
cannot observe the delay from the lag metrics of the source(Kafka). Instead,
what we can tell is that the moment of output is much later than expected.
*In another word, we should never let the late processing timer wait 1ms, I
think a simple change would be as below:*
{code:java}
return Math.max(processingTimestamp - currentTimestamp, -1) + 1;
{code}
> Late processing timers need to wait 1ms at least to be fired
>
>
> Key: FLINK-23208
> URL: https://issues.apache.org/jira/browse/FLINK-23208
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.11.0
>Reporter: Jiayi Liao
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> The problem is from the codes below:
> {code:java}
> public static long getProcessingTimeDelay(long processingTimestamp, long
> currentTimestamp) {
> // delay the firing of the timer by 1 ms to align the semantics with
> watermark. A watermark
> // T says we won't see elements in the future with a timestamp smaller
> or equal to T.
> // With processing time, we therefore need to delay firing the timer by
> one ms.
> return Math.max(processingTimestamp - currentTimestamp, 0) + 1;
> }
> {code}
> Assuming a Flink job creates 1 timer per millionseconds, and is able
[jira] [Updated] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired
[
https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Liao updated FLINK-23208:
---
Attachment: screenshot-1.png
> Late processing timers need to wait 1ms at least to be fired
>
>
> Key: FLINK-23208
> URL: https://issues.apache.org/jira/browse/FLINK-23208
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.11.0
>Reporter: Jiayi Liao
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> The problem is from the codes below:
> {code:java}
> public static long getProcessingTimeDelay(long processingTimestamp, long
> currentTimestamp) {
> // delay the firing of the timer by 1 ms to align the semantics with
> watermark. A watermark
> // T says we won't see elements in the future with a timestamp smaller
> or equal to T.
> // With processing time, we therefore need to delay firing the timer by
> one ms.
> return Math.max(processingTimestamp - currentTimestamp, 0) + 1;
> }
> {code}
> Assuming a Flink job creates 1 timer per millionseconds, and is able to
> consume 1 timer/ms. Here is what will happen:
> * Timestmap1(1st ms): timer1 is registered and will be triggered on
> Timestamp2.
> * Timestamp2(2nd ms): timer2 is registered and timer1 is triggered
> * Timestamp3(3rd ms): timer3 is registered and timer1 is consumed, after
> this, {{InternalTimerServiceImpl}} registers next timer, which is timer2, and
> timer2 will be triggered on Timestamp4(wait 1ms at least)
> * Timestamp4(4th ms): timer4 is registered and timer2 is triggered
> * Timestamp5(5th ms): timer5 is registered and timer2 is consumed, after
> this, {{InternalTimerServiceImpl}} registers next timer, which is timer3, and
> timer3 will be triggered on Timestamp6(wait 1ms at least)
> As we can see here, the ability of the Flink job is consuming 1 timer/ms, but
> it's actually able to consume 0.5 timer/ms. And the biggest problem is that
> we cannot observe the delay from the lag metrics of the source(Kafka).
> Instead, what we can tell is that the moment of output is much later than
> expected.
> *In another word, we should never let the late processing timer wait 1ms, I
> think a simple change would be as below:*
> {code:java}
> return Math.max(processingTimestamp - currentTimestamp, -1) + 1;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] lirui-apache edited a comment on pull request #16335: [FLINK-23168] Catalog shouldn't merge properties for alter DB operation
lirui-apache edited a comment on pull request #16335: URL: https://github.com/apache/flink/pull/16335#issuecomment-872673902 The e2e_ci passed in my personal pipeline: https://dev.azure.com/lirui-apache/flink/_build/results?buildId=559=results -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lirui-apache commented on pull request #16335: [FLINK-23168] Catalog shouldn't merge properties for alter DB operation
lirui-apache commented on pull request #16335: URL: https://github.com/apache/flink/pull/16335#issuecomment-872673902 The e2e_ci passed in my personal pipeline: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16192: [FLINK-22954][table-planner-blink] Rewrite Join on constant TableFunctionScan to Correlate
flinkbot edited a comment on pull request #16192: URL: https://github.com/apache/flink/pull/16192#issuecomment-863860019 ## CI report: * 5bca7943d5ca42ee8340f05b8102e7d3ecb41b75 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19783) * 7832a85bb50b435242438fee99a53c474e388355 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11877: [FLINK-16641][network] Announce sender's backlog to solve the deadlock issue without exclusive buffers
flinkbot edited a comment on pull request #11877: URL: https://github.com/apache/flink/pull/11877#issuecomment-618273998 ## CI report: * c161aff32e5aed483ca0722e0504ec8493348947 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19767) * 159596e8f11586211a5f54e6aaae2c9fdee532a2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16347: [hotfix][javadocs] Fix broken inline link
flinkbot edited a comment on pull request #16347: URL: https://github.com/apache/flink/pull/16347#issuecomment-872653761 ## CI report: * 6d73e9b3f98a7a47da02ec27bdb0babfb9ef1a00 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19802) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20461) YARNFileReplicationITCase.testPerJobModeWithDefaultFileReplication
[
https://issues.apache.org/jira/browse/FLINK-20461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373148#comment-17373148
]
Xintong Song commented on FLINK-20461:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19797=logs=8fd975ef-f478-511d-4997-6f15fe8a1fd3=ac0fa443-5d45-5a6b-3597-0310ecc1d2ab=31002
> YARNFileReplicationITCase.testPerJobModeWithDefaultFileReplication
> --
>
> Key: FLINK-20461
> URL: https://issues.apache.org/jira/browse/FLINK-20461
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.3, 1.12.0, 1.13.0, 1.14.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: test-stability
> Fix For: 1.14.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10450=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf]
> {code:java}
> [ERROR]
> testPerJobModeWithDefaultFileReplication(org.apache.flink.yarn.YARNFileReplicationITCase)
> Time elapsed: 32.501 s <<< ERROR! java.io.FileNotFoundException: File does
> not exist:
> hdfs://localhost:46072/user/agent04_azpcontainer/.flink/application_1606950278664_0001/flink-dist_2.11-1.12-SNAPSHOT.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1441)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1434)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1434)
> at
> org.apache.flink.yarn.YARNFileReplicationITCase.extraVerification(YARNFileReplicationITCase.java:148)
> at
> org.apache.flink.yarn.YARNFileReplicationITCase.deployPerJob(YARNFileReplicationITCase.java:113)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-23149) Introduce Java code splitter for code generation
[ https://issues.apache.org/jira/browse/FLINK-23149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-23149. Resolution: Fixed Implements via: master: 392dcb07586ab7a3f42811ac05f1ebffef07fc54 > Introduce Java code splitter for code generation > > > Key: FLINK-23149 > URL: https://issues.apache.org/jira/browse/FLINK-23149 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Caizhi Weng >Assignee: Caizhi Weng >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > > In this first step we introduce a Java code splitter for the generated Java > code. This splitter comes into effect after the original Java code is > generated, hoping to solve the 64KB problem in one shot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] JingsongLi merged pull request #16283: [FLINK-23149][table-code-splitter] Introduce the new Java code splitter module
JingsongLi merged pull request #16283: URL: https://github.com/apache/flink/pull/16283 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #11877: [FLINK-16641][network] Announce sender's backlog to solve the deadlock issue without exclusive buffers
wsry commented on a change in pull request #11877:
URL: https://github.com/apache/flink/pull/11877#discussion_r662689103
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/BufferManager.java
##
@@ -215,9 +221,15 @@ public void recycle(MemorySegment segment) {
}
void releaseFloatingBuffers() {
+Queue buffers;
synchronized (bufferQueue) {
numRequiredBuffers = 0;
-bufferQueue.releaseFloatingBuffers();
+buffers = bufferQueue.clearFloatingBuffers();
+}
+
+// recycle all buffers out of the synchronization block to avoid dead
lock
Review comment:
It is the new logic added by this PR causes the problem.
RemoteInputChannel#onBlockingUpstream will call
BufferManager#releaseFloatingBuffers in netty thread and different
RemoteInputChannels can have different netty thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #11877: [FLINK-16641][network] Announce sender's backlog to solve the deadlock issue without exclusive buffers
wsry commented on a change in pull request #11877:
URL: https://github.com/apache/flink/pull/11877#discussion_r662688120
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartition.java
##
@@ -288,9 +288,7 @@ BufferAndBacklog pollBuffer() {
if (buffers.isEmpty()) {
flushRequested = false;
-}
-
-while (!buffers.isEmpty()) {
+} else {
Review comment:
If there is only one buffer in the buffer queue, after all data is read,
the flushRequested flag will be turned off. Consider the following scenario
(correct me if I am wrong):
1. There is an unfinished buffer in the queue and all data is read.
2. Then the buffer is finished but no new data is appended. Note appending
data and finish buffer is not an atomic operation.
3. New buffer or event is added to the buffer queue.
4. When polling buffer, an empty buffer is at the head of the queue and we
already allocate a credit for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16347: [hotfix][javadocs] Fix broken inline link
flinkbot commented on pull request #16347: URL: https://github.com/apache/flink/pull/16347#issuecomment-872653761 ## CI report: * 6d73e9b3f98a7a47da02ec27bdb0babfb9ef1a00 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22387) UpsertKafkaTableITCase hangs when setting up kafka
[
https://issues.apache.org/jira/browse/FLINK-22387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373141#comment-17373141
]
Xintong Song commented on FLINK-22387:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19796=logs=4be4ed2b-549a-533d-aa33-09e28e360cc8=0db94045-2aa0-53fa-f444-0130d6933518=7817
> UpsertKafkaTableITCase hangs when setting up kafka
> --
>
> Key: FLINK-22387
> URL: https://issues.apache.org/jira/browse/FLINK-22387
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / Ecosystem
>Affects Versions: 1.14.0, 1.13.1, 1.12.4
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: auto-deprioritized-critical, test-stability
> Fix For: 1.14.0, 1.12.5, 1.13.2
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16901=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=6932
> {code}
> 2021-04-20T20:01:32.2276988Z Apr 20 20:01:32 "main" #1 prio=5 os_prio=0
> tid=0x7fe87400b000 nid=0x4028 runnable [0x7fe87df22000]
> 2021-04-20T20:01:32.2277666Z Apr 20 20:01:32java.lang.Thread.State:
> RUNNABLE
> 2021-04-20T20:01:32.2278338Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.Buffer.getByte(Buffer.java:312)
> 2021-04-20T20:01:32.2279325Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:310)
> 2021-04-20T20:01:32.2280656Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:492)
> 2021-04-20T20:01:32.2281603Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471)
> 2021-04-20T20:01:32.2282163Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.skipAll(Util.java:204)
> 2021-04-20T20:01:32.2282870Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.discard(Util.java:186)
> 2021-04-20T20:01:32.2283494Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.close(Http1ExchangeCodec.java:511)
> 2021-04-20T20:01:32.2284460Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.ForwardingSource.close(ForwardingSource.java:43)
> 2021-04-20T20:01:32.2285183Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.close(Exchange.java:313)
> 2021-04-20T20:01:32.2285756Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.RealBufferedSource.close(RealBufferedSource.java:476)
> 2021-04-20T20:01:32.2286287Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.closeQuietly(Util.java:139)
> 2021-04-20T20:01:32.2286795Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.ResponseBody.close(ResponseBody.java:192)
> 2021-04-20T20:01:32.2287270Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.Response.close(Response.java:290)
> 2021-04-20T20:01:32.2287913Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.okhttp.OkDockerHttpClient$OkResponse.close(OkDockerHttpClient.java:285)
> 2021-04-20T20:01:32.2288606Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$null$0(DefaultInvocationBuilder.java:272)
> 2021-04-20T20:01:32.2289295Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$$Lambda$340/2058508175.close(Unknown
> Source)
> 2021-04-20T20:01:32.2289886Z Apr 20 20:01:32 at
> com.github.dockerjava.api.async.ResultCallbackTemplate.close(ResultCallbackTemplate.java:77)
> 2021-04-20T20:01:32.2290567Z Apr 20 20:01:32 at
> org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:202)
> 2021-04-20T20:01:32.2291051Z Apr 20 20:01:32 at
> org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:205)
> 2021-04-20T20:01:32.2291879Z Apr 20 20:01:32 - locked <0xe9cd50f8>
> (a [Ljava.lang.Object;)
> 2021-04-20T20:01:32.2292313Z Apr 20 20:01:32 at
> org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
> 2021-04-20T20:01:32.2292870Z Apr 20 20:01:32 at
> org.testcontainers.LazyDockerClient.authConfig(LazyDockerClient.java:12)
> 2021-04-20T20:01:32.2293383Z Apr 20 20:01:32 at
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:310)
> 2021-04-20T20:01:32.2293890Z Apr 20 20:01:32 at
> org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1029)
> 2021-04-20T20:01:32.2294578Z Apr 20 20:01:32 at
> org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
>
[jira] [Updated] (FLINK-22387) UpsertKafkaTableITCase hangs when setting up kafka
[
https://issues.apache.org/jira/browse/FLINK-22387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-22387:
-
Affects Version/s: 1.14.0
> UpsertKafkaTableITCase hangs when setting up kafka
> --
>
> Key: FLINK-22387
> URL: https://issues.apache.org/jira/browse/FLINK-22387
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / Ecosystem
>Affects Versions: 1.14.0, 1.13.1, 1.12.4
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: auto-deprioritized-critical, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16901=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=6932
> {code}
> 2021-04-20T20:01:32.2276988Z Apr 20 20:01:32 "main" #1 prio=5 os_prio=0
> tid=0x7fe87400b000 nid=0x4028 runnable [0x7fe87df22000]
> 2021-04-20T20:01:32.2277666Z Apr 20 20:01:32java.lang.Thread.State:
> RUNNABLE
> 2021-04-20T20:01:32.2278338Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.Buffer.getByte(Buffer.java:312)
> 2021-04-20T20:01:32.2279325Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:310)
> 2021-04-20T20:01:32.2280656Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:492)
> 2021-04-20T20:01:32.2281603Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471)
> 2021-04-20T20:01:32.2282163Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.skipAll(Util.java:204)
> 2021-04-20T20:01:32.2282870Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.discard(Util.java:186)
> 2021-04-20T20:01:32.2283494Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.close(Http1ExchangeCodec.java:511)
> 2021-04-20T20:01:32.2284460Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.ForwardingSource.close(ForwardingSource.java:43)
> 2021-04-20T20:01:32.2285183Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.close(Exchange.java:313)
> 2021-04-20T20:01:32.2285756Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.RealBufferedSource.close(RealBufferedSource.java:476)
> 2021-04-20T20:01:32.2286287Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.closeQuietly(Util.java:139)
> 2021-04-20T20:01:32.2286795Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.ResponseBody.close(ResponseBody.java:192)
> 2021-04-20T20:01:32.2287270Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.Response.close(Response.java:290)
> 2021-04-20T20:01:32.2287913Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.okhttp.OkDockerHttpClient$OkResponse.close(OkDockerHttpClient.java:285)
> 2021-04-20T20:01:32.2288606Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$null$0(DefaultInvocationBuilder.java:272)
> 2021-04-20T20:01:32.2289295Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$$Lambda$340/2058508175.close(Unknown
> Source)
> 2021-04-20T20:01:32.2289886Z Apr 20 20:01:32 at
> com.github.dockerjava.api.async.ResultCallbackTemplate.close(ResultCallbackTemplate.java:77)
> 2021-04-20T20:01:32.2290567Z Apr 20 20:01:32 at
> org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:202)
> 2021-04-20T20:01:32.2291051Z Apr 20 20:01:32 at
> org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:205)
> 2021-04-20T20:01:32.2291879Z Apr 20 20:01:32 - locked <0xe9cd50f8>
> (a [Ljava.lang.Object;)
> 2021-04-20T20:01:32.2292313Z Apr 20 20:01:32 at
> org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
> 2021-04-20T20:01:32.2292870Z Apr 20 20:01:32 at
> org.testcontainers.LazyDockerClient.authConfig(LazyDockerClient.java:12)
> 2021-04-20T20:01:32.2293383Z Apr 20 20:01:32 at
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:310)
> 2021-04-20T20:01:32.2293890Z Apr 20 20:01:32 at
> org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1029)
> 2021-04-20T20:01:32.2294578Z Apr 20 20:01:32 at
> org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
> 2021-04-20T20:01:32.2295157Z Apr 20 20:01:32 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-04-20T20:01:32.2295622Z Apr 20 20:01:32 at
>
[jira] [Updated] (FLINK-22387) UpsertKafkaTableITCase hangs when setting up kafka
[
https://issues.apache.org/jira/browse/FLINK-22387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-22387:
-
Fix Version/s: 1.13.2
1.12.5
1.14.0
> UpsertKafkaTableITCase hangs when setting up kafka
> --
>
> Key: FLINK-22387
> URL: https://issues.apache.org/jira/browse/FLINK-22387
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / Ecosystem
>Affects Versions: 1.14.0, 1.13.1, 1.12.4
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: auto-deprioritized-critical, test-stability
> Fix For: 1.14.0, 1.12.5, 1.13.2
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16901=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=6932
> {code}
> 2021-04-20T20:01:32.2276988Z Apr 20 20:01:32 "main" #1 prio=5 os_prio=0
> tid=0x7fe87400b000 nid=0x4028 runnable [0x7fe87df22000]
> 2021-04-20T20:01:32.2277666Z Apr 20 20:01:32java.lang.Thread.State:
> RUNNABLE
> 2021-04-20T20:01:32.2278338Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.Buffer.getByte(Buffer.java:312)
> 2021-04-20T20:01:32.2279325Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:310)
> 2021-04-20T20:01:32.2280656Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:492)
> 2021-04-20T20:01:32.2281603Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471)
> 2021-04-20T20:01:32.2282163Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.skipAll(Util.java:204)
> 2021-04-20T20:01:32.2282870Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.discard(Util.java:186)
> 2021-04-20T20:01:32.2283494Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.close(Http1ExchangeCodec.java:511)
> 2021-04-20T20:01:32.2284460Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.ForwardingSource.close(ForwardingSource.java:43)
> 2021-04-20T20:01:32.2285183Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.close(Exchange.java:313)
> 2021-04-20T20:01:32.2285756Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.RealBufferedSource.close(RealBufferedSource.java:476)
> 2021-04-20T20:01:32.2286287Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.closeQuietly(Util.java:139)
> 2021-04-20T20:01:32.2286795Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.ResponseBody.close(ResponseBody.java:192)
> 2021-04-20T20:01:32.2287270Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.Response.close(Response.java:290)
> 2021-04-20T20:01:32.2287913Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.okhttp.OkDockerHttpClient$OkResponse.close(OkDockerHttpClient.java:285)
> 2021-04-20T20:01:32.2288606Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$null$0(DefaultInvocationBuilder.java:272)
> 2021-04-20T20:01:32.2289295Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$$Lambda$340/2058508175.close(Unknown
> Source)
> 2021-04-20T20:01:32.2289886Z Apr 20 20:01:32 at
> com.github.dockerjava.api.async.ResultCallbackTemplate.close(ResultCallbackTemplate.java:77)
> 2021-04-20T20:01:32.2290567Z Apr 20 20:01:32 at
> org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:202)
> 2021-04-20T20:01:32.2291051Z Apr 20 20:01:32 at
> org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:205)
> 2021-04-20T20:01:32.2291879Z Apr 20 20:01:32 - locked <0xe9cd50f8>
> (a [Ljava.lang.Object;)
> 2021-04-20T20:01:32.2292313Z Apr 20 20:01:32 at
> org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
> 2021-04-20T20:01:32.2292870Z Apr 20 20:01:32 at
> org.testcontainers.LazyDockerClient.authConfig(LazyDockerClient.java:12)
> 2021-04-20T20:01:32.2293383Z Apr 20 20:01:32 at
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:310)
> 2021-04-20T20:01:32.2293890Z Apr 20 20:01:32 at
> org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1029)
> 2021-04-20T20:01:32.2294578Z Apr 20 20:01:32 at
> org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
> 2021-04-20T20:01:32.2295157Z Apr 20 20:01:32 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
>
[jira] [Commented] (FLINK-22387) UpsertKafkaTableITCase hangs when setting up kafka
[
https://issues.apache.org/jira/browse/FLINK-22387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373140#comment-17373140
]
Xintong Song commented on FLINK-22387:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19796=logs=72d4811f-9f0d-5fd0-014a-0bc26b72b642=c1d93a6a-ba91-515d-3196-2ee8019fbda7=6543
> UpsertKafkaTableITCase hangs when setting up kafka
> --
>
> Key: FLINK-22387
> URL: https://issues.apache.org/jira/browse/FLINK-22387
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / Ecosystem
>Affects Versions: 1.13.1, 1.12.4
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: auto-deprioritized-critical, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16901=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=6932
> {code}
> 2021-04-20T20:01:32.2276988Z Apr 20 20:01:32 "main" #1 prio=5 os_prio=0
> tid=0x7fe87400b000 nid=0x4028 runnable [0x7fe87df22000]
> 2021-04-20T20:01:32.2277666Z Apr 20 20:01:32java.lang.Thread.State:
> RUNNABLE
> 2021-04-20T20:01:32.2278338Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.Buffer.getByte(Buffer.java:312)
> 2021-04-20T20:01:32.2279325Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:310)
> 2021-04-20T20:01:32.2280656Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:492)
> 2021-04-20T20:01:32.2281603Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471)
> 2021-04-20T20:01:32.2282163Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.skipAll(Util.java:204)
> 2021-04-20T20:01:32.2282870Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.discard(Util.java:186)
> 2021-04-20T20:01:32.2283494Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.close(Http1ExchangeCodec.java:511)
> 2021-04-20T20:01:32.2284460Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.ForwardingSource.close(ForwardingSource.java:43)
> 2021-04-20T20:01:32.2285183Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.close(Exchange.java:313)
> 2021-04-20T20:01:32.2285756Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.RealBufferedSource.close(RealBufferedSource.java:476)
> 2021-04-20T20:01:32.2286287Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.closeQuietly(Util.java:139)
> 2021-04-20T20:01:32.2286795Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.ResponseBody.close(ResponseBody.java:192)
> 2021-04-20T20:01:32.2287270Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.Response.close(Response.java:290)
> 2021-04-20T20:01:32.2287913Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.okhttp.OkDockerHttpClient$OkResponse.close(OkDockerHttpClient.java:285)
> 2021-04-20T20:01:32.2288606Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$null$0(DefaultInvocationBuilder.java:272)
> 2021-04-20T20:01:32.2289295Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$$Lambda$340/2058508175.close(Unknown
> Source)
> 2021-04-20T20:01:32.2289886Z Apr 20 20:01:32 at
> com.github.dockerjava.api.async.ResultCallbackTemplate.close(ResultCallbackTemplate.java:77)
> 2021-04-20T20:01:32.2290567Z Apr 20 20:01:32 at
> org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:202)
> 2021-04-20T20:01:32.2291051Z Apr 20 20:01:32 at
> org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:205)
> 2021-04-20T20:01:32.2291879Z Apr 20 20:01:32 - locked <0xe9cd50f8>
> (a [Ljava.lang.Object;)
> 2021-04-20T20:01:32.2292313Z Apr 20 20:01:32 at
> org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
> 2021-04-20T20:01:32.2292870Z Apr 20 20:01:32 at
> org.testcontainers.LazyDockerClient.authConfig(LazyDockerClient.java:12)
> 2021-04-20T20:01:32.2293383Z Apr 20 20:01:32 at
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:310)
> 2021-04-20T20:01:32.2293890Z Apr 20 20:01:32 at
> org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1029)
> 2021-04-20T20:01:32.2294578Z Apr 20 20:01:32 at
> org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
> 2021-04-20T20:01:32.2295157Z Apr 20 20:01:32 at
>
[jira] [Commented] (FLINK-23196) JobMasterITCase fail on azure due to BindException
[
https://issues.apache.org/jira/browse/FLINK-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373139#comment-17373139
]
Xintong Song commented on FLINK-23196:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19796=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=c2734c79-73b6-521c-e85a-67c7ecae9107=9800
> JobMasterITCase fail on azure due to BindException
> --
>
> Key: FLINK-23196
> URL: https://issues.apache.org/jira/browse/FLINK-23196
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19753=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56=4251
> {code}
> Jul 01 00:00:27 [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 4.272 s <<< FAILURE! - in
> org.apache.flink.runtime.jobmaster.JobMasterITCase
> Jul 01 00:00:27 [ERROR]
> testRejectionOfEmptyJobGraphs(org.apache.flink.runtime.jobmaster.JobMasterITCase)
> Time elapsed: 3.009 s <<< ERROR!
> Jul 01 00:00:27 org.apache.flink.util.FlinkException: Could not create the
> DispatcherResourceManagerComponent.
> Jul 01 00:00:27 at
> org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:275)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.createDispatcherResourceManagerComponents(MiniCluster.java:470)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.setupDispatcherResourceManagerComponents(MiniCluster.java:429)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:373)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.jobmaster.JobMasterITCase.testRejectionOfEmptyJobGraphs(JobMasterITCase.java:56)
> Jul 01 00:00:27 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jul 01 00:00:27 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jul 01 00:00:27 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 01 00:00:27 at java.lang.reflect.Method.invoke(Method.java:498)
> Jul 01 00:00:27 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jul 01 00:00:27 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jul 01 00:00:27 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Jul 01 00:00:27 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jul 01 00:00:27 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Jul 01 00:00:27 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> Jul 01 00:00:27 at
>
[jira] [Commented] (FLINK-23196) JobMasterITCase fail on azure due to BindException
[
https://issues.apache.org/jira/browse/FLINK-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373138#comment-17373138
]
Xintong Song commented on FLINK-23196:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19796=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56=4251
> JobMasterITCase fail on azure due to BindException
> --
>
> Key: FLINK-23196
> URL: https://issues.apache.org/jira/browse/FLINK-23196
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19753=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56=4251
> {code}
> Jul 01 00:00:27 [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 4.272 s <<< FAILURE! - in
> org.apache.flink.runtime.jobmaster.JobMasterITCase
> Jul 01 00:00:27 [ERROR]
> testRejectionOfEmptyJobGraphs(org.apache.flink.runtime.jobmaster.JobMasterITCase)
> Time elapsed: 3.009 s <<< ERROR!
> Jul 01 00:00:27 org.apache.flink.util.FlinkException: Could not create the
> DispatcherResourceManagerComponent.
> Jul 01 00:00:27 at
> org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:275)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.createDispatcherResourceManagerComponents(MiniCluster.java:470)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.setupDispatcherResourceManagerComponents(MiniCluster.java:429)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:373)
> Jul 01 00:00:27 at
> org.apache.flink.runtime.jobmaster.JobMasterITCase.testRejectionOfEmptyJobGraphs(JobMasterITCase.java:56)
> Jul 01 00:00:27 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jul 01 00:00:27 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jul 01 00:00:27 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 01 00:00:27 at java.lang.reflect.Method.invoke(Method.java:498)
> Jul 01 00:00:27 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jul 01 00:00:27 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jul 01 00:00:27 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Jul 01 00:00:27 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jul 01 00:00:27 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Jul 01 00:00:27 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Jul 01 00:00:27 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Jul 01 00:00:27 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> Jul 01 00:00:27 at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> Jul 01 00:00:27 at
>
[GitHub] [flink] flinkbot commented on pull request #16347: [hotfix][javadocs] Fix broken inline link
flinkbot commented on pull request #16347: URL: https://github.com/apache/flink/pull/16347#issuecomment-872648546 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6d73e9b3f98a7a47da02ec27bdb0babfb9ef1a00 (Fri Jul 02 01:29:11 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23198) An old interface method is used in this section of [Passing Options Factory to RocksDB].
[ https://issues.apache.org/jira/browse/FLINK-23198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373136#comment-17373136 ] Carl commented on FLINK-23198: -- [~yunta], Thank you for your reply! I have create two PRs, another Jira link is https://issues.apache.org/jira/browse/FLINK-23211. > An old interface method is used in this section of [Passing Options Factory > to RocksDB]. > > > Key: FLINK-23198 > URL: https://issues.apache.org/jira/browse/FLINK-23198 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.12.0 >Reporter: Carl >Priority: Minor > Attachments: image-2021-07-01-11-30-57-676.png, > image-2021-07-01-11-32-25-200.png > > > [https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/state/state_backends.html] > > !image-2021-07-01-11-30-57-676.png! > In version 1.12 of Flink, this method has been replaced by the following one： > !image-2021-07-01-11-32-25-200.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] rovo98 opened a new pull request #16347: [hotfix][javadocs] Fix broken inline link
rovo98 opened a new pull request #16347: URL: https://github.com/apache/flink/pull/16347 ## What is the purpose of the change To fix broken javadoc inline link. ## Brief change log Fixed javadoc inline link ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22085) KafkaSourceLegacyITCase hangs/fails on azure
[
https://issues.apache.org/jira/browse/FLINK-22085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373128#comment-17373128
]
Dong Lin commented on FLINK-22085:
--
Thanks you [~gaoyunhaii] for the explanation!
Do you plan to submit a PR to fix this? I will help take a deeper look into
this explanation and the PR.
> KafkaSourceLegacyITCase hangs/fails on azure
>
>
> Key: FLINK-22085
> URL: https://issues.apache.org/jira/browse/FLINK-22085
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Dawid Wysakowicz
>Assignee: Yun Gao
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> 1) Observations
> a) The Azure pipeline would occasionally hang without printing any test error
> information.
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15939=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=8219]
> b) By running the test KafkaSourceLegacyITCase::testBrokerFailure() with INFO
> level logging, the the test would hang with the following error message
> printed repeatedly:
> {code:java}
> 20451 [New I/O boss #50] ERROR
> org.apache.flink.networking.NetworkFailureHandler [] - Closing communication
> channel because of an exception
> java.net.ConnectException: Connection refused: localhost/127.0.0.1:50073
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> ~[?:1.8.0_151]
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> ~[?:1.8.0_151]
> at
> org.apache.flink.shaded.testutils.org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
> ~[flink-test-utils_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.shaded.testutils.org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
> [flink-test-utils_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.shaded.testutils.org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
> [flink-test-utils_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.shaded.testutils.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
> [flink-test-utils_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.shaded.testutils.org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
> [flink-test-utils_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.shaded.testutils.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> [flink-test-utils_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.shaded.testutils.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> [flink-test-utils_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_151]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_151]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
> {code}
> *2) Root cause explanations*
> The test would hang because it enters the following loop:
> - closeOnFlush() is called for a given channel
> - closeOnFlush() calls channel.write(..)
> - channel.write() triggers the exceptionCaught(...) callback
> - closeOnFlush() is called for the same channel again.
> *3) Solution*
> Update closeOnFlush() so that, if a channel is being closed by this method,
> then closeOnFlush() would not try to write to this channel if it is called on
> this channel again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23211) An old interface method is used in this section of [Passing Options Factory to RocksDB].
Carl created FLINK-23211: Summary: An old interface method is used in this section of [Passing Options Factory to RocksDB]. Key: FLINK-23211 URL: https://issues.apache.org/jira/browse/FLINK-23211 Project: Flink Issue Type: Bug Components: Documentation Affects Versions: 1.13.0 Reporter: Carl Attachments: image-2021-07-02-09-19-30-547.png, image-2021-07-02-09-20-03-344.png [https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/state/state_backends/] !image-2021-07-02-09-19-30-547.png! *In version 1.13 of Flink, this method has been replaced by the following interface：* !image-2021-07-02-09-20-03-344.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16346: [Hotfix][Docs] Fix variable name in PubSub sample code
flinkbot edited a comment on pull request #16346: URL: https://github.com/apache/flink/pull/16346#issuecomment-872567152 ## CI report: * 80a3d8c549174dbd3fe76c6d5af5d7585241ccd3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19801) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23203) Program cannot parse the parameter value with special characters
[
https://issues.apache.org/jira/browse/FLINK-23203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373118#comment-17373118
]
Jacob.Q.Cao commented on FLINK-23203:
-
I let the job run on yarn，Tested several times and it still remains the same
> Program cannot parse the parameter value with special characters
>
>
> Key: FLINK-23203
> URL: https://issues.apache.org/jira/browse/FLINK-23203
> Project: Flink
> Issue Type: Bug
>Reporter: Jacob.Q.Cao
>Priority: Major
> Attachments: image-2021-07-01-18-56-24-263.png, test.png
>
>
> *When I start Flink Job with a shell command, the program cannot parse the
> parameter value with {color:#ff}special characters{color}*
>
> h3. _Job Shell Command:_
> ./bin/flink run-application -t yarn-application
> -Dyarn.application.name="Test_Flink_Job" -Dtaskmanager.numberOfTaskSlots=1
> -Djobmanager.memory.process.size=10240m
> -Dtaskmanager.memory.process.size=10240m -c com.jacob.main
> /opt/app/Flink/JAR/test.jar *{color:#ff}--test 'test#123'{color}*
>
> Just like the above command, the test parameter value should be "test#123"
> instead of "test"
>
> I have used quotation marks, backslashes, etc. to escape the "#", but it
> still has no effect
>
> !test.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15972: Add common source and operator metrics.
flinkbot edited a comment on pull request #15972: URL: https://github.com/apache/flink/pull/15972#issuecomment-844955772 ## CI report: * 9223b2bfe148ae335891393f244dc2c29f39d2ee UNKNOWN * 1e6b0f37803e0fa5326184c17b3a1f3669667de2 UNKNOWN * 9bf9ec12f659236c7d63f0631c93dc760bbe05cb UNKNOWN * 6ed8f520ee6bdddf9688c234d7ac5d1a93d4c155 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19795) * 93ae7ec71ee7958ef5ec295a6eba871b00c94574 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19800) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-10896) Extend state schema evolution support for more types
[
https://issues.apache.org/jira/browse/FLINK-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-10896:
---
Labels: stale-major (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Extend state schema evolution support for more types
>
>
> Key: FLINK-10896
> URL: https://issues.apache.org/jira/browse/FLINK-10896
> Project: Flink
> Issue Type: New Feature
> Components: API / Type Serialization System
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Major
> Labels: stale-major
>
> Whether or not a state's schema can be evolved (e.g., removing / adding
> fields to a POJO-typed state, modifying an Avro-type state's schema, etc.)
> depends on whether or not the type's corresponding {{TypeSerializer}} and its
> {{TypeSerializerSnapshot}} properly supports it.
> As of Flink 1.7, we currently only have support for evolving Avro types (with
> FLINK-10605).
> This tracks the support for other composite types that would benefit from an
> evolvable schema, such as POJOs, tuples, Scala case classes etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21928) DuplicateJobSubmissionException after JobManager failover
[ https://issues.apache.org/jira/browse/FLINK-21928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-21928: --- Labels: stale-critical (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Critical but is unassigned and neither itself nor its Sub-Tasks have been updated for 7 days. I have gone ahead and marked it "stale-critical". If this ticket is critical, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > DuplicateJobSubmissionException after JobManager failover > - > > Key: FLINK-21928 > URL: https://issues.apache.org/jira/browse/FLINK-21928 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.10.3, 1.11.3, 1.12.2, 1.13.0 > Environment: StandaloneApplicationClusterEntryPoint using a fixed job > ID, High Availability enabled >Reporter: Ufuk Celebi >Priority: Critical > Labels: stale-critical > Fix For: 1.14.0 > > > Consider the following scenario: > * Environment: StandaloneApplicationClusterEntryPoint using a fixed job ID, > high availability enabled > * Flink job reaches a globally terminal state > * Flink job is marked as finished in the high-availability service's > RunningJobsRegistry > * The JobManager fails over > On recovery, the [Dispatcher throws DuplicateJobSubmissionException, because > the job is marked as done in the > RunningJobsRegistry|https://github.com/apache/flink/blob/release-1.12.2/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L332-L340]. > When this happens, users cannot get out of the situation without manually > redeploying the JobManager process and changing the job ID^1^. > The desired semantics are that we don't want to re-execute a job that has > reached a globally terminal state. In this particular case, we know that the > job has already reached such a state (as it has been marked in the registry). > Therefore, we could handle this case by executing the regular termination > sequence instead of throwing a DuplicateJobSubmission. > --- > ^1^ With ZooKeeper HA, the respective node is not ephemeral. In Kubernetes > HA, there is no notion of ephemeral data that is tied to a session in the > first place afaik. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22984) UnsupportedOperationException when using Python UDF to generate watermark
[
https://issues.apache.org/jira/browse/FLINK-22984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22984:
---
Labels: auto-deprioritized-critical (was: stale-critical)
Priority: Major (was: Critical)
This issue was labeled "stale-critical" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Critical,
please raise the priority and ask a committer to assign you the issue or revive
the public discussion.
> UnsupportedOperationException when using Python UDF to generate watermark
> -
>
> Key: FLINK-22984
> URL: https://issues.apache.org/jira/browse/FLINK-22984
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.13.0, 1.13.1
>Reporter: Maciej Bryński
>Priority: Major
> Labels: auto-deprioritized-critical
>
> Hi,
> I'm trying to use output of Python UDF (parse_data) to set watermark for the
> table
> {code:java}
> CREATE TABLE test (
> data BYTES,
> ts as parse_data(data).ts,
> WATERMARK for ts as ts
> ) WITH (
>'connector' = 'kafka',
>'topic' = 'test',
>'properties.bootstrap.servers' = 'localhost:9092',
>'properties.group.id' = 'flink',
>'scan.startup.mode' = 'earliest-offset',
>'format' = 'raw'
> ){code}
> Then running SELECT on this table gives me exception
> {code:java}
> Py4JJavaError: An error occurred while calling o311.hasNext.
> : java.lang.RuntimeException: Failed to fetch next result
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> at
> org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:370)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> at
> org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> at
> org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
> at
> org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> at
> org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
> at
> org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.io.IOException: Failed to fetch job execution result
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.getAccumulatorResults(CollectResultFetcher.java:177)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:120)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> ... 13 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
> at
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.getAccumulatorResults(CollectResultFetcher.java:175)
> ... 15 more
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
> at
> java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:680)
> at
> java.base/java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658)
> at
> java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2094)
> at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.getJobExecutionResult(MiniClusterJobClient.java:134)
> at
>
[jira] [Updated] (FLINK-22949) java.io.InvalidClassException With Flink Kafka Beam
[
https://issues.apache.org/jira/browse/FLINK-22949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22949:
---
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as a
Blocker but is unassigned and neither itself nor its Sub-Tasks have been
updated for 1 days. I have gone ahead and marked it "stale-blocker". If this
ticket is a Blocker, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> java.io.InvalidClassException With Flink Kafka Beam
> ---
>
> Key: FLINK-22949
> URL: https://issues.apache.org/jira/browse/FLINK-22949
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
>Reporter: Ravikiran Borse
>Priority: Blocker
> Labels: stale-blocker
>
> Beam: 2.30.0
> Flink: 1.12.0
> Kafka: 2.6.0
> ERROR:root:java.io.InvalidClassException:
> org.apache.flink.streaming.api.graph.StreamConfig$NetworkInputConfig; local
> class incompatible: stream classdesc serialVersionUID = 3698633776553163849,
> local class serialVersionUID = -3137689219135046939
>
> In Flink Logs
> KafkaIO.Read.ReadFromKafkaViaSDF/{ParDo(GenerateKafkaSourceDescriptor),
> KafkaIO.ReadSourceDescriptors} (1/1)#0 (b0c31371874208adb0ccaff85b971883)
> switched from RUNNING to FAILED.
> org.apache.flink.streaming.runtime.tasks.StreamTaskException: Could not
> deserialize inputs
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getInputs(StreamConfig.java:265)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getTypeSerializerIn(StreamConfig.java:280)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getTypeSerializerIn1(StreamConfig.java:271)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.wrapOperatorIntoOutput(OperatorChain.java:639)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:591)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:526)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:164)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:485)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:531)
> ~[flink-dist_2.12-1.12.0.jar:1.12.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> [flink-dist_2.12-1.12.0.jar:1.12.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> [flink-dist_2.12-1.12.0.jar:1.12.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
> Caused by: java.io.InvalidClassException:
> org.apache.flink.streaming.api.graph.StreamConfig$NetworkInputConfig; local
> class incompatible: stream classdesc serialVersionUID = 3698633776553163849,
> local class serialVersionUID = -3137689219135046939
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21979) Job can be restarted from the beginning after it reached a terminal state
[
https://issues.apache.org/jira/browse/FLINK-21979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-21979:
---
Labels: stale-critical (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 7 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> Job can be restarted from the beginning after it reached a terminal state
> -
>
> Key: FLINK-21979
> URL: https://issues.apache.org/jira/browse/FLINK-21979
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.3, 1.12.2, 1.13.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: stale-critical
> Fix For: 1.14.0
>
>
> Currently, the {{JobMaster}} removes all checkpoints after a job reaches a
> globally terminal state. Then it notifies the {{Dispatcher}} about the
> termination of the job. The {{Dispatcher}} then removes the job from the
> {{SubmittedJobGraphStore}}. If the {{Dispatcher}} process fails before doing
> that it might get restarted. In this case, the {{Dispatcher}} would still
> find the job in the {{SubmittedJobGraphStore}} and recover it. Since the
> {{CompletedCheckpointStore}} is empty, it would start executing this job from
> the beginning.
> I think we must not remove job state before the job has not been marked as
> done or made inaccessible for any restarted processes. Concretely, we should
> first remove the job from the {{SubmittedJobGraphStore}} and only then delete
> the checkpoints. Ideally all the job related cleanup operation happens
> atomically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18810) Golang remote functions SDK
[ https://issues.apache.org/jira/browse/FLINK-18810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-18810: --- Labels: stale-major (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Golang remote functions SDK > --- > > Key: FLINK-18810 > URL: https://issues.apache.org/jira/browse/FLINK-18810 > Project: Flink > Issue Type: New Feature > Components: Stateful Functions >Affects Versions: statefun-3.0.0 >Reporter: Francesco Guardiani >Priority: Major > Labels: stale-major > > Hi, > I was wondering if there's already some WIP for a Golang SDK to create remote > functions. If not, I'm willing to give it a try. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17196) Improve the implementation of Fabric8FlinkKubeClient#getRestEndpoint
[
https://issues.apache.org/jira/browse/FLINK-17196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-17196:
---
Labels: stale-major (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Improve the implementation of Fabric8FlinkKubeClient#getRestEndpoint
>
>
> Key: FLINK-17196
> URL: https://issues.apache.org/jira/browse/FLINK-17196
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0, 1.10.1
>Reporter: Canbin Zheng
>Priority: Major
> Labels: stale-major
>
> Currently there are some bugs in the
> {{Fabric8FlinkKubeClient#getRestEndpoint}} and serveral implicit
> fallback/toleration behaviors when retrieving the Endpoint.
> This ticket proposes to fix the bugs and improve the implementation by
> deprecating some implicit fallback/toleration behaviors.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16634) The PartitionDiscoverer in FlinkKafkaConsumer should not use the user provided client.id.
[
https://issues.apache.org/jira/browse/FLINK-16634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16634:
---
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> The PartitionDiscoverer in FlinkKafkaConsumer should not use the user
> provided client.id.
> -
>
> Key: FLINK-16634
> URL: https://issues.apache.org/jira/browse/FLINK-16634
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.10.0
>Reporter: Jiangjie Qin
>Priority: Major
> Labels: auto-unassigned, stale-major
>
> The {{PartitionDiscoverer}} creates a {{KafkaConsumer}} using the client.id
> from the user provided properties. This may cause the MBean to collide with
> the fetching {{KafkaConsumer}}. The {{PartitionDiscoverer}} should use a
> unique client.id instead, such as "PartitionDiscoverer-RANDOM_LONG"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22816) Investigate feasibility of supporting multiple RM leader sessions within JM process
[ https://issues.apache.org/jira/browse/FLINK-22816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22816: --- Labels: stale-major (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Investigate feasibility of supporting multiple RM leader sessions within JM > process > --- > > Key: FLINK-22816 > URL: https://issues.apache.org/jira/browse/FLINK-22816 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Xintong Song >Priority: Major > Labels: stale-major > > In FLINK-21667, we decoupled RM leadership and lifecycle managements. RM is > not started after obtaining leadership, and stopped on losing leadership. > Ideally, we may start and stop multiple RMs, as the process obtains and loses > leadership. However, as discussed in the > [PR|https://github.com/apache/flink/pull/15524#pullrequestreview-663987547], > having a process to start multiple RMs may cause problems in some deployment > modes. E.g., repeated AM registration is not allowed on Yarn. > We need to investigate for all deployments that: > - Whether having multiple leader sessions causes problems. > - If it does, what can we do to solve the problem. > For information, multi-leader-session support for RM has been implemented in > FLINK-21667, but is disabled by default. To enable, add the system property > "flink.tests.enable-rm-multi-leader-session". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22834) KafkaITCase.testBigRecordJob times out on azure
[
https://issues.apache.org/jira/browse/FLINK-22834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22834:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> KafkaITCase.testBigRecordJob times out on azure
>
>
> Key: FLINK-22834
> URL: https://issues.apache.org/jira/browse/FLINK-22834
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: stale-major, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18481=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=6992
> {code}
> java.lang.InterruptedException
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:49)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runBigRecordTestTopology(KafkaConsumerTestBase.java:1471)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaITCase.testBigRecordJob(KafkaITCase.java:119)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> Jun 01 07:06:39 [ERROR] Tests run: 23, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 214.086 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.KafkaITCase
> Jun 01 07:06:39 [ERROR]
> testBigRecordJob(org.apache.flink.streaming.connectors.kafka.KafkaITCase)
> Time elapsed: 60.015 s <<< ERROR!
> Jun 01 07:06:39 org.junit.runners.model.TestTimedOutException: test timed out
> after 6 milliseconds
> Jun 01 07:06:39 at sun.misc.Unsafe.park(Native Method)
> Jun 01 07:06:39 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> Jun 01 07:06:39 at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> Jun 01 07:06:39 at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> Jun 01 07:06:39 at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> Jun 01 07:06:39 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> Jun 01 07:06:39 at
> org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:49)
> Jun 01 07:06:39 at
> org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runBigRecordTestTopology(KafkaConsumerTestBase.java:1471)
> Jun 01 07:06:39 at
> org.apache.flink.streaming.connectors.kafka.KafkaITCase.testBigRecordJob(KafkaITCase.java:119)
> Jun 01 07:06:39 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jun 01 07:06:39 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jun 01 07:06:39 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jun 01 07:06:39 at java.lang.reflect.Method.invoke(Method.java:498)
> Jun 01 07:06:39 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jun 01 07:06:39 at
>
[jira] [Updated] (FLINK-22449) Casting an invalid constant string to int throws exception from SinkNotNullEnforcer
[
https://issues.apache.org/jira/browse/FLINK-22449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22449:
---
Labels: stale-major (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Casting an invalid constant string to int throws exception from
> SinkNotNullEnforcer
> ---
>
> Key: FLINK-22449
> URL: https://issues.apache.org/jira/browse/FLINK-22449
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.13.0
>Reporter: Caizhi Weng
>Priority: Major
> Labels: stale-major
>
> Add the following test case to {{CalcITCase}} to reproduce this bug:
> {code:scala}
> @Test
> def myTest(): Unit = {
> checkResult("SELECT CAST('haha' AS INT)", Seq(row(null)))
> }
> {code}
> The exception stack is
> {code}
> Caused by: org.apache.flink.table.api.TableException: Column 'EXPR$0' is NOT
> NULL, however, a null value is being written into it. You can set job
> configuration 'table.exec.sink.not-null-enforcer'='drop' to suppress this
> exception and drop such records silently.
> at
> org.apache.flink.table.runtime.operators.sink.SinkNotNullEnforcer.filter(SinkNotNullEnforcer.java:56)
> at
> org.apache.flink.table.runtime.operators.sink.SinkNotNullEnforcer.filter(SinkNotNullEnforcer.java:30)
> at
> org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:38)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.pushToOperator(ChainingOutput.java:112)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:93)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:39)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28)
> at BatchExecCalc$33.processElement(Unknown Source)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.pushToOperator(ChainingOutput.java:112)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:93)
> at
> org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:39)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:317)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:411)
> at
> org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:92)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)
> {code}
> This is because the result type of CAST is inferred as NOT NULL (see
> {{SqlCastFunction#inferReturnType}} and
> {{StandardConvertletTable#convertCast}}, the nullability is the same with the
> input argument), however parsing an invalid string to int will produce null
> values.
> One way I could think of is to change the result type of CAST to always
> nullable (at least for some specific types of casting, for example casting
> from string to int), but as CAST is a very low-level function this might have
> a big impact (for example, if a rule adds casting, the resulting row type
> might not be equal to the original row type due to mismatch in nullability).
> So it seems that at the current stage we should set all columns in a select
> sink to be nullable. However this indicates that one cannot truly rely on the
> nullability of any result type.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22738) org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest no output for 900 seconds due to block on docker container removal
[
https://issues.apache.org/jira/browse/FLINK-22738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22738:
---
Labels: auto-deprioritized-major test-stability (was: stale-major
test-stability)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest no output for
> 900 seconds due to block on docker container removal
>
>
> Key: FLINK-22738
> URL: https://issues.apache.org/jira/browse/FLINK-22738
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Guowei Ma
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18207=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20=13626
> {code:java}
> 2021-05-20T23:12:33.7620808Z May 20 23:12:33 "main" #1 prio=5 os_prio=0
> tid=0x7fa6c400b800 nid=0x45f3 runnable [0x7fa6cb1cd000]
> 2021-05-20T23:12:33.7621514Z May 20 23:12:33java.lang.Thread.State:
> RUNNABLE
> 2021-05-20T23:12:33.7622205Z May 20 23:12:33 at
> org.testcontainers.shaded.okio.Buffer.indexOf(Buffer.java:1463)
> 2021-05-20T23:12:33.7623070Z May 20 23:12:33 at
> org.testcontainers.shaded.okio.RealBufferedSource.indexOf(RealBufferedSource.java:352)
> 2021-05-20T23:12:33.7624186Z May 20 23:12:33 at
> org.testcontainers.shaded.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230)
> 2021-05-20T23:12:33.7625161Z May 20 23:12:33 at
> org.testcontainers.shaded.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:224)
> 2021-05-20T23:12:33.7626488Z May 20 23:12:33 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:489)
> 2021-05-20T23:12:33.7627762Z May 20 23:12:33 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471)
> 2021-05-20T23:12:33.7628864Z May 20 23:12:33 at
> org.testcontainers.shaded.okhttp3.internal.Util.skipAll(Util.java:204)
> 2021-05-20T23:12:33.7629713Z May 20 23:12:33 at
> org.testcontainers.shaded.okhttp3.internal.Util.discard(Util.java:186)
> 2021-05-20T23:12:33.7630720Z May 20 23:12:33 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.close(Http1ExchangeCodec.java:511)
> 2021-05-20T23:12:33.7631677Z May 20 23:12:33 at
> org.testcontainers.shaded.okio.ForwardingSource.close(ForwardingSource.java:43)
> 2021-05-20T23:12:33.7632685Z May 20 23:12:33 at
> org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.close(Exchange.java:313)
> 2021-05-20T23:12:33.7633800Z May 20 23:12:33 at
> org.testcontainers.shaded.okio.RealBufferedSource.close(RealBufferedSource.java:476)
> 2021-05-20T23:12:33.7634690Z May 20 23:12:33 at
> org.testcontainers.shaded.okhttp3.internal.Util.closeQuietly(Util.java:139)
> 2021-05-20T23:12:33.763Z May 20 23:12:33 at
> org.testcontainers.shaded.okhttp3.ResponseBody.close(ResponseBody.java:192)
> 2021-05-20T23:12:33.7636403Z May 20 23:12:33 at
> org.testcontainers.shaded.okhttp3.Response.close(Response.java:290)
> 2021-05-20T23:12:33.7637381Z May 20 23:12:33 at
> org.testcontainers.shaded.com.github.dockerjava.okhttp.OkDockerHttpClient$OkResponse.close(OkDockerHttpClient.java:285)
> 2021-05-20T23:12:33.7638608Z May 20 23:12:33 at
> org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$null$0(DefaultInvocationBuilder.java:272)
> 2021-05-20T23:12:33.7639473Z May 20 23:12:33 at
> org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$$Lambda$83/733374718.close(Unknown
> Source)
> 2021-05-20T23:12:33.7640236Z May 20 23:12:33 at
> com.github.dockerjava.api.async.ResultCallbackTemplate.close(ResultCallbackTemplate.java:77)
> 2021-05-20T23:12:33.7641079Z May 20 23:12:33 at
> org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:202)
> 2021-05-20T23:12:33.7641912Z May 20 23:12:33 at
> org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:205)
> 2021-05-20T23:12:33.7643123Z May 20 23:12:33 - locked <0x991ac238>
> (a [Ljava.lang.Object;)
> 2021-05-20T23:12:33.7644023Z May 20 23:12:33 at
> org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
> 2021-05-20T23:12:33.7644880Z May 20 23:12:33 at
[jira] [Updated] (FLINK-1526) Add Minimum Spanning Tree library method and example
[ https://issues.apache.org/jira/browse/FLINK-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-1526: -- Labels: algorithm auto-deprioritized-major auto-unassigned (was: algorithm auto-unassigned stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Add Minimum Spanning Tree library method and example > > > Key: FLINK-1526 > URL: https://issues.apache.org/jira/browse/FLINK-1526 > Project: Flink > Issue Type: New Feature > Components: Library / Graph Processing (Gelly) >Reporter: Vasia Kalavri >Priority: Minor > Labels: algorithm, auto-deprioritized-major, auto-unassigned > > This issue proposes the addition of a library method and an example for > distributed minimum spanning tree in Gelly. > The DMST algorithm is very interesting because it is quite different from > PageRank-like iterative graph algorithms. It consists of distinct phases > inside the same iteration and requires a mechanism to detect convergence of > one phase to proceed to the next one. Current implementations in > vertex-centric models are quite long (>1000 lines) and hard to understand. > You can find a description of the algorithm [here | > http://ilpubs.stanford.edu:8090/1077/3/p535-salihoglu.pdf] and [here | > http://www.vldb.org/pvldb/vol7/p1047-han.pdf]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22742) Lookup join condition with process time throws org.codehaus.commons.compiler.CompileException
[
https://issues.apache.org/jira/browse/FLINK-22742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22742:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Lookup join condition with process time throws
> org.codehaus.commons.compiler.CompileException
> -
>
> Key: FLINK-22742
> URL: https://issues.apache.org/jira/browse/FLINK-22742
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.12.0, 1.13.0, 1.14.0
>Reporter: Caizhi Weng
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.14.0
>
>
> Add the following test case to
> {{org.apache.flink.table.api.TableEnvironmentITCase}} to reproduce this bug.
> {code:scala}
> @Test
> def myTest(): Unit = {
> val id1 = TestValuesTableFactory.registerData(
> Seq(Row.of("abc", LocalDateTime.of(2000, 1, 1, 0, 0
> val ddl1 =
> s"""
>|CREATE TABLE Ta (
>| id VARCHAR,
>| ts TIMESTAMP,
>| proc AS PROCTIME()
>|) WITH (
>| 'connector' = 'values',
>| 'data-id' = '$id1',
>| 'bounded' = 'true'
>|)
>|""".stripMargin
> tEnv.executeSql(ddl1)
> val id2 = TestValuesTableFactory.registerData(
> Seq(Row.of("abc", LocalDateTime.of(2000, 1, 2, 0, 0
> val ddl2 =
> s"""
>|CREATE TABLE Tb (
>| id VARCHAR,
>| ts TIMESTAMP
>|) WITH (
>| 'connector' = 'values',
>| 'data-id' = '$id2',
>| 'bounded' = 'true'
>|)
>|""".stripMargin
> tEnv.executeSql(ddl2)
> val it = tEnv.executeSql(
> """
> |SELECT * FROM Ta AS t1
> |INNER JOIN Tb FOR SYSTEM_TIME AS OF t1.proc AS t2
> |ON t1.id = t2.id
> |WHERE CAST(coalesce(t1.ts, t2.ts) AS VARCHAR) >=
> CONCAT(DATE_FORMAT(t1.proc, '-MM-dd'), ' 00:00:00')
> |""".stripMargin).collect()
> while (it.hasNext) {
> System.out.println(it.next())
> }
> }
> {code}
> The exception stack is
> {code}
> /* 1 */
> /* 2 */ public class JoinTableFuncCollector$25 extends
> org.apache.flink.table.runtime.collector.TableFunctionCollector {
> /* 3 */
> /* 4 */org.apache.flink.table.data.GenericRowData out = new
> org.apache.flink.table.data.GenericRowData(2);
> /* 5 */org.apache.flink.table.data.utils.JoinedRowData joinedRow$9 = new
> org.apache.flink.table.data.utils.JoinedRowData();
> /* 6 */
> /* 7 */private final org.apache.flink.table.data.binary.BinaryStringData
> str$17 =
> org.apache.flink.table.data.binary.BinaryStringData.fromString("-MM-dd");
> /* 8 */
> /* 9 */private static final java.util.TimeZone timeZone =
> /* 10 */ java.util.TimeZone.getTimeZone("Asia/Shanghai");
> /* 11 */
> /* 12 */private final org.apache.flink.table.data.binary.BinaryStringData
> str$20 = org.apache.flink.table.data.binary.BinaryStringData.fromString("
> 00:00:00");
> /* 13 */
> /* 14 */
> /* 15 */public JoinTableFuncCollector$25(Object[] references) throws
> Exception {
> /* 16 */
> /* 17 */}
> /* 18 */
> /* 19 */@Override
> /* 20 */public void open(org.apache.flink.configuration.Configuration
> parameters) throws Exception {
> /* 21 */
> /* 22 */}
> /* 23 */
> /* 24 */@Override
> /* 25 */public void collect(Object record) throws Exception {
> /* 26 */ org.apache.flink.table.data.RowData in1 =
> (org.apache.flink.table.data.RowData) getInput();
> /* 27 */ org.apache.flink.table.data.RowData in2 =
> (org.apache.flink.table.data.RowData) record;
> /* 28 */
> /* 29 */ org.apache.flink.table.data.binary.BinaryStringData field$7;
> /* 30 */boolean isNull$7;
> /* 31 */org.apache.flink.table.data.TimestampData field$8;
> /* 32 */boolean isNull$8;
> /* 33 */org.apache.flink.table.data.TimestampData field$10;
> /* 34 */boolean isNull$10;
> /* 35 */boolean isNull$13;
> /* 36 */org.apache.flink.table.data.binary.BinaryStringData result$14;
> /* 37 */org.apache.flink.table.data.TimestampData field$15;
> /* 38 */boolean isNull$15;
> /* 39 */org.apache.flink.table.data.TimestampData result$16;
> /* 40 */boolean isNull$18;
> /* 41 */org.apache.flink.table.data.binary.BinaryStringData result$19;
> /* 42 */boolean isNull$21;
> /* 43 */org.apache.flink.table.data.binary.BinaryStringData result$22;
> /* 44 */boolean isNull$23;
> /* 45 */boolean result$24;
[jira] [Updated] (FLINK-22761) Cannot remove POJO fields
[
https://issues.apache.org/jira/browse/FLINK-22761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22761:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Cannot remove POJO fields
> -
>
> Key: FLINK-22761
> URL: https://issues.apache.org/jira/browse/FLINK-22761
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Affects Versions: 1.12.1
>Reporter: Ygor Allan de Fraga
>Priority: Minor
> Labels: auto-deprioritized-major
>
> I tested a schema evolution in a state using POJO and no problem was found
> when trying to add a new field, it was executed just fine. This same field
> was removed from the POJO as it was just a test, but the application could
> not restore the state due to an error.
>
> Here is the error:
> {code:java}
> 2021-05-24 13:05:31,958 WARN org.apache.flink.runtime.taskmanager.Task
> [] - Co-Flat Map -> Map (3/3)#464
> (e0e6d41a18214eab0a1d3c089d8672de) switched from RUNNING to FAILED.2021-05-24
> 13:05:31,958 WARN org.apache.flink.runtime.taskmanager.Task
> [] - Co-Flat Map -> Map (3/3)#464 (e0e6d41a18214eab0a1d3c089d8672de)
> switched from RUNNING to FAILED.java.lang.Exception: Exception while creating
> StreamOperatorStateContext. at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:254)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:272)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:425)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:535)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:525)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:565)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
> [zdata-flink-streams.jar:0.1] at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
> [zdata-flink-streams.jar:0.1] at java.lang.Thread.run(Thread.java:748)
> [?:1.8.0_282]
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for CoStreamFlatMap_b101f370952ea85c2104e98dd54bf7f9_(3/3) from
> any of the 1 provided restore options. at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)
> ~[zdata-flink-streams.jar:0.1] ... 9 more
> Caused by: org.apache.flink.runtime.state.BackendBuildingException: Caught
> unexpected exception. at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:361)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:587)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:93)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:328)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
> ~[zdata-flink-streams.jar:0.1] at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
>
[jira] [Updated] (FLINK-22827) Hive dialect supports CLUSTERED BY clause of CREATE TABLE DDL
[
https://issues.apache.org/jira/browse/FLINK-22827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22827:
---
Labels: stale-major (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Hive dialect supports CLUSTERED BY clause of CREATE TABLE DDL
> -
>
> Key: FLINK-22827
> URL: https://issues.apache.org/jira/browse/FLINK-22827
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Hive
>Affects Versions: 1.13.1
>Reporter: Ma Jun
>Priority: Major
> Labels: stale-major
>
> {code:java}
> # hive syntax:
> CREATE [ EXTERNAL ] TABLE [ IF NOT EXISTS ] table_identifier
> [ ( col_name1[:] col_type1 [ COMMENT col_comment1 ], ... ) ]
> [ COMMENT table_comment ]
> [ PARTITIONED BY ( col_name2[:] col_type2 [ COMMENT col_comment2 ], ... )
> | ( col_name1, col_name2, ... ) ]
> [ CLUSTERED BY ( col_name1, col_name2, ...)
> [ SORTED BY ( col_name1 [ ASC | DESC ], col_name2 [ ASC | DESC ], ...
> ) ]
> INTO num_buckets BUCKETS ]
> [ ROW FORMAT row_format ]
> [ STORED AS file_format ]
> [ LOCATION path ]
> [ TBLPROPERTIES ( key1=val1, key2=val2, ... ) ]
> [ AS select_statement ]
> {code}
>
> {code:java}
> [ CLUSTERED BY ( col_name1, col_name2, ...) [ SORTED BY ( col_name1 [ ASC |
> DESC ], col_name2 [ ASC | DESC ], ... ) ]
> {code}
> Will Flink support the way of creating tables and supporting clustered by |
> sort by into buckets in later versions？
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22326) Job contains Iterate Operator always fails on Checkpoint
[ https://issues.apache.org/jira/browse/FLINK-22326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22326: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 14 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. If the "warning_label" label is not removed in 7 days, the issue will be automatically unassigned. > Job contains Iterate Operator always fails on Checkpoint > - > > Key: FLINK-22326 > URL: https://issues.apache.org/jira/browse/FLINK-22326 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.11.1 >Reporter: Lu Niu >Assignee: Lu Niu >Priority: Major > Labels: pull-request-available, stale-assigned > Attachments: Screen Shot 2021-04-16 at 12.40.34 PM.png, Screen Shot > 2021-04-16 at 12.43.38 PM.png > > > Job contains Iterate Operator will always fail on checkpoint. > How to reproduce: > [https://gist.github.com/qqibrow/f297babadb0bb662ee398b9088870785] > this is based on > [https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/iteration/IterateExample.java,] > but a few line difference: > 1. Make maxWaitTime large enough when create IterativeStream > 2. No output back to Itertive Source > Result: > The same code is able to checkpoint in 1.9.1 > !Screen Shot 2021-04-16 at 12.43.38 PM.png! > > but always fail on checkpoint in 1.11 > !Screen Shot 2021-04-16 at 12.40.34 PM.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16341: [FLINK-21804][state/changelog] Create and wire changelog storage with state backend
flinkbot edited a comment on pull request #16341: URL: https://github.com/apache/flink/pull/16341#issuecomment-872173720 ## CI report: * 01c17423b7f48b807c395ede07d2ee28e23292d3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19793) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16295: [FLINK-11622][chinese-translation,Documentation]Translate the "Command-Line Interface" page into Chinese
flinkbot edited a comment on pull request #16295: URL: https://github.com/apache/flink/pull/16295#issuecomment-868781381 ## CI report: * d15e054e38b45b98c735f9ff19b46e1a5616bc4b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19794) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16346: [Hotfix][Docs] Fix variable name in PubSub sample code
flinkbot edited a comment on pull request #16346: URL: https://github.com/apache/flink/pull/16346#issuecomment-872567152 ## CI report: * 80a3d8c549174dbd3fe76c6d5af5d7585241ccd3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19801) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16346: [Hotfix][Docs] Fix variable name in PubSub sample code
flinkbot commented on pull request #16346: URL: https://github.com/apache/flink/pull/16346#issuecomment-872567152 ## CI report: * 80a3d8c549174dbd3fe76c6d5af5d7585241ccd3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15972: Add common source and operator metrics.
flinkbot edited a comment on pull request #15972: URL: https://github.com/apache/flink/pull/15972#issuecomment-844955772 ## CI report: * de50dd3109f4a0d809a680030be242b9623f3eda Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19791) * 9223b2bfe148ae335891393f244dc2c29f39d2ee UNKNOWN * 1e6b0f37803e0fa5326184c17b3a1f3669667de2 UNKNOWN * 9bf9ec12f659236c7d63f0631c93dc760bbe05cb UNKNOWN * 6ed8f520ee6bdddf9688c234d7ac5d1a93d4c155 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19795) * 93ae7ec71ee7958ef5ec295a6eba871b00c94574 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19800) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15322: [FLINK-21353][state] Add DFS-based StateChangelog (TM-owned state)
flinkbot edited a comment on pull request #15322: URL: https://github.com/apache/flink/pull/15322#issuecomment-804015738 ## CI report: * 5f4747e55bf479d11349f3beba1bdb907f5418eb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15186) * c6ae187f7949bb09b629fa51375dd9becd406b28 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19799) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16346: [Hotfix][Docs] Fix variable name in PubSub sample code
flinkbot commented on pull request #16346: URL: https://github.com/apache/flink/pull/16346#issuecomment-872544421 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 80a3d8c549174dbd3fe76c6d5af5d7585241ccd3 (Thu Jul 01 20:56:13 UTC 2021) **Warnings:** * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] adavis9592 opened a new pull request #16346: [Hotfix][Docs] Fix variable name in PubSub sample code
adavis9592 opened a new pull request #16346: URL: https://github.com/apache/flink/pull/16346 ## What is the purpose of the change To fix sample code so that it actually uses the variable it created. ## Brief change log -Fixed variable names ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-23204) Provide StateBackends access to MailboxExecutor
[ https://issues.apache.org/jira/browse/FLINK-23204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-23204: -- Priority: Minor (was: Major) > Provide StateBackends access to MailboxExecutor > --- > > Key: FLINK-23204 > URL: https://issues.apache.org/jira/browse/FLINK-23204 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends, Runtime / Task >Reporter: Roman Khachatryan >Priority: Minor > Fix For: 1.14.0 > > > StateBackends are assumed to be not-thread-safe and accessed from the task > thread only. > In ChangelogStateBackend, there are (more) async operations. In addition to > the usual methods, task thread is needed for: > * DFS writer: collect so far uploaded changes; handle upload results after > completion > * ChangelogKeyedStateBackend: combining state handles upon upload completion > by writer > * ChangelogKeyedStateBackend: materialization - take snapshot (sync phase); > handle results of the async phase > Direct synchronization can be used instead, but executing ^^^ by the Task > thread would simpilfy the code (and ilkely improve performance). > The only way to do this is via MailboxExecutor (because task thread runs mail > actions in a loop until shutdown). > > However, it is currently created in StreamTask and classes reside in > flink-streaming-java. So one subtask is to change creation/lifecycle and move > the classes. The location is flink-core (at least for interfaces) and > flink-runtime/flink-core (for implementations). > > — > Another subtask is to actually expose Executor to state backends (can be > extracted into a separate task). > StateBackend.createKeyedStateBackend already has Environment/TaskStateManager > argument which can be used. > However, Environment > # is available to the user (via getContainingTask) > # has too wide scope (e.g. InputGates not needed in state backends) > # has too many responsibilities - also true for TaskStateManager which has > e.g. reportIncompleteTaskStateSnapshots > Probably, there is a better way to expose it. > > — > Note that MailboxExecutor will likely be used in future in other places like > ProcessFunction. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23144) Add percentiles to checkpoint stats
[ https://issues.apache.org/jira/browse/FLINK-23144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-23144: -- Priority: Minor (was: Major) > Add percentiles to checkpoint stats > --- > > Key: FLINK-23144 > URL: https://issues.apache.org/jira/browse/FLINK-23144 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / Web Frontend >Reporter: Roman Khachatryan >Priority: Minor > Fix For: 1.14.0 > > > Currently, only min/avg/max are shown, which doesn't allow to easily assess > checkpointing times. > Ideally, with breakdown by operator/channel state write times and sync/async > phases (exact requirements up to implementer) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23143) Support state migration
[ https://issues.apache.org/jira/browse/FLINK-23143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-23143: -- Priority: Minor (was: Major) > Support state migration > --- > > Key: FLINK-23143 > URL: https://issues.apache.org/jira/browse/FLINK-23143 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Priority: Minor > Fix For: 1.14.0 > > > ChangelogKeyedStateBackend.getOrCreateKeyedState is currently used during > recovery; on 1st user access, it doesn't update metadata nor migrate state > (as opposed to other backends). > > The proposed solution is to > # wrap serializers (and maybe other objects) in getOrCreateKeyedState > # store wrapping objects in a new map keyed by state name > # pass wrapped objects to delegatedBackend.createInternalState > # on 1st user access, lookup wrapper and upgrade its wrapped serializer > This should be done for both KV/PQ states. > > See also [https://github.com/apache/flink/pull/15420#discussion_r656934791] > > cc: [~yunta] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16345: [FLINK-18783] Load AkkaRpcSystem through separate classloader
flinkbot edited a comment on pull request #16345: URL: https://github.com/apache/flink/pull/16345#issuecomment-872302449 ## CI report: * 3c74e6208e91e48260fb5d1036680fc40e58a7f5 UNKNOWN * 195daf48ddf067bb949c3465f4381ac675b6f88d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19790) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
