[GitHub] [flink] Airblader removed a comment on pull request #16006: [FLINK-22770][sql-parser][planner-blink] Expose SET/RESET
Airblader removed a comment on pull request #16006: URL: https://github.com/apache/flink/pull/16006#issuecomment-849333982 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #15767: [FLINK-22393][docs-zh] Translate the page of "Execution Mode (Batch/Streaming)" into Chinese
wuchong commented on pull request #15767: URL: https://github.com/apache/flink/pull/15767#issuecomment-849339161 @PatrickRen once you are fine with the translation. I can help to merge this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-21464) Support ADD JAR command in sql client
[ https://issues.apache.org/jira/browse/FLINK-21464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-21464. --- Resolution: Fixed Fixed in master: 34ff7537d3746ca6c6e14742eab0982ebde733b0 > Support ADD JAR command in sql client > - > > Key: FLINK-21464 > URL: https://issues.apache.org/jira/browse/FLINK-21464 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Client >Affects Versions: 1.13.0 >Reporter: Shengkai Fang >Assignee: xiangtao >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.14.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong merged pull request #15925: [FLINK-21464][sql-client] Support ADD JAR in SQL Client
wuchong merged pull request #15925: URL: https://github.com/apache/flink/pull/15925 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Airblader commented on pull request #16006: [FLINK-22770][sql-parser][planner-blink] Expose SET/RESET
Airblader commented on pull request #16006: URL: https://github.com/apache/flink/pull/16006#issuecomment-849333982 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15924: [FLINK-22670][FLIP-150][connector/common] Hybrid source baseline
flinkbot edited a comment on pull request #15924: URL: https://github.com/apache/flink/pull/15924#issuecomment-841943851 ## CI report: * 2e147a5aff7ce35143c42557b651affcd132dbb4 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18356) * 3a01cd27d1fb36e1fa7faf5500bf97617ce22ed2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18358) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] beyond1920 commented on a change in pull request #15997: [FLINK-22680][table-planner-blink] Fix IndexOutOfBoundsException when apply WatermarkAssignerChangelogNormalizeTransposeRule
beyond1920 commented on a change in pull request #15997: URL: https://github.com/apache/flink/pull/15997#discussion_r640272825 ## File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/rules/physical/stream/WatermarkAssignerChangelogNormalizeTransposeRuleTest.xml ## @@ -0,0 +1,154 @@ + + + + + + + + + + + +
[GitHub] [flink] beyond1920 commented on a change in pull request #15997: [FLINK-22680][table-planner-blink] Fix IndexOutOfBoundsException when apply WatermarkAssignerChangelogNormalizeTransposeRule
beyond1920 commented on a change in pull request #15997: URL: https://github.com/apache/flink/pull/15997#discussion_r640272825 ## File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/rules/physical/stream/WatermarkAssignerChangelogNormalizeTransposeRuleTest.xml ## @@ -0,0 +1,154 @@ + + + + + + + + + + + +
[GitHub] [flink] fsk119 commented on pull request #15980: [FLINK-22655]When using -i option to initialize SQL Client session It should be possible to annotate the script with --
fsk119 commented on pull request #15980: URL: https://github.com/apache/flink/pull/15980#issuecomment-849303032 As far as I know, SQL Client can use comments in both mode(-i and -f). The main problem is we can't use comments around the SQL Client commands, e.g. SET/RESET. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15924: [FLINK-22670][FLIP-150][connector/common] Hybrid source baseline
flinkbot edited a comment on pull request #15924: URL: https://github.com/apache/flink/pull/15924#issuecomment-841943851 ## CI report: * 958751f1cbe06cf9163c40e550bc553d6180a77b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18338) * 2e147a5aff7ce35143c42557b651affcd132dbb4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18356) * 3a01cd27d1fb36e1fa7faf5500bf97617ce22ed2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18358) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #14916: [FLINK-21345][Table SQL / Planner] Fix BUG of Union All join Temporal…
godfreyhe commented on a change in pull request #14916:
URL: https://github.com/apache/flink/pull/14916#discussion_r640270303
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/LogicalCorrelateToJoinFromTemporalTableFunctionRule.scala
##
@@ -153,9 +153,8 @@ class LogicalCorrelateToJoinFromTemporalTableFunctionRule
*/
private def getRelOptSchema(relNode: RelNode): RelOptSchema = relNode match {
Review comment:
we should avoid to get `RelOptSchema` from RelNode, another approach is
we can add a method to get `FlinkRelBuilder` directly in `FlinkOptimizeContext`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] beyond1920 commented on a change in pull request #15997: [FLINK-22680][table-planner-blink] Fix IndexOutOfBoundsException when apply WatermarkAssignerChangelogNormalizeTransposeRule
beyond1920 commented on a change in pull request #15997:
URL: https://github.com/apache/flink/pull/15997#discussion_r640267297
##
File path:
flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/plan/rules/physical/stream/WatermarkAssignerChangelogNormalizeTransposeRuleTest.scala
##
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.stream
+
+import org.apache.flink.table.api.ExplainDetail
+import org.apache.flink.table.planner.utils.{StreamTableTestUtil,
TableTestBase}
+
+import org.junit.{Before, Test}
+
+/**
+ * Tests for [[WatermarkAssignerChangelogNormalizeTransposeRule]]
+ */
+class WatermarkAssignerChangelogNormalizeTransposeRuleTest extends
TableTestBase {
+ private val util: StreamTableTestUtil = streamTestUtil()
+
+ @Before
+ def setup(): Unit = {
+util.addTable(
+ s"""
+ |CREATE TABLE simple_src (
+ | currency STRING,
+ | currency_no STRING,
+ | rate BIGINT,
+ | currency_time TIMESTAMP(3),
+ | WATERMARK FOR currency_time AS currency_time - interval '5' SECOND,
+ | PRIMARY KEY(currency) NOT ENFORCED
+ |) WITH (
+ | 'connector' = 'values',
+ | 'changelog-mode' = 'UA,D',
+ | 'enable-watermark-push-down' = 'true'
+ |)
+ |""".stripMargin)
+
+util.addTable(
+ s"""
+ |CREATE TABLE src_with_computed_column (
+ | currency STRING,
+ | currency_no STRING,
+ | rate BIGINT,
+ | c STRING,
+ | currency_time as to_timestamp(c),
+ | WATERMARK FOR currency_time AS currency_time - interval '5' SECOND,
+ | PRIMARY KEY(currency) NOT ENFORCED
+ |) WITH (
+ | 'connector' = 'values',
+ | 'changelog-mode' = 'UA,D',
+ | 'enable-watermark-push-down' = 'true'
+ |)
+ |""".stripMargin)
+ }
+
+ //
+ // Tests for queries matches WITHOUT_CALC patten
+ // Rewrite always happens in the case
+ //
+ @Test
+ def testPushdownWatermarkWithoutCalc(): Unit = {
+val sql =
+ """
+|SELECT
+|currency,
+|COUNT(1) AS cnt,
+|TUMBLE_START(currency_time, INTERVAL '5' SECOND) as w_start,
+|TUMBLE_END(currency_time, INTERVAL '5' SECOND) as w_end
+|FROM simple_src
+|GROUP BY currency, TUMBLE(currency_time, INTERVAL '5' SECOND)
+|""".stripMargin
+util.verifyRelPlan(sql, ExplainDetail.CHANGELOG_MODE)
+ }
+
+ //
+ // Tests for queries matches WITH_CALC patten
+ //
+
+ /** push down calc and watermark assigner as a whole if shuffle keys are
kept after Calc. */
+ @Test
+ def testPushdownCalcAndWatermarkAssignerWithCalc(): Unit = {
+val sql =
+ """
+|SELECT
+|currency,
+|COUNT(1) AS cnt,
+|TUMBLE_START(currency_time, INTERVAL '5' SECOND) as w_start,
+|TUMBLE_END(currency_time, INTERVAL '5' SECOND) as w_end
+|FROM src_with_computed_column
+|GROUP BY currency, TUMBLE(currency_time, INTERVAL '5' SECOND)
+|""".stripMargin
+util.verifyRelPlan(sql, ExplainDetail.CHANGELOG_MODE)
+ }
+
+ /** only push down watermark assigner if satisfy all the following condition:
+ * 1. shuffle keys are not kept after Calc
+ * 2. row time field does not depend on computed column
+ */
+ @Test
+ def testPushdownWatermarkAssignerWithCalc(): Unit = {
+val sql =
+ """
+|SELECT
+|TUMBLE_START(currency_time, INTERVAL '5' SECOND) as w_start,
+|TUMBLE_END(currency_time, INTERVAL '5' SECOND) as w_end,
+|MAX(rate) AS max_rate
+|FROM simple_src
+|GROUP BY TUMBLE(currency_time, INTERVAL '5' SECOND)
+|""".stripMargin
+util.verifyRelPlan(sql,
[GitHub] [flink] beyond1920 commented on a change in pull request #15997: [FLINK-22680][table-planner-blink] Fix IndexOutOfBoundsException when apply WatermarkAssignerChangelogNormalizeTransposeRule
beyond1920 commented on a change in pull request #15997: URL: https://github.com/apache/flink/pull/15997#discussion_r640267152 ## File path: flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/planner/plan/rules/physical/stream/WatermarkAssignerChangelogNormalizeTransposeRuleTest.xml ## @@ -0,0 +1,154 @@ + + + + + + + + + + + +
[GitHub] [flink] 95chenjz commented on pull request #15767: [FLINK-22393][docs-zh] Translate the page of "Execution Mode (Batch/Streaming)" into Chinese
95chenjz commented on pull request #15767: URL: https://github.com/apache/flink/pull/15767#issuecomment-849291080 > Hi @95chenjz , this translation looks good to me, only some typos and trivial problems. I left some comments. > > Also I don't have the permission to merge PRs, you can ping @wuchong after these typos are fixed. > > Thanks again for the contribution! Hi @PatrickRen thanks for your comments. I think it's all set. @wuchong is there anything else I should do? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #15997: [FLINK-22680][table-planner-blink] Fix IndexOutOfBoundsException when apply WatermarkAssignerChangelogNormalizeTransposeRule
godfreyhe commented on a change in pull request #15997:
URL: https://github.com/apache/flink/pull/15997#discussion_r640260862
##
File path:
flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/plan/rules/physical/stream/WatermarkAssignerChangelogNormalizeTransposeRuleTest.scala
##
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.rules.physical.stream
+
+import org.apache.flink.table.api.ExplainDetail
+import org.apache.flink.table.planner.utils.{StreamTableTestUtil,
TableTestBase}
+
+import org.junit.{Before, Test}
+
+/**
+ * Tests for [[WatermarkAssignerChangelogNormalizeTransposeRule]]
+ */
+class WatermarkAssignerChangelogNormalizeTransposeRuleTest extends
TableTestBase {
+ private val util: StreamTableTestUtil = streamTestUtil()
+
+ @Before
+ def setup(): Unit = {
+util.addTable(
+ s"""
+ |CREATE TABLE simple_src (
+ | currency STRING,
+ | currency_no STRING,
+ | rate BIGINT,
+ | currency_time TIMESTAMP(3),
+ | WATERMARK FOR currency_time AS currency_time - interval '5' SECOND,
+ | PRIMARY KEY(currency) NOT ENFORCED
+ |) WITH (
+ | 'connector' = 'values',
+ | 'changelog-mode' = 'UA,D',
+ | 'enable-watermark-push-down' = 'true'
+ |)
+ |""".stripMargin)
+
+util.addTable(
+ s"""
+ |CREATE TABLE src_with_computed_column (
+ | currency STRING,
+ | currency_no STRING,
+ | rate BIGINT,
+ | c STRING,
+ | currency_time as to_timestamp(c),
+ | WATERMARK FOR currency_time AS currency_time - interval '5' SECOND,
+ | PRIMARY KEY(currency) NOT ENFORCED
+ |) WITH (
+ | 'connector' = 'values',
+ | 'changelog-mode' = 'UA,D',
+ | 'enable-watermark-push-down' = 'true'
+ |)
+ |""".stripMargin)
+ }
+
+ //
+ // Tests for queries matches WITHOUT_CALC patten
+ // Rewrite always happens in the case
+ //
+ @Test
+ def testPushdownWatermarkWithoutCalc(): Unit = {
+val sql =
+ """
+|SELECT
+|currency,
+|COUNT(1) AS cnt,
+|TUMBLE_START(currency_time, INTERVAL '5' SECOND) as w_start,
+|TUMBLE_END(currency_time, INTERVAL '5' SECOND) as w_end
+|FROM simple_src
+|GROUP BY currency, TUMBLE(currency_time, INTERVAL '5' SECOND)
+|""".stripMargin
+util.verifyRelPlan(sql, ExplainDetail.CHANGELOG_MODE)
+ }
+
+ //
+ // Tests for queries matches WITH_CALC patten
+ //
+
+ /** push down calc and watermark assigner as a whole if shuffle keys are
kept after Calc. */
+ @Test
+ def testPushdownCalcAndWatermarkAssignerWithCalc(): Unit = {
+val sql =
+ """
+|SELECT
+|currency,
+|COUNT(1) AS cnt,
+|TUMBLE_START(currency_time, INTERVAL '5' SECOND) as w_start,
+|TUMBLE_END(currency_time, INTERVAL '5' SECOND) as w_end
+|FROM src_with_computed_column
+|GROUP BY currency, TUMBLE(currency_time, INTERVAL '5' SECOND)
+|""".stripMargin
+util.verifyRelPlan(sql, ExplainDetail.CHANGELOG_MODE)
+ }
+
+ /** only push down watermark assigner if satisfy all the following condition:
+ * 1. shuffle keys are not kept after Calc
+ * 2. row time field does not depend on computed column
+ */
+ @Test
+ def testPushdownWatermarkAssignerWithCalc(): Unit = {
+val sql =
+ """
+|SELECT
+|TUMBLE_START(currency_time, INTERVAL '5' SECOND) as w_start,
+|TUMBLE_END(currency_time, INTERVAL '5' SECOND) as w_end,
+|MAX(rate) AS max_rate
+|FROM simple_src
+|GROUP BY TUMBLE(currency_time, INTERVAL '5' SECOND)
+|""".stripMargin
+util.verifyRelPlan(sql,
[GitHub] [flink] flinkbot edited a comment on pull request #15924: [FLINK-22670][FLIP-150][connector/common] Hybrid source baseline
flinkbot edited a comment on pull request #15924: URL: https://github.com/apache/flink/pull/15924#issuecomment-841943851 ## CI report: * 958751f1cbe06cf9163c40e550bc553d6180a77b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18338) * 2e147a5aff7ce35143c42557b651affcd132dbb4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18356) * 3a01cd27d1fb36e1fa7faf5500bf97617ce22ed2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JasonLeeCoding commented on pull request #15980: [FLINK-22655]When using -i option to initialize SQL Client session It should be possible to annotate the script with --
JasonLeeCoding commented on pull request #15980: URL: https://github.com/apache/flink/pull/15980#issuecomment-849287222 In fact, I have some doubts that init and file use the same method parseStatement to parse SQL, but file can use comments and init cannot use comments. I think we should solve this problem from the root cause. What do you think ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tweise commented on a change in pull request #15924: [FLINK-22670][FLIP-150][connector/common] Hybrid source baseline
tweise commented on a change in pull request #15924:
URL: https://github.com/apache/flink/pull/15924#discussion_r640256315
##
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/HybridSourceReader.java
##
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base.source.hybrid;
+
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceEvent;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Hybrid source reader that delegates to the actual source reader.
+ *
+ * This reader is setup with a sequence of underlying source readers. At a
given point in time,
+ * one of these readers is active. Underlying readers are opened and closed on
demand as determined
+ * by the enumerator, which selects the active reader via {@link
SwitchSourceEvent}.
+ *
+ * When the underlying reader has consumed all input, {@link
HybridSourceReader} sends {@link
+ * SourceReaderFinishedEvent} to the coordinator and waits for the {@link
SwitchSourceEvent}.
+ */
+public class HybridSourceReader
+implements SourceReader> {
+private static final Logger LOG =
LoggerFactory.getLogger(HybridSourceReader.class);
+// Controls the interval at which to indicate to the coordinator that the
current reader
+// has consumed all input and the next source can be activated. This is
necessary to not flood
+// the coordinator with duplicate events.
+private static final int SOURCE_READER_FINISHED_EVENT_DELAY = 250;
+private SourceReaderContext readerContext;
+private List> realReaders;
+private int currentSourceIndex = -1;
+private long lastCheckpointId = -1;
+private SourceReader currentReader;
+private long lastReaderFinishedMs;
+
+public HybridSourceReader(
+SourceReaderContext readerContext, List>
readers) {
+this.readerContext = readerContext;
+this.realReaders = readers;
+}
+
+@Override
+public void start() {
+setCurrentReader(0);
+}
+
+@Override
+public InputStatus pollNext(ReaderOutput output) throws Exception {
+InputStatus status = currentReader.pollNext(output);
+if (status == InputStatus.END_OF_INPUT) {
+// trap END_OF_INPUT if this wasn't the final reader
+LOG.debug(
Review comment:
I made the change to `NOTHING_AVAILABLE` and that appears to address the
issue. The reader poll will be suspended until the previous future is marked
complete, which occurs upon source switch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15501: [FLINK-22054][k8s] Using a shared watcher for ConfigMap watching
flinkbot edited a comment on pull request #15501: URL: https://github.com/apache/flink/pull/15501#issuecomment-814722337 ## CI report: * 8774a68fe435f2e677a31c7647610c87550fe480 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18310) * abd1e84115fbc9b2026b2f13d4acb4a7c2857c96 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18357) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] beyond1920 commented on pull request #15997: [FLINK-22680][table-planner-blink] Fix IndexOutOfBoundsException when apply WatermarkAssignerChangelogNormalizeTransposeRule
beyond1920 commented on pull request #15997: URL: https://github.com/apache/flink/pull/15997#issuecomment-849271772 cc @godfreyhe -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18229) Pending worker requests should be properly cleared
[
https://issues.apache.org/jira/browse/FLINK-18229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17352240#comment-17352240
]
Nicolas Ferrario commented on FLINK-18229:
--
Hi [~xintongsong], thank you very much! I'll try those workarounds, and
otherwise I'll just wait for 1.14.
> Pending worker requests should be properly cleared
> --
>
> Key: FLINK-18229
> URL: https://issues.apache.org/jira/browse/FLINK-18229
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes, Deployment / YARN, Runtime /
> Coordination
>Affects Versions: 1.9.3, 1.10.1, 1.11.0
>Reporter: Xintong Song
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently, if Kubernetes/Yarn does not have enough resources to fulfill
> Flink's resource requirement, there will be pending pod/container requests on
> Kubernetes/Yarn. These pending resource requirements are never cleared until
> either fulfilled or the Flink cluster is shutdown.
> However, sometimes Flink no longer needs the pending resources. E.g., the
> slot request is then fulfilled by another slots that become available, or the
> job failed due to slot request timeout (in a session cluster). In such cases,
> Flink does not remove the resource request until the resource is allocated,
> then it discovers that it no longer needs the allocated resource and release
> them. This would affect the underlying Kubernetes/Yarn cluster, especially
> when the cluster is under heavy workload.
> It would be good for Flink to cancel pod/container requests as earlier as
> possible if it can discover that some of the pending workers are no longer
> needed.
> There are several approaches potentially achieve this.
> # We can always check whether there's a pending worker that can be canceled
> when a \{{PendingTaskManagerSlot}} is unassigned.
> # We can have a separate timeout for requesting new worker. If the resource
> cannot be allocated within the given time since requested, we should cancel
> that resource request and claim a resource allocation failure.
> # We can share the same timeout for starting new worker (proposed in
> FLINK-13554). This is similar to 2), but it requires the worker to be
> registered, rather than allocated, before timeout.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-18229) Pending worker requests should be properly cleared
[ https://issues.apache.org/jira/browse/FLINK-18229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17352236#comment-17352236 ] Xintong Song edited comment on FLINK-18229 at 5/27/21, 2:19 AM: Hi [~nferrario], Thanks for sharing the information about your problem. AFAIK, this ticket is no longer blocked on any other efforts. We just haven't find enough manpower for this. I'll try to find time for this in the 1.14 release cycle. Meantime, the reason Flink requested 6 more TMs when there's only 1 TM lost is as follow. - There's an idle timeout in JobMaster, that if a slot is not being used for a certain time, it will be released. - Thus, if the job failover takes long, some of the slots might be released by the JobMaster, and be re-requested very soon. - There's a chance that the ResourceManager receives slot requests from JobMaster before learning that the previous slots are released from a TM heartbeat, causing more TMs than needed being requested. To workaround this, you can try the following. - Increase 'slot.idle.timeout'. This reduces the chance that JobMaster releases slots during job failover. The side effect is that, if there are multiple jobs sharing the same Flink session, it might take longer for a slot released by one job to be reused by another. - For Flink 1.12, you can set 'cluster.declarative-resource-management.enabled' to 'true'. That makes JobMaster, instead of requesting individual slots, declare its total resource requirements to ResourceManager at once. That should help avoid requesting more resources during failover. Notice that the above workarounds only apply to the cases where the JobManager process is not crashed. In case of JM pod failure, Flink can still request more than needed resources. Thus, this ticket is still valid. was (Author: xintongsong): Hi [~nferrario], Thanks for sharing the information about your problem. AFAIK, this ticket is no longer blocked on any other efforts. We just haven't find enough manpower for this. I'll try to find time for this in the 1.14 release cycle. Meantime, the reason Flink requested 6 more TMs when there's only 1 TM lost is as follow. - There's an idle timeout in JobMaster, that if a slot is not being used for a certain time, it will be released. - Thus, if the job failover takes long, some of the slots might be released by the JobMaster, and be re-requested very soon. - There's a chance that the ResourceManager receives slot requests from JobMaster before learning that the previous slots are released from a TM heartbeat, causing more TMs than needed being requested. To workaround this, you can try the following. - Increase 'slot.idle.timeout'. This reduce the chance that JobMaster release slots during job failover. The side effect is that, if there are multiple jobs sharing the same Flink session, it might take longer for a slot released by one job being reused by another. - For Flink 1.12, you can set 'cluster.declarative-resource-management.enabled' to 'true'. That makes JobMaster, instead of requesting individual slots, declare its total resource requirements to ResourceManager at once. That should help avoid requesting more resources during failover. Notice that the above workarounds only apply to the cases where the JobManager process is not crashed. In case of JM pod failure, Flink can still request more than needed resources. Thus, this ticket is still valid. > Pending worker requests should be properly cleared > -- > > Key: FLINK-18229 > URL: https://issues.apache.org/jira/browse/FLINK-18229 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes, Deployment / YARN, Runtime / > Coordination >Affects Versions: 1.9.3, 1.10.1, 1.11.0 >Reporter: Xintong Song >Priority: Major > Fix For: 1.14.0 > > > Currently, if Kubernetes/Yarn does not have enough resources to fulfill > Flink's resource requirement, there will be pending pod/container requests on > Kubernetes/Yarn. These pending resource requirements are never cleared until > either fulfilled or the Flink cluster is shutdown. > However, sometimes Flink no longer needs the pending resources. E.g., the > slot request is then fulfilled by another slots that become available, or the > job failed due to slot request timeout (in a session cluster). In such cases, > Flink does not remove the resource request until the resource is allocated, > then it discovers that it no longer needs the allocated resource and release > them. This would affect the underlying Kubernetes/Yarn cluster, especially > when the cluster is under heavy workload. > It would be good for Flink to cancel pod/container requests as earlier as > possible if it can discover that some of the pending workers
[jira] [Comment Edited] (FLINK-18229) Pending worker requests should be properly cleared
[ https://issues.apache.org/jira/browse/FLINK-18229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17352236#comment-17352236 ] Xintong Song edited comment on FLINK-18229 at 5/27/21, 2:18 AM: Hi [~nferrario], Thanks for sharing the information about your problem. AFAIK, this ticket is no longer blocked on any other efforts. We just haven't find enough manpower for this. I'll try to find time for this in the 1.14 release cycle. Meantime, the reason Flink requested 6 more TMs when there's only 1 TM lost is as follow. - There's an idle timeout in JobMaster, that if a slot is not being used for a certain time, it will be released. - Thus, if the job failover takes long, some of the slots might be released by the JobMaster, and be re-requested very soon. - There's a chance that the ResourceManager receives slot requests from JobMaster before learning that the previous slots are released from a TM heartbeat, causing more TMs than needed being requested. To workaround this, you can try the following. - Increase 'slot.idle.timeout'. This reduce the chance that JobMaster release slots during job failover. The side effect is that, if there are multiple jobs sharing the same Flink session, it might take longer for a slot released by one job being reused by another. - For Flink 1.12, you can set 'cluster.declarative-resource-management.enabled' to 'true'. That makes JobMaster, instead of requesting individual slots, declare its total resource requirements to ResourceManager at once. That should help avoid requesting more resources during failover. Notice that the above workarounds only apply to the cases where the JobManager process is not crashed. In case of JM pod failure, Flink can still request more than needed resources. Thus, this ticket is still valid. was (Author: xintongsong): Hi [~nferrario], Thanks for sharing the information about your problem. AFAIK, this ticket is no longer blocked on any other efforts. We just haven't find enough manpower for this. I'll try to find time for this in the 1.14 release cycle. Meantime, the reason Flink requested 6 more TMs when there's only 1 TM lost is as follow. - There's an idle timeout in JobMaster, that if a slot is not being used for a certain time, it will be released. - Thus, if the job failover takes long, some of the slots might be released by the JobMaster, and be re-requested very soon. - There's a chance the the ResourceManager receives slot requests from JobMaster before learning the the previous slots are released from a TM heartbeat, causing more TMs than needed being requested. To workaround this, you can try the following. - Increase 'slot.idle.timeout'. This reduce the chance that JobMaster release slots during job failover. The side effect is that, if there are multiple jobs sharing the same Flink session, it might take longer for a slot released by one job being reused by another. - For Flink 1.12, you can set 'cluster.declarative-resource-management.enabled' to 'true'. That makes JobMaster, instead of requesting individual slots, declare its total resource requirements to ResourceManager at once. That should help avoid requesting more resources during failover. Notice that the above workarounds only apply to the cases where the JobManager process is not crashed. In case of JM pod failure, Flink can still request more than needed resources. Thus, this ticket is still valid. > Pending worker requests should be properly cleared > -- > > Key: FLINK-18229 > URL: https://issues.apache.org/jira/browse/FLINK-18229 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes, Deployment / YARN, Runtime / > Coordination >Affects Versions: 1.9.3, 1.10.1, 1.11.0 >Reporter: Xintong Song >Priority: Major > Fix For: 1.14.0 > > > Currently, if Kubernetes/Yarn does not have enough resources to fulfill > Flink's resource requirement, there will be pending pod/container requests on > Kubernetes/Yarn. These pending resource requirements are never cleared until > either fulfilled or the Flink cluster is shutdown. > However, sometimes Flink no longer needs the pending resources. E.g., the > slot request is then fulfilled by another slots that become available, or the > job failed due to slot request timeout (in a session cluster). In such cases, > Flink does not remove the resource request until the resource is allocated, > then it discovers that it no longer needs the allocated resource and release > them. This would affect the underlying Kubernetes/Yarn cluster, especially > when the cluster is under heavy workload. > It would be good for Flink to cancel pod/container requests as earlier as > possible if it can discover that some of the pending workers are
[GitHub] [flink] flinkbot edited a comment on pull request #15501: [FLINK-22054][k8s] Using a shared watcher for ConfigMap watching
flinkbot edited a comment on pull request #15501: URL: https://github.com/apache/flink/pull/15501#issuecomment-814722337 ## CI report: * 8774a68fe435f2e677a31c7647610c87550fe480 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18310) * abd1e84115fbc9b2026b2f13d4acb4a7c2857c96 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18229) Pending worker requests should be properly cleared
[
https://issues.apache.org/jira/browse/FLINK-18229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17352236#comment-17352236
]
Xintong Song commented on FLINK-18229:
--
Hi [~nferrario],
Thanks for sharing the information about your problem.
AFAIK, this ticket is no longer blocked on any other efforts. We just haven't
find enough manpower for this. I'll try to find time for this in the 1.14
release cycle.
Meantime, the reason Flink requested 6 more TMs when there's only 1 TM lost is
as follow.
- There's an idle timeout in JobMaster, that if a slot is not being used for a
certain time, it will be released.
- Thus, if the job failover takes long, some of the slots might be released by
the JobMaster, and be re-requested very soon.
- There's a chance the the ResourceManager receives slot requests from
JobMaster before learning the the previous slots are released from a TM
heartbeat, causing more TMs than needed being requested.
To workaround this, you can try the following.
- Increase 'slot.idle.timeout'. This reduce the chance that JobMaster release
slots during job failover. The side effect is that, if there are multiple jobs
sharing the same Flink session, it might take longer for a slot released by one
job being reused by another.
- For Flink 1.12, you can set 'cluster.declarative-resource-management.enabled'
to 'true'. That makes JobMaster, instead of requesting individual slots,
declare its total resource requirements to ResourceManager at once. That should
help avoid requesting more resources during failover.
Notice that the above workarounds only apply to the cases where the JobManager
process is not crashed. In case of JM pod failure, Flink can still request more
than needed resources. Thus, this ticket is still valid.
> Pending worker requests should be properly cleared
> --
>
> Key: FLINK-18229
> URL: https://issues.apache.org/jira/browse/FLINK-18229
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes, Deployment / YARN, Runtime /
> Coordination
>Affects Versions: 1.9.3, 1.10.1, 1.11.0
>Reporter: Xintong Song
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently, if Kubernetes/Yarn does not have enough resources to fulfill
> Flink's resource requirement, there will be pending pod/container requests on
> Kubernetes/Yarn. These pending resource requirements are never cleared until
> either fulfilled or the Flink cluster is shutdown.
> However, sometimes Flink no longer needs the pending resources. E.g., the
> slot request is then fulfilled by another slots that become available, or the
> job failed due to slot request timeout (in a session cluster). In such cases,
> Flink does not remove the resource request until the resource is allocated,
> then it discovers that it no longer needs the allocated resource and release
> them. This would affect the underlying Kubernetes/Yarn cluster, especially
> when the cluster is under heavy workload.
> It would be good for Flink to cancel pod/container requests as earlier as
> possible if it can discover that some of the pending workers are no longer
> needed.
> There are several approaches potentially achieve this.
> # We can always check whether there's a pending worker that can be canceled
> when a \{{PendingTaskManagerSlot}} is unassigned.
> # We can have a separate timeout for requesting new worker. If the resource
> cannot be allocated within the given time since requested, we should cancel
> that resource request and claim a resource allocation failure.
> # We can share the same timeout for starting new worker (proposed in
> FLINK-13554). This is similar to 2), but it requires the worker to be
> registered, rather than allocated, before timeout.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] stevenzwu commented on a change in pull request #15924: [FLINK-22670][FLIP-150][connector/common] Hybrid source baseline
stevenzwu commented on a change in pull request #15924:
URL: https://github.com/apache/flink/pull/15924#discussion_r640239284
##
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/HybridSourceReader.java
##
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base.source.hybrid;
+
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceEvent;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Hybrid source reader that delegates to the actual source reader.
+ *
+ * This reader is setup with a sequence of underlying source readers. At a
given point in time,
+ * one of these readers is active. Underlying readers are opened and closed on
demand as determined
+ * by the enumerator, which selects the active reader via {@link
SwitchSourceEvent}.
+ *
+ * When the underlying reader has consumed all input, {@link
HybridSourceReader} sends {@link
+ * SourceReaderFinishedEvent} to the coordinator and waits for the {@link
SwitchSourceEvent}.
+ */
+public class HybridSourceReader
+implements SourceReader> {
+private static final Logger LOG =
LoggerFactory.getLogger(HybridSourceReader.class);
+// Controls the interval at which to indicate to the coordinator that the
current reader
+// has consumed all input and the next source can be activated. This is
necessary to not flood
+// the coordinator with duplicate events.
+private static final int SOURCE_READER_FINISHED_EVENT_DELAY = 250;
+private SourceReaderContext readerContext;
+private List> realReaders;
+private int currentSourceIndex = -1;
+private long lastCheckpointId = -1;
+private SourceReader currentReader;
+private long lastReaderFinishedMs;
+
+public HybridSourceReader(
+SourceReaderContext readerContext, List>
readers) {
+this.readerContext = readerContext;
+this.realReaders = readers;
+}
+
+@Override
+public void start() {
+setCurrentReader(0);
+}
+
+@Override
+public InputStatus pollNext(ReaderOutput output) throws Exception {
+InputStatus status = currentReader.pollNext(output);
+if (status == InputStatus.END_OF_INPUT) {
+// trap END_OF_INPUT if this wasn't the final reader
+LOG.debug(
Review comment:
I see. I misunderstood earlier. Yeah, `MORE_AVAILABLE` will cause the
tight loop. `NOTHING_AVAILABLE` is probably the right choice. But I am not sure
how often the mailbox thread polls it with `NOTHING_AVAILABLE` status.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] stevenzwu commented on a change in pull request #15924: [FLINK-22670][FLIP-150][connector/common] Hybrid source baseline
stevenzwu commented on a change in pull request #15924:
URL: https://github.com/apache/flink/pull/15924#discussion_r640239284
##
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/HybridSourceReader.java
##
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base.source.hybrid;
+
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceEvent;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Hybrid source reader that delegates to the actual source reader.
+ *
+ * This reader is setup with a sequence of underlying source readers. At a
given point in time,
+ * one of these readers is active. Underlying readers are opened and closed on
demand as determined
+ * by the enumerator, which selects the active reader via {@link
SwitchSourceEvent}.
+ *
+ * When the underlying reader has consumed all input, {@link
HybridSourceReader} sends {@link
+ * SourceReaderFinishedEvent} to the coordinator and waits for the {@link
SwitchSourceEvent}.
+ */
+public class HybridSourceReader
+implements SourceReader> {
+private static final Logger LOG =
LoggerFactory.getLogger(HybridSourceReader.class);
+// Controls the interval at which to indicate to the coordinator that the
current reader
+// has consumed all input and the next source can be activated. This is
necessary to not flood
+// the coordinator with duplicate events.
+private static final int SOURCE_READER_FINISHED_EVENT_DELAY = 250;
+private SourceReaderContext readerContext;
+private List> realReaders;
+private int currentSourceIndex = -1;
+private long lastCheckpointId = -1;
+private SourceReader currentReader;
+private long lastReaderFinishedMs;
+
+public HybridSourceReader(
+SourceReaderContext readerContext, List>
readers) {
+this.readerContext = readerContext;
+this.realReaders = readers;
+}
+
+@Override
+public void start() {
+setCurrentReader(0);
+}
+
+@Override
+public InputStatus pollNext(ReaderOutput output) throws Exception {
+InputStatus status = currentReader.pollNext(output);
+if (status == InputStatus.END_OF_INPUT) {
+// trap END_OF_INPUT if this wasn't the final reader
+LOG.debug(
Review comment:
I see. I misunderstood earlier. Yeah, `MORE_AVAILABLE` will cause the
tight loop. `NOTHING_AVAILABLE` is probably the right choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-19973) 【Flink-Deployment】YARN CLI Parameter doesn't work when set `execution.target: yarn-per-job` config
[ https://issues.apache.org/jira/browse/FLINK-19973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17352234#comment-17352234 ] Shengkai Fang commented on FLINK-19973: --- [~twalthr] Sure. I will take a look recently. > 【Flink-Deployment】YARN CLI Parameter doesn't work when set `execution.target: > yarn-per-job` config > -- > > Key: FLINK-19973 > URL: https://issues.apache.org/jira/browse/FLINK-19973 > Project: Flink > Issue Type: Bug > Components: Command Line Client, Deployment / YARN >Affects Versions: 1.12.0 >Reporter: zhisheng >Assignee: Kostas Kloudas >Priority: Major > Attachments: image-2020-11-04-20-58-49-738.png, > image-2020-11-04-21-00-06-180.png > > > when i use flink-sql-client to deploy job to yarn(per job mod), I set > `execution.target: yarn-per-job` in flink-conf.yaml, job will deploy to yarn. > > when I deploy jar job to yarn, The command is `./bin/flink run -m > yarn-cluster -ynm flink-1.12-test -ytm 3g -yjm 3g > examples/streaming/StateMachineExample.jar`, it will deploy ok, but the > `-ynm`、`-ytm 3g` and `-yjm 3g` doesn't work. > > !image-2020-11-04-20-58-49-738.png|width=912,height=235! > > > when i remove the config `execution.target: yarn-per-job`, it work well. > > !image-2020-11-04-21-00-06-180.png|width=1047,height=150! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] yittg commented on pull request #15501: [FLINK-22054][k8s] Using a shared watcher for ConfigMap watching
yittg commented on pull request #15501: URL: https://github.com/apache/flink/pull/15501#issuecomment-849258434 Hi @wangyang0918 , i've added the comment why we use Long.MAX_VALUE. Furthermore, i add a listener to deal with the exception thrown by the informer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22627) Remove SlotManagerImpl
[
https://issues.apache.org/jira/browse/FLINK-22627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yangze Guo updated FLINK-22627:
---
Labels: pull-request-available (was: pull-request-available stale-assigned)
> Remove SlotManagerImpl
> --
>
> Key: FLINK-22627
> URL: https://issues.apache.org/jira/browse/FLINK-22627
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Yangze Guo
>Assignee: Yangze Guo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0
>
>
> As the declarative resource management is completed (FLINK-10404) and the old
> {{SlotPoolImpl}} is removed in FLINK-22477, it's time to remove the
> {{SlotManagerImpl}} and
> all related classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] Samuel-Fiddis commented on pull request #15808: [FLINK-21229] add confluent schema registry ssl support
Samuel-Fiddis commented on pull request #15808:
URL: https://github.com/apache/flink/pull/15808#issuecomment-849248037
I've updated and compiled the source code appending the following to the
CachedSchemaCoderProvider class with the SchemaCoder get() method:
import io.confluent.kafka.schemaregistry.client.SchemaRegistryClientConfig;
import
io.confluent.kafka.schemaregistry.client.security.basicauth.UserInfoCredentialProvider;
@Override
public SchemaCoder get() {
RestService restService = new RestService(url);
if(registryConfigs != null &&
Objects.equals(registryConfigs.get(SchemaRegistryClientConfig.BASIC_AUTH_CREDENTIALS_SOURCE),
"USER_INFO")) {
UserInfoCredentialProvider provider = new
UserInfoCredentialProvider();
provider.configure(registryConfigs);
restService.setBasicAuthCredentialProvider(provider);
}
}
But it doesn't look like the basic auth properties are passed from the Table
API through to the registryConfigs map. When I've hard coded the properties
into the registryConfigs everything works fine and the confluent schema
registry works with basic auth access.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15924: [FLINK-22670][FLIP-150][connector/common] Hybrid source baseline
flinkbot edited a comment on pull request #15924: URL: https://github.com/apache/flink/pull/15924#issuecomment-841943851 ## CI report: * 958751f1cbe06cf9163c40e550bc553d6180a77b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18338) * 2e147a5aff7ce35143c42557b651affcd132dbb4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18356) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tweise commented on a change in pull request #15924: [FLINK-22670][FLIP-150][connector/common] Hybrid source baseline
tweise commented on a change in pull request #15924:
URL: https://github.com/apache/flink/pull/15924#discussion_r640180553
##
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/HybridSourceReader.java
##
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base.source.hybrid;
+
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceEvent;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Hybrid source reader that delegates to the actual source reader.
+ *
+ * This reader is setup with a sequence of underlying source readers. At a
given point in time,
+ * one of these readers is active. Underlying readers are opened and closed on
demand as determined
+ * by the enumerator, which selects the active reader via {@link
SwitchSourceEvent}.
+ *
+ * When the underlying reader has consumed all input, {@link
HybridSourceReader} sends {@link
+ * SourceReaderFinishedEvent} to the coordinator and waits for the {@link
SwitchSourceEvent}.
+ */
+public class HybridSourceReader
+implements SourceReader> {
+private static final Logger LOG =
LoggerFactory.getLogger(HybridSourceReader.class);
+// Controls the interval at which to indicate to the coordinator that the
current reader
+// has consumed all input and the next source can be activated. This is
necessary to not flood
+// the coordinator with duplicate events.
+private static final int SOURCE_READER_FINISHED_EVENT_DELAY = 250;
+private SourceReaderContext readerContext;
+private List> realReaders;
+private int currentSourceIndex = -1;
+private long lastCheckpointId = -1;
+private SourceReader currentReader;
+private long lastReaderFinishedMs;
+
+public HybridSourceReader(
+SourceReaderContext readerContext, List>
readers) {
+this.readerContext = readerContext;
+this.realReaders = readers;
+}
+
+@Override
+public void start() {
+setCurrentReader(0);
+}
+
+@Override
+public InputStatus pollNext(ReaderOutput output) throws Exception {
+InputStatus status = currentReader.pollNext(output);
+if (status == InputStatus.END_OF_INPUT) {
+// trap END_OF_INPUT if this wasn't the final reader
+LOG.debug(
Review comment:
It's related to the jitter question. Please take a look at the comment
that I added. The important part is that this repeats in a tight loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-22017) Regions may never be scheduled when there are cross-region blocking edges
[
https://issues.apache.org/jira/browse/FLINK-22017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22017:
---
Labels: auto-deprioritized-critical (was: stale-critical)
Priority: Major (was: Critical)
This issue was labeled "stale-critical" 7 ago and has not received any updates
so it is being deprioritized. If this ticket is actually Critical, please raise
the priority and ask a committer to assign you the issue or revive the public
discussion.
> Regions may never be scheduled when there are cross-region blocking edges
> -
>
> Key: FLINK-22017
> URL: https://issues.apache.org/jira/browse/FLINK-22017
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.3, 1.12.2, 1.13.0
>Reporter: Zhilong Hong
>Priority: Major
> Labels: auto-deprioritized-critical
> Attachments: Illustration.jpg
>
>
> For the topology with cross-region blocking edges, there are regions that may
> never be scheduled. The case is illustrated in the figure below.
> !Illustration.jpg!
> Let's denote the vertices with layer_number. It's clear that the edge
> connects v2_2 and v3_2 crosses region 1 and region 2. Since region 1 has no
> blocking edges connected to other regions, it will be scheduled first. When
> vertex2_2 is finished, PipelinedRegionSchedulingStrategy will trigger
> {{onExecutionStateChange}} for it.
> As expected, region 2 will be scheduled since all its consumed partitions are
> consumable. But in fact region 2 won't be scheduled, because the result
> partition of vertex2_2 is not tagged as consumable. Whether it is consumable
> or not is determined by its IntermediateDataSet.
> However, an IntermediateDataSet is consumable if and only if all the
> producers of its IntermediateResultPartitions are finished. This
> IntermediateDataSet will never be consumable since vertex2_3 is not
> scheduled. All in all, this forms a deadlock that a region will never be
> scheduled because it's not scheduled.
> As a solution we should let BLOCKING result partitions be consumable
> individually. Note that this will result in the scheduling to become
> execution-vertex-wise instead of stage-wise, with a nice side effect towards
> better resource utilization. The PipelinedRegionSchedulingStrategy can be
> simplified along with change to get rid of the correlatedResultPartitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tweise commented on a change in pull request #15924: [FLINK-22670][FLIP-150][connector/common] Hybrid source baseline
tweise commented on a change in pull request #15924:
URL: https://github.com/apache/flink/pull/15924#discussion_r640179753
##
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/HybridSourceReader.java
##
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base.source.hybrid;
+
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceEvent;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Hybrid source reader that delegates to the actual source reader.
+ *
+ * This reader is setup with a sequence of underlying source readers. At a
given point in time,
+ * one of these readers is active. Underlying readers are opened and closed on
demand as determined
+ * by the enumerator, which selects the active reader via {@link
SwitchSourceEvent}.
+ *
+ * When the underlying reader has consumed all input, {@link
HybridSourceReader} sends {@link
+ * SourceReaderFinishedEvent} to the coordinator and waits for the {@link
SwitchSourceEvent}.
+ */
+public class HybridSourceReader
+implements SourceReader> {
Review comment:
Agreed, using the common SplitT is confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-21025) SQLClientHBaseITCase fails when untarring HBase
[
https://issues.apache.org/jira/browse/FLINK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-21025:
---
Labels: auto-deprioritized-critical test-stability (was: stale-critical
test-stability)
Priority: Major (was: Critical)
This issue was labeled "stale-critical" 7 ago and has not received any updates
so it is being deprioritized. If this ticket is actually Critical, please raise
the priority and ask a committer to assign you the issue or revive the public
discussion.
> SQLClientHBaseITCase fails when untarring HBase
> ---
>
> Key: FLINK-21025
> URL: https://issues.apache.org/jira/browse/FLINK-21025
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Ecosystem, Tests
>Affects Versions: 1.12.2, 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: auto-deprioritized-critical, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12210=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 908.614 s <<< FAILURE! - in
> org.apache.flink.tests.util.hbase.SQLClientHBaseITCase
> Jan 19 08:19:36 [ERROR] testHBase[1:
> hbase-version:2.2.3](org.apache.flink.tests.util.hbase.SQLClientHBaseITCase)
> Time elapsed: 615.099 s <<< ERROR!
> Jan 19 08:19:36 java.io.IOException:
> Jan 19 08:19:36 Process execution failed due error. Error output:
> Jan 19 08:19:36 at
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:133)
> Jan 19 08:19:36 at
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:108)
> Jan 19 08:19:36 at
> org.apache.flink.tests.util.AutoClosableProcess.runBlocking(AutoClosableProcess.java:70)
> Jan 19 08:19:36 at
> org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource.setupHBaseDist(LocalStandaloneHBaseResource.java:86)
> Jan 19 08:19:36 at
> org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource.before(LocalStandaloneHBaseResource.java:76)
> Jan 19 08:19:36 at
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:46)
> Jan 19 08:19:36 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> Jan 19 08:19:36 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> Jan 19 08:19:36 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> Jan 19 08:19:36 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> Jan 19 08:19:36 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> Jan 19 08:19:36 at org.junit.runners.Suite.runChild(Suite.java:128)
> Jan 19 08:19:36 at org.junit.runners.Suite.runChild(Suite.java:27)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> Jan 19 08:19:36 at
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
> Jan 19 08:19:36 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> Jan 19 08:19:36 at org.junit.runners.Suite.runChild(Suite.java:128)
> Jan 19 08:19:36 at org.junit.runners.Suite.runChild(Suite.java:27)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> Jan 19 08:19:36 at
>
[jira] [Updated] (FLINK-22069) Check Log Pollution for 1.14 release
[ https://issues.apache.org/jira/browse/FLINK-22069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22069: --- Labels: auto-deprioritized-critical pull-request-available (was: pull-request-available stale-critical) Priority: Major (was: Critical) This issue was labeled "stale-critical" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Critical, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Check Log Pollution for 1.14 release > > > Key: FLINK-22069 > URL: https://issues.apache.org/jira/browse/FLINK-22069 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Reporter: Stephan Ewen >Priority: Major > Labels: auto-deprioritized-critical, pull-request-available > Fix For: 1.14.0 > > > We should check for log pollution and confusing log lines before the release. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22483) Recover checkpoints when JobMaster gains leadership
[
https://issues.apache.org/jira/browse/FLINK-22483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22483:
---
Labels: stale-critical (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 7 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> Recover checkpoints when JobMaster gains leadership
> ---
>
> Key: FLINK-22483
> URL: https://issues.apache.org/jira/browse/FLINK-22483
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: stale-critical
> Fix For: 1.14.0
>
>
> Recovering checkpoints (from the CompletedCheckpointStore) is a potentially
> long-lasting/blocking operation, for example if the file system
> implementation is retrying to connect to a unavailable storage backend.
> Currently, we are calling the CompletedCheckpointStore.recover() method from
> the main thread of the JobManager, making it unresponsive to any RPC call
> while the recover method is blocked:
> {code}
> 2021-04-02 20:33:31,384 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job XXX
> switched from state RUNNING to RESTARTING.
> com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to
> minio.minio.svc:9000 [minio.minio.svc/] failed: Connection refused
> (Connection refused)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
> ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
> ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
> ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
> ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
> ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
> ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
> ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) ~[?:?]
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) ~[?:?]
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062)
> ~[?:?]
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008)
> ~[?:?]
> at
> com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1490)
> ~[?:?]
> at
> com.facebook.presto.hive.s3.PrestoS3FileSystem$PrestoS3InputStream.lambda$openStream$1(PrestoS3FileSystem.java:905)
> ~[?:?]
> at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138) ~[?:?]
> at
> com.facebook.presto.hive.s3.PrestoS3FileSystem$PrestoS3InputStream.openStream(PrestoS3FileSystem.java:902)
> ~[?:?]
> at
> com.facebook.presto.hive.s3.PrestoS3FileSystem$PrestoS3InputStream.openStream(PrestoS3FileSystem.java:887)
> ~[?:?]
> at
> com.facebook.presto.hive.s3.PrestoS3FileSystem$PrestoS3InputStream.seekStream(PrestoS3FileSystem.java:880)
> ~[?:?]
> at
> com.facebook.presto.hive.s3.PrestoS3FileSystem$PrestoS3InputStream.lambda$read$0(PrestoS3FileSystem.java:819)
> ~[?:?]
> at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138) ~[?:?]
> at
> com.facebook.presto.hive.s3.PrestoS3FileSystem$PrestoS3InputStream.read(PrestoS3FileSystem.java:818)
> ~[?:?]
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> ~[?:1.8.0_282]
> at XXX.recover(KubernetesHaCheckpointStore.java:69)
> ~[vvp-flink-ha-kubernetes-flink112-1.1.0.jar:?]
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreLatestCheckpointedStateInternal(CheckpointCoordinator.java:1511)
> ~[flink-dist_2.12-1.12.2-stream1.jar:1.12.2-stream1]
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreLatestCheckpointedStateToAll(CheckpointCoordinator.java:1451)
> ~[flink-dist_2.12-1.12.2-stream1.jar:1.12.2-stream1]
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.restoreState(SchedulerBase.java:421)
> ~[flink-dist_2.12-1.12.2-stream1.jar:1.12.2-stream1]
> at
>
[jira] [Updated] (FLINK-22416) UpsertKafkaTableITCase hangs when collecting results
[
https://issues.apache.org/jira/browse/FLINK-22416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22416:
---
Labels: auto-deprioritized-critical test-stability (was: stale-critical
test-stability)
Priority: Major (was: Critical)
This issue was labeled "stale-critical" 7 ago and has not received any updates
so it is being deprioritized. If this ticket is actually Critical, please raise
the priority and ask a committer to assign you the issue or revive the public
discussion.
> UpsertKafkaTableITCase hangs when collecting results
>
>
> Key: FLINK-22416
> URL: https://issues.apache.org/jira/browse/FLINK-22416
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / Ecosystem
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: auto-deprioritized-critical, test-stability
> Fix For: 1.14.0
>
> Attachments: idea-test.png, threads_report.txt
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17037=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=7002
> {code}
> 2021-04-22T11:16:35.6812919Z Apr 22 11:16:35 [ERROR]
> testSourceSinkWithKeyAndPartialValue[format =
> csv](org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase)
> Time elapsed: 30.01 s <<< ERROR!
> 2021-04-22T11:16:35.6814151Z Apr 22 11:16:35
> org.junit.runners.model.TestTimedOutException: test timed out after 30 seconds
> 2021-04-22T11:16:35.6814781Z Apr 22 11:16:35 at
> java.lang.Thread.sleep(Native Method)
> 2021-04-22T11:16:35.6815444Z Apr 22 11:16:35 at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:237)
> 2021-04-22T11:16:35.6816250Z Apr 22 11:16:35 at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:113)
> 2021-04-22T11:16:35.6817033Z Apr 22 11:16:35 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> 2021-04-22T11:16:35.6817719Z Apr 22 11:16:35 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> 2021-04-22T11:16:35.6818351Z Apr 22 11:16:35 at
> org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:370)
> 2021-04-22T11:16:35.6818980Z Apr 22 11:16:35 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestUtils.collectRows(KafkaTableTestUtils.java:52)
> 2021-04-22T11:16:35.6819978Z Apr 22 11:16:35 at
> org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase.testSourceSinkWithKeyAndPartialValue(UpsertKafkaTableITCase.java:147)
> 2021-04-22T11:16:35.6820803Z Apr 22 11:16:35 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-04-22T11:16:35.6821365Z Apr 22 11:16:35 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-04-22T11:16:35.6822072Z Apr 22 11:16:35 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-04-22T11:16:35.6822656Z Apr 22 11:16:35 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-04-22T11:16:35.6823124Z Apr 22 11:16:35 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-04-22T11:16:35.6823672Z Apr 22 11:16:35 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-04-22T11:16:35.6824202Z Apr 22 11:16:35 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-04-22T11:16:35.6824709Z Apr 22 11:16:35 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-04-22T11:16:35.6825230Z Apr 22 11:16:35 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-04-22T11:16:35.6825716Z Apr 22 11:16:35 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2021-04-22T11:16:35.6826204Z Apr 22 11:16:35 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-04-22T11:16:35.6826807Z Apr 22 11:16:35 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> 2021-04-22T11:16:35.6827378Z Apr 22 11:16:35 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> 2021-04-22T11:16:35.6827926Z Apr 22 11:16:35 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2021-04-22T11:16:35.6828331Z Apr 22 11:16:35 at
> java.lang.Thread.run(Thread.java:748)
> 2021-04-22T11:16:35.6828600Z Apr 22 11:16:35
>
[jira] [Updated] (FLINK-22459) FlinkKafkaProducerITCase testScaleUpAfterScalingDown failed because of Timeout expired after 60000milliseconds while awaiting EndTxn(COMMIT)
[
https://issues.apache.org/jira/browse/FLINK-22459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22459:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> FlinkKafkaProducerITCase testScaleUpAfterScalingDown failed because of
> Timeout expired after 6milliseconds while awaiting EndTxn(COMMIT)
>
>
> Key: FLINK-22459
> URL: https://issues.apache.org/jira/browse/FLINK-22459
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.2
>Reporter: Guowei Ma
>Priority: Major
> Labels: stale-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17180=logs=1fc6e7bf-633c-5081-c32a-9dea24b05730=80a658d1-f7f6-5d93-2758-53ac19fd5b19=6694
> {code:java}
> [ERROR] Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 261.699 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase
> [ERROR]
> testScaleUpAfterScalingDown(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
> Time elapsed: 67.768 s <<< ERROR!
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 6milliseconds while awaiting EndTxn(COMMIT)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] FlinkKafkaProducerITCase.testScaleUpAfterScalingDown » Timeout
> Timeout expired...
> [INFO]
> [ERROR] Tests run: 136, Failures: 0, Errors: 1, Skipped: 0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22414) Consider env.hadoop.conf.dir when looking for the hadoop configuration
[
https://issues.apache.org/jira/browse/FLINK-22414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22414:
---
Labels: stale-major (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Consider env.hadoop.conf.dir when looking for the hadoop configuration
> --
>
> Key: FLINK-22414
> URL: https://issues.apache.org/jira/browse/FLINK-22414
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Runtime / Configuration
>Affects Versions: 1.11.3, 1.12.2, 1.13.0, 1.14.0
>Reporter: Matthias
>Priority: Major
> Labels: stale-major
>
> The option of setting {{env.hadoop.conf.dir}} is allowed to be used as an
> alternative to setting the environment variable {{HADOOP_CONF_DIR}} based [on
> the
> documentation|https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#env-hadoop-conf-dir].
> But the variable is not checked when determining the configuration location
> in the code (see
> [HadoopUtils.getHadoopConfiguration|https://github.com/apache/flink/blob/ba027e0b3e2cf950cd0bd7fb4f632a9c3449d9ad/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L53]).
> This issue is about adding {{env.hadoop.conf.dir}} as an option to specify
> the Hadoop configuration directory.
> Initial proposal came from [this user mailing list
> thread|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Hadoop-config-on-docker-compose-td42967.html#a42994].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19442) Unaligned checkpoints 1.1
[ https://issues.apache.org/jira/browse/FLINK-19442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-19442: --- Labels: stale-major (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Unaligned checkpoints 1.1 > - > > Key: FLINK-19442 > URL: https://issues.apache.org/jira/browse/FLINK-19442 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Reporter: Arvid Heise >Priority: Major > Labels: stale-major > > This is an umbrella issue for all efforts that increase the usability of > Unaligned checkpoints. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22473) Kinesis connector options are not listed by its factory
[ https://issues.apache.org/jira/browse/FLINK-22473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22473: --- Labels: stale-major (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Kinesis connector options are not listed by its factory > --- > > Key: FLINK-22473 > URL: https://issues.apache.org/jira/browse/FLINK-22473 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Affects Versions: 1.12.2, 1.13.0 >Reporter: Ingo Bürk >Priority: Major > Labels: stale-major > > The documentation for the Kinesis connector lists a lot of options, but none > of them are actually returned by its factory in either required/optional > options. > The factory should correctly return at least documented options so that > eventually the connector options documentation can be generated from the code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17133) Support CsvParser Features for CSV format via SQL DDL
[ https://issues.apache.org/jira/browse/FLINK-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-17133: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Support CsvParser Features for CSV format via SQL DDL > - > > Key: FLINK-17133 > URL: https://issues.apache.org/jira/browse/FLINK-17133 > Project: Flink > Issue Type: New Feature > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Kai Chen >Priority: Major > Labels: pull-request-available, stale-major > Time Spent: 20m > Remaining Estimate: 0h > > Should we support CsvParser Features for RFC-compliant CSV format via SQL > DDL? > All CsvParser Features list in > [https://fasterxml.github.io/jackson-dataformats-text/javadoc/csv/2.10/com/fasterxml/jackson/dataformat/csv/CsvParser.Feature.html] > Such as : > IGNORE_TRAILING_UNMAPPABLE > SKIP_EMPTY_LINES > FAIL_ON_MISSING_COLUMNS > ... > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13815) Implement the SpaceAllocator
[
https://issues.apache.org/jira/browse/FLINK-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-13815:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 14, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it. If the "warning_label" label is not removed in 7 days, the
issue will be automatically unassigned.
> Implement the SpaceAllocator
>
>
> Key: FLINK-13815
> URL: https://issues.apache.org/jira/browse/FLINK-13815
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As described in the design doc, we need a {{SpaceAllocator}} to allocate
> space on off-heap/disk to store the spilled key-group data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15924: [FLINK-22670][FLIP-150][connector/common] Hybrid source baseline
flinkbot edited a comment on pull request #15924: URL: https://github.com/apache/flink/pull/15924#issuecomment-841943851 ## CI report: * 958751f1cbe06cf9163c40e550bc553d6180a77b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18338) * 2e147a5aff7ce35143c42557b651affcd132dbb4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22075) Incorrect null outputs in left join
[ https://issues.apache.org/jira/browse/FLINK-22075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22075: --- Labels: auto-deprioritized-critical auto-unassigned (was: auto-unassigned stale-critical) Priority: Major (was: Critical) This issue was labeled "stale-critical" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Critical, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Incorrect null outputs in left join > --- > > Key: FLINK-22075 > URL: https://issues.apache.org/jira/browse/FLINK-22075 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.12.2 > Environment: > https://github.com/jamii/streaming-consistency/blob/4e5d144dacf85e512bdc7afd77d031b5974d733e/pkgs.nix#L25-L46 > ``` > [nix-shell:~/streaming-consistency/flink]$ java -version > openjdk version "1.8.0_265" > OpenJDK Runtime Environment (build 1.8.0_265-ga) > OpenJDK 64-Bit Server VM (build 25.265-bga, mixed mode) > [nix-shell:~/streaming-consistency/flink]$ flink --version > Version: 1.12.2, Commit ID: 4dedee0 > [nix-shell:~/streaming-consistency/flink]$ nix-info > system: "x86_64-linux", multi-user?: yes, version: nix-env (Nix) 2.3.10, > channels(jamie): "", channels(root): "nixos-20.09.3554.f8929dce13e", nixpkgs: > /nix/var/nix/profiles/per-user/root/channels/nixos > ``` >Reporter: Jamie Brandon >Priority: Major > Labels: auto-deprioritized-critical, auto-unassigned > > I'm left joining a table with itself > [here](https://github.com/jamii/streaming-consistency/blob/4e5d144dacf85e512bdc7afd77d031b5974d733e/flink/src/main/java/Demo.java#L55-L66). > The output should have no nulls, or at least emit nulls and then retract > them. Instead I see: > ``` > jamie@machine:~/streaming-consistency/flink$ wc -l tmp/outer_join_with_time > 10 tmp/outer_join_with_time > jamie@machine:~/streaming-consistency/flink$ grep -c insert > tmp/outer_join_with_time > 10 > jamie@machine:~/streaming-consistency/flink$ grep -c 'null' > tmp/outer_join_with_time > 16943 > ``` > ~1.7% of the outputs are incorrect and never retracted. > [Full > output](https://gist.githubusercontent.com/jamii/983fee41609b1425fe7fa59d3249b249/raw/069b9dcd4faf9f6113114381bc7028c6642ca787/gistfile1.txt) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17200) Add connectors to ClickHouse
[ https://issues.apache.org/jira/browse/FLINK-17200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-17200: --- Labels: stale-major (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add connectors to ClickHouse > > > Key: FLINK-17200 > URL: https://issues.apache.org/jira/browse/FLINK-17200 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: jinhai >Priority: Major > Labels: stale-major > > Clickhouse is a powerful OLAP query engine and supports real-time data mini > batch writing. > We can add flink connectors to ClickHouse > [weisite: https://clickhouse.tech/|https://clickhouse.tech/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22397) OrcFileSystemITCase fails on Azure
[
https://issues.apache.org/jira/browse/FLINK-22397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22397:
---
Labels: auto-deprioritized-critical test-stability (was: stale-critical
test-stability)
Priority: Major (was: Critical)
This issue was labeled "stale-critical" 7 ago and has not received any updates
so it is being deprioritized. If this ticket is actually Critical, please raise
the priority and ask a committer to assign you the issue or revive the public
discussion.
> OrcFileSystemITCase fails on Azure
> --
>
> Key: FLINK-22397
> URL: https://issues.apache.org/jira/browse/FLINK-22397
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ORC, Table SQL / Runtime
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: auto-deprioritized-critical, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16923=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20=12429
> {code}
> 2021-04-21T06:00:43.1989525Z Apr 21 06:00:43 [ERROR]
> testOrcFilterPushDown[false](org.apache.flink.orc.OrcFileSystemITCase) Time
> elapsed: 8.733 s <<< ERROR!
> 2021-04-21T06:00:43.1991576Z Apr 21 06:00:43 java.lang.RuntimeException:
> Failed to fetch next result
> 2021-04-21T06:00:43.1992690Z Apr 21 06:00:43 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109)
> 2021-04-21T06:00:43.1999796Z Apr 21 06:00:43 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> 2021-04-21T06:00:43.2021072Z Apr 21 06:00:43 at
> org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:351)
> 2021-04-21T06:00:43.2022469Z Apr 21 06:00:43 at
> java.util.Iterator.forEachRemaining(Iterator.java:115)
> 2021-04-21T06:00:43.2023496Z Apr 21 06:00:43 at
> org.apache.flink.util.CollectionUtil.iteratorToList(CollectionUtil.java:109)
> 2021-04-21T06:00:43.2024451Z Apr 21 06:00:43 at
> org.apache.flink.table.planner.runtime.utils.BatchTestBase.executeQuery(BatchTestBase.scala:300)
> 2021-04-21T06:00:43.2025472Z Apr 21 06:00:43 at
> org.apache.flink.table.planner.runtime.utils.BatchTestBase.check(BatchTestBase.scala:140)
> 2021-04-21T06:00:43.2026438Z Apr 21 06:00:43 at
> org.apache.flink.table.planner.runtime.utils.BatchTestBase.checkResult(BatchTestBase.scala:106)
> 2021-04-21T06:00:43.2027531Z Apr 21 06:00:43 at
> org.apache.flink.table.planner.runtime.batch.sql.BatchFileSystemITCaseBase.check(BatchFileSystemITCaseBase.scala:46)
> 2021-04-21T06:00:43.2028933Z Apr 21 06:00:43 at
> org.apache.flink.table.planner.runtime.FileSystemITCaseBase$class.check(FileSystemITCaseBase.scala:57)
> 2021-04-21T06:00:43.2030041Z Apr 21 06:00:43 at
> org.apache.flink.table.planner.runtime.batch.sql.BatchFileSystemITCaseBase.check(BatchFileSystemITCaseBase.scala:33)
> 2021-04-21T06:00:43.2031068Z Apr 21 06:00:43 at
> org.apache.flink.orc.OrcFileSystemITCase.testOrcFilterPushDown(OrcFileSystemITCase.java:139)
> 2021-04-21T06:00:43.2031935Z Apr 21 06:00:43 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-04-21T06:00:43.2032743Z Apr 21 06:00:43 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-04-21T06:00:43.2033847Z Apr 21 06:00:43 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-04-21T06:00:43.2034729Z Apr 21 06:00:43 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-04-21T06:00:43.2035558Z Apr 21 06:00:43 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-04-21T06:00:43.2036546Z Apr 21 06:00:43 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-04-21T06:00:43.2037466Z Apr 21 06:00:43 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-04-21T06:00:43.2038683Z Apr 21 06:00:43 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-04-21T06:00:43.2039611Z Apr 21 06:00:43 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-04-21T06:00:43.2040473Z Apr 21 06:00:43 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2021-04-21T06:00:43.2041308Z Apr 21 06:00:43 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-04-21T06:00:43.2042580Z Apr 21 06:00:43 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-04-21T06:00:43.2043561Z Apr 21 06:00:43 at
>
[jira] [Updated] (FLINK-17225) Support native k8s for scala shell
[ https://issues.apache.org/jira/browse/FLINK-17225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-17225: --- Labels: stale-major (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Support native k8s for scala shell > -- > > Key: FLINK-17225 > URL: https://issues.apache.org/jira/browse/FLINK-17225 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Reporter: Yang Wang >Priority: Major > Labels: stale-major > > Currently, the Flink scala shell could create a new or retrieve an existing > YARN session cluster automatically. It is very convenient for the users. > Then it will be great we could also support the K8s deployment. Benefit from > native K8s integration, it is not very difficult for the implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22551) checkpoints: strange behaviour
[
https://issues.apache.org/jira/browse/FLINK-22551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22551:
---
Labels: auto-deprioritized-critical (was: stale-critical)
Priority: Major (was: Critical)
This issue was labeled "stale-critical" 7 ago and has not received any updates
so it is being deprioritized. If this ticket is actually Critical, please raise
the priority and ask a committer to assign you the issue or revive the public
discussion.
> checkpoints: strange behaviour
> ---
>
> Key: FLINK-22551
> URL: https://issues.apache.org/jira/browse/FLINK-22551
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.13.0
> Environment: {code:java}
> java -version
> openjdk version "11.0.2" 2019-01-15
> OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
> {code}
>Reporter: buom
>Priority: Major
> Labels: auto-deprioritized-critical
>
> * +*Case 1*:+ Work as expected
> {code:java}
> public class Example {
> public static class ExampleSource extends RichSourceFunction
> implements CheckpointedFunction {
> private volatile boolean isRunning = true;
> @Override
> public void open(Configuration parameters) throws Exception {
> System.out.println("[source] invoke open()");
> }
> @Override
> public void close() throws Exception {
> isRunning = false;
> System.out.println("[source] invoke close()");
> }
> @Override
> public void run(SourceContext ctx) throws Exception {
> System.out.println("[source] invoke run()");
> while (isRunning) {
> ctx.collect("Flink");
> Thread.sleep(500);
> }
> }
> @Override
> public void cancel() {
> isRunning = false;
> System.out.println("[source] invoke cancel()");
> }
> @Override
> public void snapshotState(FunctionSnapshotContext context) throws
> Exception {
> System.out.println("[source] invoke snapshotState()");
> }
> @Override
> public void initializeState(FunctionInitializationContext context)
> throws Exception {
> System.out.println("[source] invoke initializeState()");
> }
> }
> public static class ExampleSink extends PrintSinkFunction
> implements CheckpointedFunction {
> @Override
> public void snapshotState(FunctionSnapshotContext context) throws
> Exception {
> System.out.println("[sink] invoke snapshotState()");
> }
> @Override
> public void initializeState(FunctionInitializationContext context)
> throws Exception {
> System.out.println("[sink] invoke initializeState()");
> }
> }
> public static void main(String[] args) throws Exception {
> final StreamExecutionEnvironment env =
>
> StreamExecutionEnvironment.getExecutionEnvironment().enableCheckpointing(1000);
> DataStream stream = env.addSource(new ExampleSource());
> stream.addSink(new ExampleSink()).setParallelism(1);
> env.execute();
> }
> }
> {code}
> {code:java}
> $ java -jar ./example.jar
> [sink] invoke initializeState()
> [source] invoke initializeState()
> [source] invoke open()
> [source] invoke run()
> Flink
> [sink] invoke snapshotState()
> [source] invoke snapshotState()
> Flink
> Flink
> [sink] invoke snapshotState()
> [source] invoke snapshotState()
> Flink
> Flink
> [sink] invoke snapshotState()
> [source] invoke snapshotState()
> ^C
> {code}
> * *+Case 2:+* Run as unexpected (w/ _parallelism = 1_)
> {code:java}
> public class Example {
> public static class ExampleSource extends RichSourceFunction
> implements CheckpointedFunction {
> private volatile boolean isRunning = true;
> @Override
> public void open(Configuration parameters) throws Exception {
> System.out.println("[source] invoke open()");
> }
> @Override
> public void close() throws Exception {
> isRunning = false;
> System.out.println("[source] invoke close()");
> }
> @Override
> public void run(SourceContext ctx) throws Exception {
> System.out.println("[source] invoke run()");
> while (isRunning) {
> ctx.collect("Flink");
> Thread.sleep(500);
> }
> }
> @Override
> public void cancel() {
> isRunning = false;
> System.out.println("[source] invoke cancel()");
> }
> @Override
> public void
[jira] [Updated] (FLINK-22387) UpsertKafkaTableITCase hangs when setting up kafka
[
https://issues.apache.org/jira/browse/FLINK-22387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22387:
---
Labels: auto-deprioritized-critical test-stability (was: stale-critical
test-stability)
Priority: Major (was: Critical)
This issue was labeled "stale-critical" 7 ago and has not received any updates
so it is being deprioritized. If this ticket is actually Critical, please raise
the priority and ask a committer to assign you the issue or revive the public
discussion.
> UpsertKafkaTableITCase hangs when setting up kafka
> --
>
> Key: FLINK-22387
> URL: https://issues.apache.org/jira/browse/FLINK-22387
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / Ecosystem
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: auto-deprioritized-critical, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16901=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=6932
> {code}
> 2021-04-20T20:01:32.2276988Z Apr 20 20:01:32 "main" #1 prio=5 os_prio=0
> tid=0x7fe87400b000 nid=0x4028 runnable [0x7fe87df22000]
> 2021-04-20T20:01:32.2277666Z Apr 20 20:01:32java.lang.Thread.State:
> RUNNABLE
> 2021-04-20T20:01:32.2278338Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.Buffer.getByte(Buffer.java:312)
> 2021-04-20T20:01:32.2279325Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:310)
> 2021-04-20T20:01:32.2280656Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:492)
> 2021-04-20T20:01:32.2281603Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471)
> 2021-04-20T20:01:32.2282163Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.skipAll(Util.java:204)
> 2021-04-20T20:01:32.2282870Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.discard(Util.java:186)
> 2021-04-20T20:01:32.2283494Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.close(Http1ExchangeCodec.java:511)
> 2021-04-20T20:01:32.2284460Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.ForwardingSource.close(ForwardingSource.java:43)
> 2021-04-20T20:01:32.2285183Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.close(Exchange.java:313)
> 2021-04-20T20:01:32.2285756Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okio.RealBufferedSource.close(RealBufferedSource.java:476)
> 2021-04-20T20:01:32.2286287Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.internal.Util.closeQuietly(Util.java:139)
> 2021-04-20T20:01:32.2286795Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.ResponseBody.close(ResponseBody.java:192)
> 2021-04-20T20:01:32.2287270Z Apr 20 20:01:32 at
> org.testcontainers.shaded.okhttp3.Response.close(Response.java:290)
> 2021-04-20T20:01:32.2287913Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.okhttp.OkDockerHttpClient$OkResponse.close(OkDockerHttpClient.java:285)
> 2021-04-20T20:01:32.2288606Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$null$0(DefaultInvocationBuilder.java:272)
> 2021-04-20T20:01:32.2289295Z Apr 20 20:01:32 at
> org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$$Lambda$340/2058508175.close(Unknown
> Source)
> 2021-04-20T20:01:32.2289886Z Apr 20 20:01:32 at
> com.github.dockerjava.api.async.ResultCallbackTemplate.close(ResultCallbackTemplate.java:77)
> 2021-04-20T20:01:32.2290567Z Apr 20 20:01:32 at
> org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:202)
> 2021-04-20T20:01:32.2291051Z Apr 20 20:01:32 at
> org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:205)
> 2021-04-20T20:01:32.2291879Z Apr 20 20:01:32 - locked <0xe9cd50f8>
> (a [Ljava.lang.Object;)
> 2021-04-20T20:01:32.2292313Z Apr 20 20:01:32 at
> org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
> 2021-04-20T20:01:32.2292870Z Apr 20 20:01:32 at
> org.testcontainers.LazyDockerClient.authConfig(LazyDockerClient.java:12)
> 2021-04-20T20:01:32.2293383Z Apr 20 20:01:32 at
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:310)
> 2021-04-20T20:01:32.2293890Z Apr 20 20:01:32 at
> org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1029)
> 2021-04-20T20:01:32.2294578Z Apr 20 20:01:32 at
>
[GitHub] [flink] tweise commented on a change in pull request #15924: [FLINK-22670][FLIP-150][connector/common] Hybrid source baseline
tweise commented on a change in pull request #15924:
URL: https://github.com/apache/flink/pull/15924#discussion_r640184455
##
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/HybridSourceReader.java
##
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base.source.hybrid;
+
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceEvent;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+/** Hybrid source reader that delegates to the actual current source reader. */
+public class HybridSourceReader
+implements SourceReader> {
+private static final Logger LOG =
LoggerFactory.getLogger(HybridSourceReader.class);
+private static final int SOURCE_READER_FINISHED_EVENT_DELAY = 250;
Review comment:
This is used within poll. I added the delay after I realized that too
many events were generated. The reason for that is that the reader converts the
underlying `END_OF_INPUT` to `MORE_AVAILABLE`. But more input will only be
available after the reader was switched. I will check if this issue would go
away when using `NOTHING_AVAILABLE` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-22458) Failed when operating empty catalog table
[
https://issues.apache.org/jira/browse/FLINK-22458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22458:
---
Labels: stale-major (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Failed when operating empty catalog table
> -
>
> Key: FLINK-22458
> URL: https://issues.apache.org/jira/browse/FLINK-22458
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.12.2
> Environment: Flink: 1.12.2
>Reporter: Spongebob
>Priority: Major
> Labels: stale-major
>
> The pipline might like: HiveTable -> FlinkCatalogTable(might be empty) ->
> HiveTable
> It runs normally when the FlinkCatalogTable is not empty, But When
> FlinkCatalogTable is empty, Jobmanager throws this exception:
> {code:java}
> java.lang.Exception: Failed to finalize execution on master at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1373)
> ~[flownData-1.0-jar-with-dependencies.jar:?] at
> org.apache.flink.runtime.executiongraph.ExecutionVertex.executionFinished(ExecutionVertex.java:877)
> ~[flownData-1.0-jar-with-dependencies.jar:?] at
> org.apache.flink.runtime.executiongraph.Execution.markFinished(Execution.java:1241)
> ~[flownData-1.0-jar-with-dependencies.jar:?] at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.updateStateInternal(ExecutionGraph.java:1610)
> ~[flownData-1.0-jar-with-dependencies.jar:?] at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.updateState(ExecutionGraph.java:1584)
> ~[flownData-1.0-jar-with-dependencies.jar:?] at
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:663)
> ~[flownData-1.0-jar-with-dependencies.jar:?] at
> org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:89)
> ~[flownData-1.0-jar-with-dependencies.jar:?] at
> org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:447)
> ~[flownData-1.0-jar-with-dependencies.jar:?] at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_251] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_251] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_251] at java.lang.reflect.Method.invoke(Method.java:498)
> ~[?:1.8.0_251] at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305)
> ~[flownData-1.0-jar-with-dependencies.jar:?] at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212)
> ~[flownData-1.0-jar-with-dependencies.jar:?] at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
> ~[flownData-1.0-jar-with-dependencies.jar:?] at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
> ~[flownData-1.0-jar-with-dependencies.jar:?] at
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> [flownData-1.0-jar-with-dependencies.jar:?] at
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> [flownData-1.0-jar-with-dependencies.jar:?] at
> scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> [flownData-1.0-jar-with-dependencies.jar:?] at
> scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> [flownData-1.0-jar-with-dependencies.jar:?] at
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> [flownData-1.0-jar-with-dependencies.jar:?] at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> [flownData-1.0-jar-with-dependencies.jar:?] at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> [flownData-1.0-jar-with-dependencies.jar:?] at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> [flownData-1.0-jar-with-dependencies.jar:?] at
> akka.actor.Actor.aroundReceive(Actor.scala:517)
> [flownData-1.0-jar-with-dependencies.jar:?] at
> akka.actor.Actor.aroundReceive$(Actor.scala:515)
> [flownData-1.0-jar-with-dependencies.jar:?] at
> akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> [flownData-1.0-jar-with-dependencies.jar:?] at
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> [flownData-1.0-jar-with-dependencies.jar:?] at
>
[jira] [Updated] (FLINK-22446) Relocate dependencies in hive connector uber jar
[
https://issues.apache.org/jira/browse/FLINK-22446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22446:
---
Labels: stale-major (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Relocate dependencies in hive connector uber jar
>
>
> Key: FLINK-22446
> URL: https://issues.apache.org/jira/browse/FLINK-22446
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Hive
>Affects Versions: 1.13.0
>Reporter: Rui Li
>Priority: Major
> Labels: stale-major
>
> {{flink-sql-connector-hive}} includes {{hive-exec}} which is also a uber jar
> and contains some un-relocated dependencies such as guava and apache commons.
> We probably should relocate these dependencies in
> {{flink-sql-connector-hive}} so as to avoid conflicts with flink or other
> connectors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21608) YARNHighAvailabilityITCase.testClusterClientRetrieval fails with "There is at least one application..."
[
https://issues.apache.org/jira/browse/FLINK-21608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-21608:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> YARNHighAvailabilityITCase.testClusterClientRetrieval fails with "There is at
> least one application..."
> ---
>
> Key: FLINK-21608
> URL: https://issues.apache.org/jira/browse/FLINK-21608
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: stale-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14108=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf
> {code}
> 2021-03-04T10:45:47.3523930Z INFO: Binding
> org.apache.hadoop.yarn.webapp.GenericExceptionHandler to
> GuiceManagedComponentProvider with the scope "Singleton"
> 2021-03-04T10:45:47.4240091Z Mar 04, 2021 10:45:47 AM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> 2021-03-04T10:45:47.4241009Z INFO: Binding
> org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices to
> GuiceManagedComponentProvider with the scope "Singleton"
> 2021-03-04T10:47:53.6102867Z [ERROR] Tests run: 3, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 132.302 s <<< FAILURE! - in
> org.apache.flink.yarn.YARNHighAvailabilityITCase
> 2021-03-04T10:47:53.6103745Z [ERROR]
> testClusterClientRetrieval(org.apache.flink.yarn.YARNHighAvailabilityITCase)
> Time elapsed: 15.906 s <<< FAILURE!
> 2021-03-04T10:47:53.6104784Z java.lang.AssertionError: There is at least one
> application on the cluster that is not finished.[App
> application_1614854744820_0003 is in state RUNNING.]
> 2021-03-04T10:47:53.6106075Z at org.junit.Assert.fail(Assert.java:88)
> 2021-03-04T10:47:53.6108977Z at
> org.apache.flink.yarn.YarnTestBase$CleanupYarnApplication.close(YarnTestBase.java:322)
> 2021-03-04T10:47:53.6109784Z at
> org.apache.flink.yarn.YarnTestBase.runTest(YarnTestBase.java:286)
> 2021-03-04T10:47:53.6110493Z at
> org.apache.flink.yarn.YARNHighAvailabilityITCase.testClusterClientRetrieval(YARNHighAvailabilityITCase.java:219)
> 2021-03-04T10:47:53.6111446Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-03-04T10:47:53.6111871Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-03-04T10:47:53.6112360Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-03-04T10:47:53.6112784Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-03-04T10:47:53.6113210Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-03-04T10:47:53.6114001Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-03-04T10:47:53.6114796Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-03-04T10:47:53.6115388Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-03-04T10:47:53.6116123Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-03-04T10:47:53.6116995Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2021-03-04T10:47:53.6117810Z at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-03-04T10:47:53.6118621Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-03-04T10:47:53.6119311Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-03-04T10:47:53.6119840Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-03-04T10:47:53.6120279Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-03-04T10:47:53.6120739Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2021-03-04T10:47:53.6121173Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-03-04T10:47:53.6121692Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-03-04T10:47:53.6122128Z at
>
[jira] [Updated] (FLINK-22115) JobManager dies with IllegalStateException SharedSlot (physical request SlotRequestId{%}) has been released
[
https://issues.apache.org/jira/browse/FLINK-22115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22115:
---
Labels: stale-major (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> JobManager dies with IllegalStateException SharedSlot (physical request
> SlotRequestId{%}) has been released
> ---
>
> Key: FLINK-22115
> URL: https://issues.apache.org/jira/browse/FLINK-22115
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.12.0
>Reporter: wym_maozi
>Priority: Major
> Labels: stale-major
> Attachments: flink-root-standalonesession-0-banyue01.zip
>
>
> After TaskManager hangs, and JobManager fails to restart the task many times,
> I experienced fatal JobManager crashes, with the following log:
> {code:java}
> 2021-04-06 14:13:10,388 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting
> nesourceProfile{UNKNOWN} with allocation id 6c99582801115ac080d407123cf92e80
> from resource manager.
> 2021-04-06 14:13:10,389 INFO
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
> Request5123cc0fd9eb3 with allocation id 6c99582801115ac080d407123cf92e80.
> 2021-04-06 14:13:10,389 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Requesting
> nesourceProfile{UNKNOWN} with allocation id 56b82914f123bdf3aec2cad538dc8e75
> from resource manager.
> 2021-04-06 14:13:10,389 INFO
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
> Request5123cc0fd9eb3 with allocation id 56b82914f123bdf3aec2cad538dc8e75.
> 2021-04-06 14:13:10,389 ERROR
> org.apache.flink.runtime.util.FatalExitExceptionHandler [] - FATAL:
> Thread. Stopping the process...
> java.util.concurrent.CompletionException: java.lang.IllegalStateException:
> SharedSlot (physical request SlotR
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> ~[?:1.8.0_181
> at
> java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:708)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:687)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
> ~[?:1.8.0_181]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:404)
> ~[flink-dist_
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:197)
> ~[flink-dis
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:154)
> ~[flink-dist_2
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> [flink-dist_2.11-1.12.0.jar:1.12.0]
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> [flink-dist_2.11-1.12.0.jar:1.12.0]
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> [flink-dist_2.11-1.12.0.jar:1.1
> at
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> [flink-dist_2.11-1.12.0.jar:1.
> at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> [flink-dist_2.11-1.12.0.jar:1.
> at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> [flink-dist_2.11-1.12.0.jar:1.
> at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> [flink-dist_2.11-1.12.0.jar:1.
> at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> [flink-dist_2.11-1.12.0.jar:1.12.0]
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> [flink-dist_2.11-1.12.0.jar:1.12.0
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> [flink-dist_2.11-1.12.0.jar:1.12.0]
> at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> [flink-dist_2.11-1.12.0.jar:1.12.0]
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> [flink-dist_2.11-1.12.0.jar:1.12.0]
> at
[jira] [Updated] (FLINK-22328) Failed to send data to Kafka: Producer attempted an operation with an old epoch
[
https://issues.apache.org/jira/browse/FLINK-22328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22328:
---
Labels: Transactional auto-deprioritized-major kafka (was: Transactional
kafka stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Failed to send data to Kafka: Producer attempted an operation with an old
> epoch
> ---
>
> Key: FLINK-22328
> URL: https://issues.apache.org/jira/browse/FLINK-22328
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.11.1
> Environment: kafka 3.0.0
> Flink 1.11.1
>Reporter: 孙峰
>Priority: Minor
> Labels: Transactional, auto-deprioritized-major, kafka
>
> Flink job fails occasionally.Here is the stacktrace:
> {code:java}
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaException: Failed to
> send data to Kafka: Producer attempted an operation with an old epoch.Either
> there is a newer producer with the same transactionalId, or the producer's
> transactioin has been expired by the broker.
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.checkErroneous(FlinkKafkaProducer.java:640)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:157)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:81)
> at
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.invoke(TwoPhaseCommitSinkFunction.java:235)
> at
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)
>
> ...
> Caused by:org.apache.kafka.common.errors.ProducerFencedException: Producer
> attempted an operation with an old epoch.Either there is a newer producer
> with the same transactionalId, or the producer's transactioin has been
> expired by the broker.{code}
> The job use FlinkKafkaProducer with EXACTLY_ONCE and deploy on Yarn.
> In the debugging information I found the transactionalId is "Source: Custom
> Source -> (Process -> Sink: errorMessageToKafka, Sink: etlMultiTopicSink)
> -03f86923ea4164263684d81917202071-0".
> In kafka server.log,the exception:
> {code:java}
> ERROR [ReplicaManager borker=1004] Error processing append on partition
> ods_source-2 (kafka.server.ReplicaManager)
> org.apache.kafka.common.errors.ProducerFencedExceptioin: Producer's epoch is
> no longer valid.There is probably another producer with a newer epoch. 158
> (request epoch), 159 (server:epoch){code}
> Here is the log that kafka increase epoch with this transactionalId "Source:
> Custom Source -> (Process -> Sink: errorMessageToKafka, Sink:
> etlMultiTopicSink) -03f86923ea4164263684d81917202071-0":
> {code:java}
> INFO [TransactionCoordinator id=1003] Initialized transactionalId Source:
> Custom Source -> (Process -> Sink: errorMessageToKafka, Sink:
> etlMultiTopicSink) -03f86923ea4164263684d81917202071-0 with producerId 21036
> and producer epoch 158 on partition _transaction_state-3
> (kafka.coordinator.transaction.TransactionCoordinator)
> INFO [TransactionCoordinator id=1003] Initialized transactionalId Source:
> Custom Source -> (Process -> Sink: errorMessageToKafka, Sink:
> etlMultiTopicSink) -03f86923ea4164263684d81917202071-0 with producerId 21036
> and producer epoch 160 on partition _transaction_state-3
> (kafka.coordinator.transaction.TransactionCoordinator)
> {code}
> There is no info that record kafka set producer epoch to 159.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22019) UnalignedCheckpointRescaleITCase hangs on azure
[ https://issues.apache.org/jira/browse/FLINK-22019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22019: --- Labels: stale-major test-stability (was: test-stability) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > UnalignedCheckpointRescaleITCase hangs on azure > --- > > Key: FLINK-22019 > URL: https://issues.apache.org/jira/browse/FLINK-22019 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.13.0 >Reporter: Dawid Wysakowicz >Priority: Major > Labels: stale-major, test-stability > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15658=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=5360d54c-8d94-5d85-304e-a89267eb785a=9347 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22461) Resuming Savepoint (file, async, scale down) end-to-end test failed because of binding port failed
[
https://issues.apache.org/jira/browse/FLINK-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22461:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Resuming Savepoint (file, async, scale down) end-to-end test failed because
> of binding port failed
> --
>
> Key: FLINK-22461
> URL: https://issues.apache.org/jira/browse/FLINK-22461
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Tests
>Affects Versions: 1.11.3
>Reporter: Guowei Ma
>Priority: Major
> Labels: stale-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17179=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=2b7514ee-e706-5046-657b-3430666e7bd9=1495
> {code:java}
> 2021-04-25 20:27:11,375 INFO org.apache.flink.core.fs.FileSystem
> [] - Hadoop is not in the classpath/dependencies. The extended
> set of supported File Systems via Hadoop is not available.
> 2021-04-25 20:27:11,420 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Install
> security context.
> 2021-04-25 20:27:11,441 INFO
> org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot
> create Hadoop Security Module because Hadoop cannot be found in the Classpath.
> 2021-04-25 20:27:11,444 INFO
> org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file
> will be created as /tmp/jaas-1367939375368529920.conf.
> 2021-04-25 20:27:11,450 INFO
> org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] -
> Cannot install HadoopSecurityContext because Hadoop cannot be found in the
> Classpath.
> 2021-04-25 20:27:11,454 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] -
> Initializing cluster services.
> 2021-04-25 20:27:11,483 INFO
> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils[] - Trying to
> start actor system, external address localhost:6123, bind address
> 0.0.0.0:6123.
> 2021-04-25 20:27:12,014 INFO akka.event.slf4j.Slf4jLogger
> [] - Slf4jLogger started
> 2021-04-25 20:27:12,040 INFO akka.remote.Remoting
> [] - Starting remoting
> 2021-04-25 20:27:12,127 ERROR akka.remote.transport.netty.NettyTransport
> [] - failed to bind to /0.0.0.0:6123, shutting down Netty
> transport
> 2021-04-25 20:27:12,138 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Shutting
> StandaloneSessionClusterEntrypoint down with application status FAILED.
> Diagnostics java.net.BindException: Could not start actor system on any port
> in port range 6123
> at
> org.apache.flink.runtime.clusterframework.BootstrapTools.startRemoteActorSystem(BootstrapTools.java:173)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils$AkkaRpcServiceBuilder.createAndStart(AkkaRpcServiceUtils.java:363)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils.createRemoteRpcService(AkkaRpcServiceUtils.java:91)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:278)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:223)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:177)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:174)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:577)
> at
> org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint.main(StandaloneSessionClusterEntrypoint.java:67)
> .
> 2021-04-25 20:27:12,139 INFO
> akka.remote.RemoteActorRefProvider$RemotingTerminator[] - Shutting
> down remote daemon.
> 2021-04-25 20:27:12,141 INFO
> akka.remote.RemoteActorRefProvider$RemotingTerminator[] - Remote
> daemon shut down; proceeding with flushing remote transports.
> 2021-04-25 20:27:12,143 ERROR akka.remote.Remoting
[jira] [Updated] (FLINK-22272) Some scenes can't drop table by hive catalog
[ https://issues.apache.org/jira/browse/FLINK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22272: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Some scenes can't drop table by hive catalog > > > Key: FLINK-22272 > URL: https://issues.apache.org/jira/browse/FLINK-22272 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.12.1 >Reporter: wangzeyu >Priority: Major > Labels: pull-request-available, stale-major > Fix For: 1.12.1 > > > There are some scenes we can't drop table by hive catalog because of we > execute error create table sql like "caeate table tableName". And then when > we execute drop table sql "drop table tableName" will throw error > "org.apache.flink.table.catalog.exceptions.CatalogException: Failed to get > table schema from properties for generic table". > So, i think maybe we should remove the check when we drop table by > third-party catalog and just drop table depend on the third-party catalog. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22455) FlinkRelBuilder#windowAggregate will throw ClassCastException when function reuse
[
https://issues.apache.org/jira/browse/FLINK-22455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22455:
---
Labels: stale-major (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> FlinkRelBuilder#windowAggregate will throw ClassCastException when function
> reuse
> -
>
> Key: FLINK-22455
> URL: https://issues.apache.org/jira/browse/FLINK-22455
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: tartarus
>Priority: Major
> Labels: stale-major
> Attachments: FlinkRelBuilderTest.scala
>
>
> If the input parameter aggCalls of FlinkRelBuilder#windowAggregate contains
> the same aggregate function. Then it will throw ClassCastException, because
> of the optimization of aggregate function reuse. We did not judge the return
> value type, but direct type conversion;
> {code:java}
> val aggregate = super.transform(
> new UnaryOperator[RelBuilder.Config] {
> override def apply(t: RelBuilder.Config)
> : RelBuilder.Config = t.withPruneInputOfAggregate(false)
> })
> .push(build())
> .aggregate(groupKey, aggCalls)
> .build()
> .asInstanceOf[LogicalAggregate]
> {code}
> I wrote a test that triggered this problem.
> You can use the attached code to reproduce this problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22199) The JDBC connector does not support TIMESTAMP(6) WITH LOCAL TIME ZONE NOT NULL.
[
https://issues.apache.org/jira/browse/FLINK-22199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22199:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> The JDBC connector does not support TIMESTAMP(6) WITH LOCAL TIME ZONE NOT
> NULL.
> ---
>
> Key: FLINK-22199
> URL: https://issues.apache.org/jira/browse/FLINK-22199
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.12.2
>Reporter: Tomoyuki NAKAMURA
>Priority: Minor
> Labels: auto-deprioritized-major
>
> I have created a PostgreSQL database with the following table definitions.
> {code:java}
> CREATE TABLE IF NOT EXISTS test (
> id bigint NOT NULL,
> name varchar(255) NOT NULL,
> created_at timestamp with time zone NOT NULL,
> updated_at timestamp with time zone NOT NULL,
> PRIMARY KEY(id)
> );
> {code}
> When I tried to create a PostgresCatalog in Flink 1.12.2, an error occurred.
> {code:java}
> CREATE CATALOG pg WITH (
> 'type'='jdbc',
> 'base-url'='jdbc:postgresql://localhost:5432/',
> 'default-database' = 'test',
> 'username' = 'postgres',
> 'password' = 'postgres'
> ){code}
> {code:java}
> 2021-04-11 17:21:57,685 INFO || Catalog pg established connection to
> jdbc:postgresql://localhost:5432/test
> [org.apache.flink.connector.jdbc.catalog.AbstractJdbcCatalog]
> Exception in thread "main" java.lang.UnsupportedOperationException:
> Unsupported type:TIMESTAMP(6) WITH LOCAL TIME ZONE NOT NULL
> at
> org.apache.flink.connector.jdbc.internal.converter.AbstractJdbcRowConverter.createInternalConverter(AbstractJdbcRowConverter.java:183)
> at
> org.apache.flink.connector.jdbc.internal.converter.PostgresRowConverter.createPrimitiveConverter(PostgresRowConverter.java:119)
> at
> org.apache.flink.connector.jdbc.internal.converter.PostgresRowConverter.createInternalConverter(PostgresRowConverter.java:60)
> at
> org.apache.flink.connector.jdbc.internal.converter.AbstractJdbcRowConverter.createNullableInternalConverter(AbstractJdbcRowConverter.java:118)
> at
> org.apache.flink.connector.jdbc.internal.converter.AbstractJdbcRowConverter.(AbstractJdbcRowConverter.java:68)
> at
> org.apache.flink.connector.jdbc.internal.converter.PostgresRowConverter.(PostgresRowConverter.java:49)
> at
> org.apache.flink.connector.jdbc.dialect.PostgresDialect.getRowConverter(PostgresDialect.java:53)
> at
> org.apache.flink.connector.jdbc.table.JdbcDynamicTableSource.getScanRuntimeProvider(JdbcDynamicTableSource.java:117)
> at
> org.apache.flink.table.planner.sources.DynamicSourceUtils.validateScanSource(DynamicSourceUtils.java:254)
> at
> org.apache.flink.table.planner.sources.DynamicSourceUtils.prepareDynamicSource(DynamicSourceUtils.java:71)
> at
> org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.java:101)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3585)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2507)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2144)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2050)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertTemporalTable(SqlToRelConverter.java:2583)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2158)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2093)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2050)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertJoin(SqlToRelConverter.java:2866)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2162)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2050)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:663)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:644)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3438)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:570)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:165)
> at
>
[jira] [Updated] (FLINK-22474) Display job configuration in web-ui
[
https://issues.apache.org/jira/browse/FLINK-22474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22474:
---
Labels: stale-major (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Display job configuration in web-ui
> ---
>
> Key: FLINK-22474
> URL: https://issues.apache.org/jira/browse/FLINK-22474
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Web Frontend
>Affects Versions: 1.11.3, 1.12.2, 1.13.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: stale-major
>
> Currently, we only display the cluster configuration in Flink's web ui.
> That's why users don't see the per job configuration options when deploying a
> Flink job to a session cluster. I think it would be great to also see the per
> job specific configuration options (those used to create the job) in Flink's
> web ui. We might also want to extend the set of displayed {{ExecutionConfig}}
> options in the web ui or alternatively remove them if everything is contained
> in the per job configuration.
> One thing to keep in mind is that we should filter out cluster configuration
> options which are not respected by the session cluster. Otherwise it might be
> a bit confusing if the cluster and job configuration contain contradicting
> values.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22246) when use HiveCatalog create table , can't set Table owner property correctly
[
https://issues.apache.org/jira/browse/FLINK-22246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22246:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> when use HiveCatalog create table , can't set Table owner property correctly
> -
>
> Key: FLINK-22246
> URL: https://issues.apache.org/jira/browse/FLINK-22246
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.11.1, 1.12.0
>Reporter: xiangtao
>Priority: Minor
> Labels: auto-deprioritized-major
>
> when i use HiveCatalog create table in sql-client , i found it can't set
> Hive Table `owner` property correctly. i debug code , i found in
> `HiveCatalog.createTable` method
> {code:java}
> Table hiveTable =
> org.apache.hadoop.hive.ql.metadata.Table.getEmptyTable(
> tablePath.getDatabaseName(), tablePath.getObjectName());
> {code}
>
> this get hiveTable obj , owner field is null . beacuse it set owner through
> {code:java}
> t.setOwner(SessionState.getUserFromAuthenticator());
> {code}
>
> but SessionState is null .
> Fix this bug , we can add one code in HiveCatalog.open method .
> {code:java}
> SessionState.setCurrentSessionState(new SessionState(hiveConf));
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22331) CLI Frontend (RestClient) doesn't work on Apple M1
[
https://issues.apache.org/jira/browse/FLINK-22331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22331:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> CLI Frontend (RestClient) doesn't work on Apple M1
> --
>
> Key: FLINK-22331
> URL: https://issues.apache.org/jira/browse/FLINK-22331
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Affects Versions: 1.12.2, 1.13.0
>Reporter: Robert Metzger
>Priority: Minor
> Labels: auto-deprioritized-major
> Attachments: flink-muthmann-client-KlemensMac.local (1).log_native,
> flink-muthmann-client-KlemensMac.local.log_rosetta
>
>
> This issue was first reported by a user:
> https://lists.apache.org/thread.html/r50bda40a69688de52c9d6e3489ac2641491387c20fdc1cecedceee76%40%3Cuser.flink.apache.org%3E
> See attached logs.
> Exception without rosetta:
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: Failed to execute job 'Streaming WordCount'.
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:366)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: org.apache.flink.util.FlinkException: Failed to execute job
> 'Streaming WordCount'.
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1918)
> at
> org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:135)
> at
> org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1782)
> at
> org.apache.flink.streaming.examples.wordcount.WordCount.main(WordCount.java:97)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349)
> ... 8 more
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to
> submit JobGraph.
> at
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$7(RestClusterClient.java:400)
> at
> java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
> at
> java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:390)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.flink.runtime.rest.RestClient$ClientHandler.exceptionCaught(RestClient.java:613)
> at
>
[jira] [Updated] (FLINK-22322) Build hang when compiling Web UI
[
https://issues.apache.org/jira/browse/FLINK-22322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22322:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Build hang when compiling Web UI
>
>
> Key: FLINK-22322
> URL: https://issues.apache.org/jira/browse/FLINK-22322
> Project: Flink
> Issue Type: Bug
> Components: Build System, Runtime / Web Frontend
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Minor
> Labels: auto-deprioritized-major
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16667=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb=3560
> {code}
> [INFO] --- frontend-maven-plugin:1.6:install-node-and-npm (install node and
> npm) @ flink-runtime-web_2.11 ---
> [INFO] Installing node version v10.9.0
> [INFO] Downloading
> https://nodejs.org/dist/v10.9.0/node-v10.9.0-linux-x64.tar.gz to
> /__w/2/.m2/repository/com/github/eirslett/node/10.9.0/node-10.9.0-linux-x64.tar.gz
> [INFO] No proxies configured
> [INFO] No proxy was configured, downloading directly
> [INFO] Unpacking
> /__w/2/.m2/repository/com/github/eirslett/node/10.9.0/node-10.9.0-linux-x64.tar.gz
> into /__w/2/s/flink-runtime-web/web-dashboard/node/tmp
> [INFO] Copying node binary from
> /__w/2/s/flink-runtime-web/web-dashboard/node/tmp/node-v10.9.0-linux-x64/bin/node
> to /__w/2/s/flink-runtime-web/web-dashboard/node/node
> [INFO] Extracting NPM
> [INFO] Installed node locally.
> [INFO]
> [INFO] --- frontend-maven-plugin:1.6:npm (npm install) @
> flink-runtime-web_2.11 ---
> [INFO] Running 'npm ci --cache-max=0 --no-save' in
> /__w/2/s/flink-runtime-web/web-dashboard
> ##[error]The operation was canceled.
> {code}
> Could be a transient Azure error
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22296) Introduce Preconditions-util into Python API
[ https://issues.apache.org/jira/browse/FLINK-22296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22296: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Introduce Preconditions-util into Python API > > > Key: FLINK-22296 > URL: https://issues.apache.org/jira/browse/FLINK-22296 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Roc Marshal >Priority: Minor > Labels: auto-deprioritized-major > > Similar to the > [Preconditions|https://github.com/apache/flink/blob/87efae4d3180a52e16240a0b4bbb197f85acd22c/flink-core/src/main/java/org/apache/flink/util/Preconditions.java#L43] > class in flink java API -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14032) Make the cache size of RocksDBPriorityQueueSetFactory configurable
[
https://issues.apache.org/jira/browse/FLINK-14032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-14032:
---
Labels: stale-assigned usability (was: usability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 14, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it. If the "warning_label" label is not removed in 7 days, the
issue will be automatically unassigned.
> Make the cache size of RocksDBPriorityQueueSetFactory configurable
> --
>
> Key: FLINK-14032
> URL: https://issues.apache.org/jira/browse/FLINK-14032
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Minor
> Labels: stale-assigned, usability
> Fix For: 1.14.0
>
>
> Currently, the cache size of {{RocksDBPriorityQueueSetFactory}} has been set
> as 128 and no any ways to configure this to other value. (We could increase
> this to obtain better performance if necessary). Actually, this is also a
> TODO for quiet a long time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22283) GraphiteReporter Metrics named xxx alredy exists
[ https://issues.apache.org/jira/browse/FLINK-22283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22283: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > GraphiteReporter Metrics named xxx alredy exists > > > Key: FLINK-22283 > URL: https://issues.apache.org/jira/browse/FLINK-22283 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: 1.9.0 >Reporter: HideOnBush >Priority: Minor > Labels: auto-deprioritized-major > > When I was using Flink 1.9 to monitor GraphiteReporter, an error was > reported, which caused the relevant indicators of my taskmanager to not be > collected. > Error message: a metric named xxx alredy exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-20090) Expose SlotId / SlotSharingGroup in Rest API
[
https://issues.apache.org/jira/browse/FLINK-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot closed FLINK-20090.
--
Resolution: Auto Closed
This issue was labeled "stale-minor" 7 ago and has not received any updates so
I have gone ahead and closed it. If you are still affected by this or would
like to raise the priority of this ticket please re-open, removing the label
"auto-closed" and raise the ticket priority accordingly.
> Expose SlotId / SlotSharingGroup in Rest API
> -
>
> Key: FLINK-20090
> URL: https://issues.apache.org/jira/browse/FLINK-20090
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: Maximilian Michels
>Priority: Minor
> Labels: auto-closed
>
> There is no information on slot sharing exposed via the Rest API which would
> be useful to monitor how tasks are assigned to task slots.
> We could include the SlotId in {{SubtaskExecutionAttemptDetailsInfo}} and
> provide a list of slots in {{TaskManagersInfo}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18402) NullPointerException in org.apache.flink.api.common.typeutils.base.GenericArraySerializer.copy(GenericArraySerializer.java:96)
[
https://issues.apache.org/jira/browse/FLINK-18402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-18402:
---
Labels: stale-minor (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I noticed that neither this issue nor its
subtasks had updates for 180 days, so I labeled it "stale-minor". If you are
still affected by this bug or are still interested in this issue, please update
and remove the label.
> NullPointerException in
> org.apache.flink.api.common.typeutils.base.GenericArraySerializer.copy(GenericArraySerializer.java:96)
> --
>
> Key: FLINK-18402
> URL: https://issues.apache.org/jira/browse/FLINK-18402
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.10.0
> Environment: flink 1.10
>Reporter: zhoupeijie
>Priority: Minor
> Labels: stale-minor
>
> I use array as follows:
> {code:java}
> DataStream> ds =
> tEnv.toRetractStream(companyBrandSource, CompanyBrandEntity.class);
> SingleOutputStreamOperator middleData = ds.map(new
> RuleMapFunction(ruleList))
> .filter(Objects::nonNull);
> {code}
> and I get this error:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.flink.api.common.typeutils.base.GenericArraySerializer.copy(GenericArraySerializer.java:96)
> at
> org.apache.flink.api.common.typeutils.base.GenericArraySerializer.copy(GenericArraySerializer.java:37)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:639)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:616)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:596)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708)
> at
> org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
> {code}
> {color:#111f2c}then I change the SingleOutputStreamOperator to
> SingleOutputStreamOperator,it begins to run correctly.{color}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14078) Introduce more JDBCDialect implementations
[ https://issues.apache.org/jira/browse/FLINK-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-14078: --- Labels: stale-assigned (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 14, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. If the "warning_label" label is not removed in 7 days, the issue will be automatically unassigned. > Introduce more JDBCDialect implementations > -- > > Key: FLINK-14078 > URL: https://issues.apache.org/jira/browse/FLINK-14078 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Reporter: Canbin Zheng >Assignee: Canbin Zheng >Priority: Major > Labels: stale-assigned > > MySQL, Derby and Postgres JDBCDialect are available now, maybe we can > introduce more JDBCDialect implementations, such as OracleDialect, > SqlServerDialect, DB2Dialect, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22247) can not pass AddressList when connecting to rabbitmq
[
https://issues.apache.org/jira/browse/FLINK-22247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22247:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> can not pass AddressList when connecting to rabbitmq
>
>
> Key: FLINK-22247
> URL: https://issues.apache.org/jira/browse/FLINK-22247
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.2
> Environment: flink: 1.12.2
> rabbitmq: 3.8.4
>Reporter: Spongebob
>Priority: Minor
> Labels: auto-deprioritized-major
>
> We hope to connect to rabbitmq cluster address when using rabbitmq connector,
> So we override the setupConnection function to pass the rabbitmq cluster
> address, but the address class is not serializable thereby flink throws
> exception.
> {code:java}
> //代码占位符
> val rabbitmqAddresses = Array(
> new Address("xxx1", 5672),
> new Address("xxx2", 5672),
> new Address("xxx3", 5672))
> val dataStream = streamEnv
> .addSource(new RMQSource[String](
> rabbitmqConfig, // rabbitmq's connection config
> "queueName", // queue name
> true, // using correlation ids, assurance of exactly-once consume from
> rabbitmq
> new SimpleStringSchema // java deserialization
> ) {
> override def setupQueue(): Unit = {}
> override def setupConnection(): Connection = {
> rabbitmqConfig.getConnectionFactory.newConnection(rabbitmqAddresses)
> }
> }).setParallelism(1)
> {code}
> Exception in thread "main"
> org.apache.flink.api.common.InvalidProgramException:
> [Lcom.rabbitmq.client.Address;@436a4e4b is not serializable. The object
> probably contains or references non serializable fields.Exception in thread
> "main" org.apache.flink.api.common.InvalidProgramException:
> [Lcom.rabbitmq.client.Address;@436a4e4b is not serializable. The object
> probably contains or references non serializable fields. at
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:164) at
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132) at
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:69) at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:2000)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:1685)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:1668)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:1652)
> at
> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.scala:693)
> at testConsumer$.main(testConsumer.scala:30) at
> testConsumer.main(testConsumer.scala)Caused by:
> java.io.NotSerializableException: com.rabbitmq.client.Address at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) at
> java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378) at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) at
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:624)
> at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:143)
> ... 9 more
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-8641) Move BootstrapTools#getTaskManagerShellCommand to flink-yarn
[
https://issues.apache.org/jira/browse/FLINK-8641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot closed FLINK-8641.
-
Resolution: Auto Closed
This issue was labeled "stale-minor" 7 ago and has not received any updates so
I have gone ahead and closed it. If you are still affected by this or would
like to raise the priority of this ticket please re-open, removing the label
"auto-closed" and raise the ticket priority accordingly.
> Move BootstrapTools#getTaskManagerShellCommand to flink-yarn
>
>
> Key: FLINK-8641
> URL: https://issues.apache.org/jira/browse/FLINK-8641
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN, Runtime / Configuration
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Priority: Minor
> Labels: auto-closed
>
> I would like to move {{getTaskManagerShellCommand()}} and
> {{getStartCommand()}} from
> {{org.apache.flink.runtime.clusterframework.BootstrapTools}} in flink-runtime
> to flink-yarn.
> Yarn is the sole user of these methods, and both methods are directly related
> to the {{YARN_CONTAINER_START_COMMAND_TEMPLATE}} {{ConfigConstants}}
> We can't move this constant to {{YarnOptions}} at this point since the
> {{YarnOptions}} are in {{flink-yarn}}, but the above methods require the
> option to be accessible from {{flink-runtime}}.
> [~till.rohrmann] Do you see any problems that this move could cause?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22238) Investigate angular native flamegraph library
[ https://issues.apache.org/jira/browse/FLINK-22238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22238: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Investigate angular native flamegraph library > - > > Key: FLINK-22238 > URL: https://issues.apache.org/jira/browse/FLINK-22238 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Reporter: Chesnay Schepler >Priority: Minor > Labels: auto-deprioritized-major > > We currently use d3 for drawing the flamegraph, but there is a angular native > library we could use. > > https://github.com/apache/flink/pull/15054#discussion_r603727674 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22330) Bumps checkstyle, junit, log4j version for security
[ https://issues.apache.org/jira/browse/FLINK-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22330: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Bumps checkstyle, junit, log4j version for security > --- > > Key: FLINK-22330 > URL: https://issues.apache.org/jira/browse/FLINK-22330 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.12.2 >Reporter: Brandon Jiang >Priority: Minor > Labels: auto-deprioritized-major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22209) Make StreamTask#triggerCheckpointAsync return void
[
https://issues.apache.org/jira/browse/FLINK-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22209:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Make StreamTask#triggerCheckpointAsync return void
> --
>
> Key: FLINK-22209
> URL: https://issues.apache.org/jira/browse/FLINK-22209
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
>Reporter: Dawid Wysakowicz
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.14.0
>
>
> The return value of {{StreamTask#triggerCheckpointAsync}} is ignored in
> production code. We should remove it to decrease chances for confustion.
> We must make sure we do not lose test coverage with it. A lot of tests depend
> on that return value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22223) Explicitly configure rest.address in distribution config
[
https://issues.apache.org/jira/browse/FLINK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-3:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Explicitly configure rest.address in distribution config
>
>
> Key: FLINK-3
> URL: https://issues.apache.org/jira/browse/FLINK-3
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration
>Reporter: Chesnay Schepler
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.14.0
>
>
> We should finally update the default config in the distribution:
> {code}
> Configuration [] - Config uses fallback configuration key
> 'jobmanager.rpc.address' instead of key 'rest.address'
> {code}
> As part of this task we also need to go over all occurrences of
> {{jobmanager.rpc.address}} in the documentation and determine whether this is
> about internal or external connectivity.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14482) Bump up rocksdb version
[ https://issues.apache.org/jira/browse/FLINK-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-14482: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 14, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. If the "warning_label" label is not removed in 7 days, the issue will be automatically unassigned. > Bump up rocksdb version > --- > > Key: FLINK-14482 > URL: https://issues.apache.org/jira/browse/FLINK-14482 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.14.0 > > > This JIRA aims at rebasing frocksdb to [newer > version|https://github.com/facebook/rocksdb/releases] of official RocksDB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22274) SQL parser fails to parse group by statements with brackets in views
[
https://issues.apache.org/jira/browse/FLINK-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22274:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> SQL parser fails to parse group by statements with brackets in views
>
>
> Key: FLINK-22274
> URL: https://issues.apache.org/jira/browse/FLINK-22274
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.13.0
>Reporter: Caizhi Weng
>Priority: Minor
> Labels: auto-deprioritized-major
>
> Add the following case to
> org.apache.flink.table.planner.runtime.batch.sql.CalcITCase to reproduce this
> bug.
> {code:scala}
> def myTest(): Unit = {
> tEnv.executeSql("CREATE VIEW my_view AS SELECT max(a) AS mx FROM Table3
> group by (b, c)")
> checkResult(
> "SELECT mx FROM my_view WHERE mx > 5",
> Seq())
> }
> {code}
> The exception stack is
> {code}
> org.apache.flink.table.api.SqlParserException: SQL parse failed. Encountered
> "." at line 3, column 22.
> Was expecting one of:
> ")" ...
> "," ...
>
> at
> org.apache.flink.table.planner.parse.CalciteParser.parse(CalciteParser.java:56)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl$ToRelContextImpl.expandView(FlinkPlannerImpl.scala:263)
> at
> org.apache.calcite.plan.ViewExpanders$1.expandView(ViewExpanders.java:52)
> at
> org.apache.flink.table.planner.catalog.SqlCatalogViewTable.convertToRel(SqlCatalogViewTable.java:63)
> at
> org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.expand(ExpandingPreparingTable.java:68)
> at
> org.apache.flink.table.planner.plan.schema.ExpandingPreparingTable.toRel(ExpandingPreparingTable.java:64)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3592)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2514)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2151)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2100)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2057)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:663)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:644)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3445)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:570)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:178)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:170)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:975)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:944)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:279)
> at
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:99)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:703)
> at
> org.apache.flink.table.planner.runtime.utils.BatchTestBase.parseQuery(BatchTestBase.scala:297)
> at
> org.apache.flink.table.planner.runtime.utils.BatchTestBase.check(BatchTestBase.scala:139)
> at
> org.apache.flink.table.planner.runtime.utils.BatchTestBase.checkResult(BatchTestBase.scala:106)
> at
> org.apache.flink.table.planner.runtime.batch.sql.CalcITCase.myTest(CalcITCase.scala:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
>
[jira] [Updated] (FLINK-22280) Add XML output format for Flink Table
[ https://issues.apache.org/jira/browse/FLINK-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22280: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Add XML output format for Flink Table > - > > Key: FLINK-22280 > URL: https://issues.apache.org/jira/browse/FLINK-22280 > Project: Flink > Issue Type: New Feature > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Affects Versions: 1.12.1 >Reporter: François Lacombe >Priority: Minor > Labels: auto-deprioritized-major > > Dear maintainers > I'm looking for the ability to output xml files from Flink Table API, just > like csv and json already supported formats. > To me, a new format could be required to make the appropriate serialization. > Am I missing any existing feature (or duplicate issue) that could allow it > without a dedicated format? > Depending on your returns and if it makes sense, I could get involved in > writing the appropriate format based on the same logic as > `JsonRowDataSerializationSchema`. > Best regards -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13803) Introduce SpillableHeapKeyedStateBackend and all necessities
[
https://issues.apache.org/jira/browse/FLINK-13803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-13803:
---
Labels: stale-assigned (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 14, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it. If the "warning_label" label is not removed in 7 days, the
issue will be automatically unassigned.
> Introduce SpillableHeapKeyedStateBackend and all necessities
>
>
> Key: FLINK-13803
> URL: https://issues.apache.org/jira/browse/FLINK-13803
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yu Li
>Assignee: PengFei Li
>Priority: Major
> Labels: stale-assigned
>
> This JIRA aims at introducing a new {{SpillableHeapKeyedStateBackend}} which
> will reuse most code of the {{HeapKeyedStateBackend}} (probably the only
> difference is the spill-able one will register a {{HybridStateTable}}), and
> allow using it in {{FsStateBackend}} and {{MemoryStateBackend}} (only as an
> option, by default still {{HeapKeyedStateBackend}}) through configuration.
> The related necessities include but are not limited to:
> * A relative backend builder class
> * Relative restore operation classes
> * Necessary configurations for using spill-able backend
> This should be the last JIRA after which the spill-able heap backend feature
> will become runnable regardless of the stability and performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-16839) Connectors Kinesis metrics can be disabled
[ https://issues.apache.org/jira/browse/FLINK-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot closed FLINK-16839. -- Resolution: Auto Closed This issue was labeled "stale-minor" 7 ago and has not received any updates so I have gone ahead and closed it. If you are still affected by this or would like to raise the priority of this ticket please re-open, removing the label "auto-closed" and raise the ticket priority accordingly. > Connectors Kinesis metrics can be disabled > -- > > Key: FLINK-16839 > URL: https://issues.apache.org/jira/browse/FLINK-16839 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis, Runtime / Metrics >Affects Versions: 1.10.0 >Reporter: Muchl >Priority: Minor > Labels: auto-closed > > Currently there are 9 metrics in the kinesis connector, each of which is > recorded according to the kinesis shard dimension. If there are enough > shards, taskmanager mtrics will be unavailable. > In our production environment, a case of a job that reads dynamodb stream > kinesis adapter, this dynamodb has more than 10,000 shards, multiplied by 9 > metrics, there are more than 100,000 metrics for kinesis, and the entire > metrics output reaches tens of MB , Cause prometheus cannot collect metrics. > We should have a configuration item that can disable kinesis metrics -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[ https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22300: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of > TimeCharacteristic > > > Key: FLINK-22300 > URL: https://issues.apache.org/jira/browse/FLINK-22300 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.11.3 >Reporter: Bo Huang >Priority: Minor > Labels: auto-deprioritized-major > > StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime). > The data after Windows process can not be evicted by TimeEvictor of Keyed > Windows Beause TimestampedValue have no timestamp value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15674) Let Java and Scala Type Extraction go through the same stack
[
https://issues.apache.org/jira/browse/FLINK-15674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15674:
---
Labels: stale-assigned usability (was: usability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 14, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it. If the "warning_label" label is not removed in 7 days, the
issue will be automatically unassigned.
> Let Java and Scala Type Extraction go through the same stack
>
>
> Key: FLINK-15674
> URL: https://issues.apache.org/jira/browse/FLINK-15674
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Stephan Ewen
>Assignee: Guowei Ma
>Priority: Major
> Labels: stale-assigned, usability
>
> Currently, the Java and Scala Type Extraction stacks are completely different.
> * Java uses the {{TypeExtractor}}
> * Scala uses the type extraction macros.
> As a result, the same class can be extracted as different types in the
> different stacks, which can lead to very confusing results. In particular,
> when you use the TypeExtractor on Scala Classes, you always get a
> {{GenericType}}.
> *Suggestion for New Design*
> There should be one type extraction stack, based on the TypeExtractor.
> * The TypeExtractor should be extensible and load additions through service
> loaders, similar as it currently loads Avro as an extension.
> * The Scala Type Extraction logic should be such an extension.
> * The Scala Marcos would only capture the {{Type}} (as in Java type), meaning
> {{Class}}, or {{ParameterizedType}}, or {{Array}} (etc.) and delegate this to
> the TypeExtractor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15532) Enable strict capacity limit for memory usage for RocksDB
[ https://issues.apache.org/jira/browse/FLINK-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-15532: --- Labels: stale-assigned (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 14, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. If the "warning_label" label is not removed in 7 days, the issue will be automatically unassigned. > Enable strict capacity limit for memory usage for RocksDB > - > > Key: FLINK-15532 > URL: https://issues.apache.org/jira/browse/FLINK-15532 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Labels: stale-assigned > Attachments: image-2020-10-23-14-39-45-997.png, > image-2020-10-23-14-41-10-584.png, image-2020-10-23-14-43-18-739.png, > image-2020-10-23-14-55-08-120.png > > > Currently, due to the limitation of RocksDB (see > [issue-6247|https://github.com/facebook/rocksdb/issues/6247]), we cannot > create a strict-capacity-limit LRUCache which shared among rocksDB > instance(s). > This issue tracks this problem and offer the ability of strict mode once we > could enable this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15507) Activate local recovery for RocksDB backends by default
[ https://issues.apache.org/jira/browse/FLINK-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-15507: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 14, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. If the "warning_label" label is not removed in 7 days, the issue will be automatically unassigned. > Activate local recovery for RocksDB backends by default > --- > > Key: FLINK-15507 > URL: https://issues.apache.org/jira/browse/FLINK-15507 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Stephan Ewen >Assignee: Zakelly Lan >Priority: Critical > Labels: pull-request-available, stale-assigned > Fix For: 1.14.0 > > > For the RocksDB state backend, local recovery has no overhead when > incremental checkpoints are used. > It should be activated by default, because it greatly helps with recovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15376) support "CREATE TABLE AS" in Flink SQL
[ https://issues.apache.org/jira/browse/FLINK-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-15376: --- Labels: auto-unassigned stale-assigned (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 14, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. If the "warning_label" label is not removed in 7 days, the issue will be automatically unassigned. > support "CREATE TABLE AS" in Flink SQL > -- > > Key: FLINK-15376 > URL: https://issues.apache.org/jira/browse/FLINK-15376 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Bowen Li >Assignee: Danny Chen >Priority: Major > Labels: auto-unassigned, stale-assigned > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22627) Remove SlotManagerImpl
[
https://issues.apache.org/jira/browse/FLINK-22627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22627:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 14, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it. If the "warning_label" label is not removed in 7 days, the
issue will be automatically unassigned.
> Remove SlotManagerImpl
> --
>
> Key: FLINK-22627
> URL: https://issues.apache.org/jira/browse/FLINK-22627
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Yangze Guo
>Assignee: Yangze Guo
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.14.0
>
>
> As the declarative resource management is completed (FLINK-10404) and the old
> {{SlotPoolImpl}} is removed in FLINK-22477, it's time to remove the
> {{SlotManagerImpl}} and
> all related classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22644) Translate "Native Kubernetes" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-22644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22644: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 14, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. If the "warning_label" label is not removed in 7 days, the issue will be automatically unassigned. > Translate "Native Kubernetes" page into Chinese > --- > > Key: FLINK-22644 > URL: https://issues.apache.org/jira/browse/FLINK-22644 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Yuchen Cheng >Assignee: Yuchen Cheng >Priority: Major > Labels: pull-request-available, stale-assigned > > The page url is > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16009: [hotfix][docs] fix a few typos in TVF documentation
flinkbot edited a comment on pull request #16009: URL: https://github.com/apache/flink/pull/16009#issuecomment-849076051 ## CI report: * 3bcebda3993ad3a01e52aea7adcd4b8b42c1a679 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18352) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18229) Pending worker requests should be properly cleared
[
https://issues.apache.org/jira/browse/FLINK-18229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17352093#comment-17352093
]
Nicolas Ferrario commented on FLINK-18229:
--
Hey [~xintongsong], what's the status of this ticket? We just had this problem
running on Native Kubernetes. So we were using all resources available in a K8s
cluster, and killed a pod (TM) intentionally. Flink somehow requested 6 more
TMs, and only one of them succeed, since there were no more resources available
for the rest. What's interesting is that the old TMs never stopped, so they got
reused after the job recovered itself from last checkpoint, but we were left
with 5 TMs in Pending state that will never be gone unless we free some slots.
> Pending worker requests should be properly cleared
> --
>
> Key: FLINK-18229
> URL: https://issues.apache.org/jira/browse/FLINK-18229
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes, Deployment / YARN, Runtime /
> Coordination
>Affects Versions: 1.9.3, 1.10.1, 1.11.0
>Reporter: Xintong Song
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently, if Kubernetes/Yarn does not have enough resources to fulfill
> Flink's resource requirement, there will be pending pod/container requests on
> Kubernetes/Yarn. These pending resource requirements are never cleared until
> either fulfilled or the Flink cluster is shutdown.
> However, sometimes Flink no longer needs the pending resources. E.g., the
> slot request is then fulfilled by another slots that become available, or the
> job failed due to slot request timeout (in a session cluster). In such cases,
> Flink does not remove the resource request until the resource is allocated,
> then it discovers that it no longer needs the allocated resource and release
> them. This would affect the underlying Kubernetes/Yarn cluster, especially
> when the cluster is under heavy workload.
> It would be good for Flink to cancel pod/container requests as earlier as
> possible if it can discover that some of the pending workers are no longer
> needed.
> There are several approaches potentially achieve this.
> # We can always check whether there's a pending worker that can be canceled
> when a \{{PendingTaskManagerSlot}} is unassigned.
> # We can have a separate timeout for requesting new worker. If the resource
> cannot be allocated within the given time since requested, we should cancel
> that resource request and claim a resource allocation failure.
> # We can share the same timeout for starting new worker (proposed in
> FLINK-13554). This is similar to 2), but it requires the worker to be
> registered, rather than allocated, before timeout.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15885: [FLINK-22376][runtime] RecoveredChannelStateHandler recycles the buffer if it was created inside and doesn't recycle if it was passed
flinkbot edited a comment on pull request #15885: URL: https://github.com/apache/flink/pull/15885#issuecomment-837001445 ## CI report: * 8be462efef265e62459453bde97db575da985e22 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18351) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] afedulov commented on pull request #15991: [FLINK-22757][docs] Adding gcs documentation. Connecting flink to gcs.
afedulov commented on pull request #15991: URL: https://github.com/apache/flink/pull/15991#issuecomment-849101571 It does.  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22774) Update Kinesis SQL connector's Guava to 27.0-jre
[ https://issues.apache.org/jira/browse/FLINK-22774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther updated FLINK-22774: - Component/s: Connectors / Kinesis > Update Kinesis SQL connector's Guava to 27.0-jre > > > Key: FLINK-22774 > URL: https://issues.apache.org/jira/browse/FLINK-22774 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis, Table SQL / Ecosystem >Reporter: Timo Walther >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.12.5, 1.13.2 > > > `flink-coonector-kinesis` uses a very old version (18.0) for compatibility > reasons. However, since we don't expose Guava and relocate it in SQL > connectors, we can use a newer version and avoid security concerns raised by > vulnerability tools. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22774) Update Kinesis SQL connector's Guava to 27.0-jre
[ https://issues.apache.org/jira/browse/FLINK-22774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther updated FLINK-22774: - Summary: Update Kinesis SQL connector's Guava to 27.0-jre (was: Package flink-sql-connector-kinesis with newer Guava version) > Update Kinesis SQL connector's Guava to 27.0-jre > > > Key: FLINK-22774 > URL: https://issues.apache.org/jira/browse/FLINK-22774 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Timo Walther >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available > > `flink-coonector-kinesis` uses a very old version (18.0) for compatibility > reasons. However, since we don't expose Guava and relocate it in SQL > connectors, we can use a newer version and avoid security concerns raised by > vulnerability tools. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-22774) Update Kinesis SQL connector's Guava to 27.0-jre
[ https://issues.apache.org/jira/browse/FLINK-22774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther closed FLINK-22774. Fix Version/s: 1.13.2 1.12.5 1.14.0 Resolution: Fixed Fixed in 1.14.0: df58a4e877dd8bb9f107db518bf55d0597700d24 Fixed in 1.13.2: 878dfe961528d2093990d02dd6e2c291c7012f90 Fixed in 1.12.5: d2153293eaa2066d959b8efdfd9501ea8b1b79aa > Update Kinesis SQL connector's Guava to 27.0-jre > > > Key: FLINK-22774 > URL: https://issues.apache.org/jira/browse/FLINK-22774 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Timo Walther >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.12.5, 1.13.2 > > > `flink-coonector-kinesis` uses a very old version (18.0) for compatibility > reasons. However, since we don't expose Guava and relocate it in SQL > connectors, we can use a newer version and avoid security concerns raised by > vulnerability tools. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] alpinegizmo commented on pull request #15991: [FLINK-22757][docs] Adding gcs documentation. Connecting flink to gcs.
alpinegizmo commented on pull request #15991: URL: https://github.com/apache/flink/pull/15991#issuecomment-849092771 @afedulov Can you confirm that GCS works for checkpointing? If so, I think the page should mention that use case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16009: [hotfix][docs] fix a few typos in TVF documentation
flinkbot edited a comment on pull request #16009: URL: https://github.com/apache/flink/pull/16009#issuecomment-849076051 ## CI report: * 3bcebda3993ad3a01e52aea7adcd4b8b42c1a679 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18352) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
