[jira] [Updated] (FLINK-19199) Add execution attempt ID to feedback channel keys
[
https://issues.apache.org/jira/browse/FLINK-19199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-19199:
---
Labels: pull-request-available (was: )
> Add execution attempt ID to feedback channel keys
> -
>
> Key: FLINK-19199
> URL: https://issues.apache.org/jira/browse/FLINK-19199
> Project: Flink
> Issue Type: Task
> Components: Stateful Functions
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
> Labels: pull-request-available
>
> The feedback brokers identify a single pair of producer / consumer with a
> {{SubtaskFeedbackKey}}, which is uniquely identified within a job by only the
> subtask index.
> This can potentially become an issue in job restart scenarios, since the
> feedback brokers are static fields, and are durable across non-TM failure job
> restarts.
> Currently it is not a problem because we do clear broker channels on close,
> but adding an extra execution attempt ID into the {{SubtaskFeedbackKey}} can
> make things safer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-statefun] tzulitai opened a new pull request #148: [FLINK-19199] [core] Add execution attempt ID to feedback channel keys
tzulitai opened a new pull request #148: URL: https://github.com/apache/flink-statefun/pull/148 The feedback brokers identify a single pair of producer / consumer with a `SubtaskFeedbackKey`, which is uniquely identified within a job by only the subtask index. This can potentially become an issue in job restart scenarios, since the feedback brokers are static fields, and are durable across non-TM failure job restarts. Currently it is not a problem because we do clear broker channels on close, but adding an extra execution attempt ID into the `SubtaskFeedbackKey` can make things safer. --- ## Verifying this change There are existing unit tests in `FeedbackChannelTest`. Moreover, it is assumed that the failure recover / exactly-once E2E tests have sufficient coverage for this change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-19199) Add execution attempt ID to feedback channel keys
Tzu-Li (Gordon) Tai created FLINK-19199:
---
Summary: Add execution attempt ID to feedback channel keys
Key: FLINK-19199
URL: https://issues.apache.org/jira/browse/FLINK-19199
Project: Flink
Issue Type: Task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
The feedback brokers identify a single pair of producer / consumer with a
{{SubtaskFeedbackKey}}, which is uniquely identified within a job by only the
subtask index.
This can potentially become an issue in job restart scenarios, since the
feedback brokers are static fields, and are durable across non-TM failure job
restarts.
Currently it is not a problem because we do clear broker channels on close, but
adding an extra execution attempt ID into the {{SubtaskFeedbackKey}} can make
things safer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19198) Invalid default value for consume-start-offset in Hive streaming
[ https://issues.apache.org/jira/browse/FLINK-19198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-19198: - Description: The default value for `streaming-source.consume-start-offset` in Hive streaming is `1970-00-00`(see [docs]([https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/hive/hive_streaming.html#streaming-source-consume-start-offset])), which is not a valid date. It does not affect the current execution though, since it's converted to milliseconds afterward (which is. -276480). But still it's not a good default, and we should change it to `1970-01-01`. was: The default value for `streaming-source.consume-start-offset` in Hive streaming is `1970-00-00`(see [docs]([https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/hive/hive_streaming.html#streaming-source-consume-start-offset])), which is not a valid date. It does not affect the current execution though, since it's converted to milliseconds afterward (which is. -276480), but still it's not a good default. We should change it to `1970-01-01`. > Invalid default value for consume-start-offset in Hive streaming > > > Key: FLINK-19198 > URL: https://issues.apache.org/jira/browse/FLINK-19198 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / API >Affects Versions: 1.11.1 >Reporter: Paul Lin >Priority: Minor > > The default value for `streaming-source.consume-start-offset` in Hive > streaming is `1970-00-00`(see > [docs]([https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/hive/hive_streaming.html#streaming-source-consume-start-offset])), > which is not a valid date. > It does not affect the current execution though, since it's converted to > milliseconds afterward (which is. -276480). But still it's not a good > default, and we should change it to `1970-01-01`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19198) Invalid default value for consume-start-offset in Hive streaming
[ https://issues.apache.org/jira/browse/FLINK-19198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-19198: - Description: The default value for `streaming-source.consume-start-offset` in Hive streaming is `1970-00-00`(see [docs]([https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/hive/hive_streaming.html#streaming-source-consume-start-offset])), which is not a valid date. It does not affect the current execution though, since it's converted to milliseconds afterward (which is. -276480), but still it's not a good default. We should change it to `1970-01-01`. was: The default value for `streaming-source.consume-start-offset` in Hive streaming is `1970-00-00`(see [docs](https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/hive/hive_streaming.html#streaming-source-consume-start-offset)), which is not a valid date. It does not affect the current execution though, since it's converted to milliseconds afterward, but still it's not a good default. We should change it to `1970-01-01`. > Invalid default value for consume-start-offset in Hive streaming > > > Key: FLINK-19198 > URL: https://issues.apache.org/jira/browse/FLINK-19198 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / API >Affects Versions: 1.11.1 >Reporter: Paul Lin >Priority: Minor > > The default value for `streaming-source.consume-start-offset` in Hive > streaming is `1970-00-00`(see > [docs]([https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/hive/hive_streaming.html#streaming-source-consume-start-offset])), > which is not a valid date. > It does not affect the current execution though, since it's converted to > milliseconds afterward (which is. -276480), but still it's not a good > default. We should change it to `1970-01-01`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19198) Invalid default value for consume-start-offset in Hive streaming
[ https://issues.apache.org/jira/browse/FLINK-19198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193978#comment-17193978 ] Paul Lin commented on FLINK-19198: -- I'd like to fix this if we all agree to. > Invalid default value for consume-start-offset in Hive streaming > > > Key: FLINK-19198 > URL: https://issues.apache.org/jira/browse/FLINK-19198 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / API >Affects Versions: 1.11.1 >Reporter: Paul Lin >Priority: Minor > > The default value for `streaming-source.consume-start-offset` in Hive > streaming is `1970-00-00`(see > [docs](https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/hive/hive_streaming.html#streaming-source-consume-start-offset)), > which is not a valid date. > It does not affect the current execution though, since it's converted to > milliseconds afterward, but still it's not a good default. We should change > it to `1970-01-01`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19198) Invalid default value for consume-start-offset in Hive streaming
Paul Lin created FLINK-19198: Summary: Invalid default value for consume-start-offset in Hive streaming Key: FLINK-19198 URL: https://issues.apache.org/jira/browse/FLINK-19198 Project: Flink Issue Type: Bug Components: Connectors / Hive, Table SQL / API Affects Versions: 1.11.1 Reporter: Paul Lin The default value for `streaming-source.consume-start-offset` in Hive streaming is `1970-00-00`(see [docs](https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/hive/hive_streaming.html#streaming-source-consume-start-offset)), which is not a valid date. It does not affect the current execution though, since it's converted to milliseconds afterward, but still it's not a good default. We should change it to `1970-01-01`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19197) Add documentation for PersistedStateRegistry
[
https://issues.apache.org/jira/browse/FLINK-19197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-19197:
---
Labels: pull-request-available (was: )
> Add documentation for PersistedStateRegistry
>
>
> Key: FLINK-19197
> URL: https://issues.apache.org/jira/browse/FLINK-19197
> Project: Flink
> Issue Type: Task
> Components: Stateful Functions
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
> Labels: pull-request-available
>
> Documentation for the new {{PersistedStateRegistry}} state construct is still
> missing. This should be added as a new sub-section beneath
> https://ci.apache.org/projects/flink/flink-statefun-docs-master/sdk/java.html#persistence.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-statefun] tzulitai opened a new pull request #147: [FLINK-19197] [docs] Add documentation for PersistedStateRegistry
tzulitai opened a new pull request #147: URL: https://github.com/apache/flink-statefun/pull/147 Preview of the new documentation:  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-19197) Add documentation for PersistedStateRegistry
Tzu-Li (Gordon) Tai created FLINK-19197:
---
Summary: Add documentation for PersistedStateRegistry
Key: FLINK-19197
URL: https://issues.apache.org/jira/browse/FLINK-19197
Project: Flink
Issue Type: Task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Documentation for the new {{PersistedStateRegistry}} state construct is still
missing. This should be added as a new sub-section beneath
https://ci.apache.org/projects/flink/flink-statefun-docs-master/sdk/java.html#persistence.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-web] xintongsong commented on pull request #378: Add xtsong to the committer list.
xintongsong commented on pull request #378: URL: https://github.com/apache/flink-web/pull/378#issuecomment-690824584 Thanks for the reminder @zhuzhurk, and thanks for the pointer @morsapaes (Marta!) PR updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13179: [FLINK-18978][state-backends] Support full table scan of key and namespace from statebackend
flinkbot edited a comment on pull request #13179: URL: https://github.com/apache/flink/pull/13179#issuecomment-674957571 ## CI report: * 32e7196f7eb7474982fa0e4e6489403316575099 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6454) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13344: [FLINK-10740] FLIP-27 File Source and various Updates on the Split Reader API
flinkbot edited a comment on pull request #13344: URL: https://github.com/apache/flink/pull/13344#issuecomment-688495541 ## CI report: * 15a81e82a4a42cd255baad4ffd3e7e20343b5389 UNKNOWN * e563be1c3fe203184c2f78bd01ed23d35e83198c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6455) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13344: [FLINK-10740] FLIP-27 File Source and various Updates on the Split Reader API
flinkbot edited a comment on pull request #13344: URL: https://github.com/apache/flink/pull/13344#issuecomment-688495541 ## CI report: * 15a81e82a4a42cd255baad4ffd3e7e20343b5389 UNKNOWN * d98718acd049396b6eafd12d81d5404d0393798d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6450) * e563be1c3fe203184c2f78bd01ed23d35e83198c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6455) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13344: [FLINK-10740] FLIP-27 File Source and various Updates on the Split Reader API
flinkbot edited a comment on pull request #13344: URL: https://github.com/apache/flink/pull/13344#issuecomment-688495541 ## CI report: * 15a81e82a4a42cd255baad4ffd3e7e20343b5389 UNKNOWN * d98718acd049396b6eafd12d81d5404d0393798d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6450) * e563be1c3fe203184c2f78bd01ed23d35e83198c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] StephanEwen commented on pull request #13344: [FLINK-10740] FLIP-27 File Source and various Updates on the Split Reader API
StephanEwen commented on pull request #13344: URL: https://github.com/apache/flink/pull/13344#issuecomment-690764909 Thanks a lot for the review! I would proceed to merge the first reviewed commits already, and then open a new PR for the remaining commits. That should make further reviewing a bit easier. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] StephanEwen commented on a change in pull request #13344: [FLINK-10740] FLIP-27 File Source and various Updates on the Split Reader API
StephanEwen commented on a change in pull request #13344:
URL: https://github.com/apache/flink/pull/13344#discussion_r486660080
##
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/SourceReaderBase.java
##
@@ -99,67 +106,94 @@ public SourceReaderBase(
this.splitFetcherManager = splitFetcherManager;
this.recordEmitter = recordEmitter;
this.splitStates = new HashMap<>();
- this.splitIter = null;
this.options = new SourceReaderOptions(config);
this.config = config;
this.context = context;
this.noMoreSplitsAssignment = false;
}
@Override
- public void start() {
-
- }
+ public void start() {}
@Override
public InputStatus pollNext(ReaderOutput output) throws Exception {
splitFetcherManager.checkErrors();
- // poll from the queue if the last element was successfully
handled. Otherwise
- // just pass the last element again.
- RecordsWithSplitIds recordsWithSplitId = null;
- boolean newFetch = splitIter == null || !splitIter.hasNext();
- if (newFetch) {
- recordsWithSplitId = elementsQueue.poll();
+
+ // make sure we have a fetch we are working on, or move to the
next
Review comment:
Thanks :-)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] StephanEwen commented on a change in pull request #13344: [FLINK-10740] FLIP-27 File Source and various Updates on the Split Reader API
StephanEwen commented on a change in pull request #13344:
URL: https://github.com/apache/flink/pull/13344#discussion_r486659810
##
File path:
flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/source/reader/mocks/TestingReaderOutput.java
##
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base.source.reader.mocks;
+
+import org.apache.flink.api.common.eventtime.Watermark;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceOutput;
+
+import java.util.ArrayList;
+
+/**
+ * A {@code ReaderOutput} for testing that collects the emitted records.
+ */
+public class TestingReaderOutput implements ReaderOutput {
Review comment:
I think we can do that, but it would require some more refactoring,
which I would like to not do within this PR, because it is orthogonal.
The approaches are a bit different:
- The `Testing*` classes here accept test data (in the constructor) and
collect results (to be asserted in the test method)
- The `Mock*` classes here generate test data and immediately verify
received records against expected sequences.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] StephanEwen commented on a change in pull request #13344: [FLINK-10740] FLIP-27 File Source and various Updates on the Split Reader API
StephanEwen commented on a change in pull request #13344:
URL: https://github.com/apache/flink/pull/13344#discussion_r486658903
##
File path:
flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/source/reader/SourceReaderBaseUnitTest.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base.source.reader;
+
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.configuration.Configuration;
+import
org.apache.flink.connector.base.source.reader.mocks.PassThroughRecordEmitter;
+import
org.apache.flink.connector.base.source.reader.mocks.TestingReaderContext;
+import org.apache.flink.connector.base.source.reader.mocks.TestingReaderOutput;
+import
org.apache.flink.connector.base.source.reader.mocks.TestingRecordsWithSplitIds;
+import org.apache.flink.connector.base.source.reader.mocks.TestingSourceSplit;
+import org.apache.flink.connector.base.source.reader.mocks.TestingSplitReader;
+import
org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue;
+import
org.apache.flink.connector.base.source.reader.synchronization.FutureNotifier;
+
+import org.junit.Test;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Targeted unit tests for the {@link SourceReaderBase}.
+ */
+public class SourceReaderBaseUnitTest {
+
+ @Test
+ public void recordsWithSplitsNotRecycledWhenRecordsLeft() throws
Exception {
Review comment:
Will do.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] StephanEwen commented on a change in pull request #13344: [FLINK-10740] FLIP-27 File Source and various Updates on the Split Reader API
StephanEwen commented on a change in pull request #13344: URL: https://github.com/apache/flink/pull/13344#discussion_r486658430 ## File path: flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/fetcher/SplitFetcherTask.java ## @@ -32,7 +34,7 @@ * @return whether the runnable has successfully finished running. * @throws InterruptedException when interrupted. */ Review comment: Thanks, will add this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13010: [FLINK-18735][table] Add support for more types to DataGen source
flinkbot edited a comment on pull request #13010: URL: https://github.com/apache/flink/pull/13010#issuecomment-665174560 ## CI report: * cbe73382b1eb400653d068a8726bc673fb7e6394 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6453) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13179: [FLINK-18978][state-backends] Support full table scan of key and namespace from statebackend
flinkbot edited a comment on pull request #13179: URL: https://github.com/apache/flink/pull/13179#issuecomment-674957571 ## CI report: * b985e226543345724377647b73884a2527ac5b30 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6406) * 32e7196f7eb7474982fa0e4e6489403316575099 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6454) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13010: [FLINK-18735][table] Add support for more types to DataGen source
flinkbot edited a comment on pull request #13010: URL: https://github.com/apache/flink/pull/13010#issuecomment-665174560 ## CI report: * a96948229811ec6f8b72a2fd2ad019a9fa6b2d67 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6407) * cbe73382b1eb400653d068a8726bc673fb7e6394 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6453) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13179: [FLINK-18978][state-backends] Support full table scan of key and namespace from statebackend
flinkbot edited a comment on pull request #13179: URL: https://github.com/apache/flink/pull/13179#issuecomment-674957571 ## CI report: * b985e226543345724377647b73884a2527ac5b30 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6406) * 32e7196f7eb7474982fa0e4e6489403316575099 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13010: [FLINK-18735][table] Add support for more types to DataGen source
flinkbot edited a comment on pull request #13010: URL: https://github.com/apache/flink/pull/13010#issuecomment-665174560 ## CI report: * a96948229811ec6f8b72a2fd2ad019a9fa6b2d67 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6407) * cbe73382b1eb400653d068a8726bc673fb7e6394 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #13010: [FLINK-18735][table] Add support for more types to DataGen source
sjwiesman commented on pull request #13010: URL: https://github.com/apache/flink/pull/13010#issuecomment-690711964 Sorry for the long back and forth, just pushed a fix addressing your comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-19196) FlinkSQL aggregation function to aggregate array typed column to Map/Multiset
sam lin created FLINK-19196:
---
Summary: FlinkSQL aggregation function to aggregate array typed
column to Map/Multiset
Key: FLINK-19196
URL: https://issues.apache.org/jira/browse/FLINK-19196
Project: Flink
Issue Type: New Feature
Components: Table SQL / API
Affects Versions: 1.11.1
Reporter: sam lin
Hi,
For FlinkSQL, how do I aggregate an array typed column to a Map/Multiset type?
e.g. Given the source events:
userID, arr_types
1, [a, b]
2, [a]
1, [b, c]
```
SELECT userID, collect_arr(arr_types) FROM src_table GROUP BY userID
```
Using the above SQL could produce results like:
1, \{a: 1, b: 2, c:1}
2, \{a: 1}
My question is do we have existing UDF to do that? If not, what is the
recommended way to achieve this? I'm happy to write a UDF if you could provide
some code pointers. Thanks in advance!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13344: [FLINK-10740] FLIP-27 File Source and various Updates on the Split Reader API
flinkbot edited a comment on pull request #13344: URL: https://github.com/apache/flink/pull/13344#issuecomment-688495541 ## CI report: * 15a81e82a4a42cd255baad4ffd3e7e20343b5389 UNKNOWN * d98718acd049396b6eafd12d81d5404d0393798d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6450) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13344: [FLINK-10740] FLIP-27 File Source and various Updates on the Split Reader API
flinkbot edited a comment on pull request #13344: URL: https://github.com/apache/flink/pull/13344#issuecomment-688495541 ## CI report: * 15a81e82a4a42cd255baad4ffd3e7e20343b5389 UNKNOWN * 43239addb3380c94c2c3452557b8ecb1f5982e1b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6438) * d98718acd049396b6eafd12d81d5404d0393798d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6450) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13343: [FLINK-19083] Remove deprecated DataStream#split
flinkbot edited a comment on pull request #13343: URL: https://github.com/apache/flink/pull/13343#issuecomment-68843 ## CI report: * 83007c1bc19b10cf5369b0e4718979d71e674fac Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6448) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13344: [FLINK-10740] FLIP-27 File Source and various Updates on the Split Reader API
flinkbot edited a comment on pull request #13344: URL: https://github.com/apache/flink/pull/13344#issuecomment-688495541 ## CI report: * 15a81e82a4a42cd255baad4ffd3e7e20343b5389 UNKNOWN * 43239addb3380c94c2c3452557b8ecb1f5982e1b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6438) * d98718acd049396b6eafd12d81d5404d0393798d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13284: [FLINK-17016][runtime] Integrate pipelined region scheduling
flinkbot edited a comment on pull request #13284: URL: https://github.com/apache/flink/pull/13284#issuecomment-683683000 ## CI report: * 6af35bfdfa14c049bf4abe1c88ee49a689f38085 UNKNOWN * acc8ba0856f79a5e7915e8612ef0789190a3806d UNKNOWN * b50f9615fa9ee90ce66d44ee70e14e9d4c780e0c UNKNOWN * a687e9ac807a260103dc3249c962890571c4ecda Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6447) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13311: [FLINK-18721] Migrate YarnResourceManager to the new YarnResourceManagerDriver
flinkbot edited a comment on pull request #13311: URL: https://github.com/apache/flink/pull/13311#issuecomment-686196434 ## CI report: * b7660009b48d48261b767855d8973c63741ab493 UNKNOWN * f16d78f22d3d66e763f5629a377936bcb20b80c4 UNKNOWN * 97cefe9ca646f66c5b0fa09dead666d1fe468029 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6443) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19195) question on security vulnerabilities in flink
[ https://issues.apache.org/jira/browse/FLINK-19195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miguel Costa updated FLINK-19195: - Issue Type: Bug (was: Wish) > question on security vulnerabilities in flink > - > > Key: FLINK-19195 > URL: https://issues.apache.org/jira/browse/FLINK-19195 > Project: Flink > Issue Type: Bug > Components: flink-docker >Affects Versions: docker-1.11.0.0 >Reporter: Miguel Costa >Priority: Major > Labels: security > Fix For: 1.12.0 > > Original Estimate: 48h > Remaining Estimate: 48h > > Hi All, > Sorry if this is the wrong place but I was in github, website and other > places and I could not find what I was looking for. > I'm starting to learn about flink and I'm using this image for some of my > explorations: > docker pull amd64/flink:1.11-scala_2.11-java11 > > I'm using it in our development cluster in my company and when generating my > image based on this I get some errors from the security report (from an > external provider) that prevent me from generating an image (it's something > on our side). > > I just wanted to know if this is indeed an error and it could be fixed in the > future. > This is what I got: > CVE Package Version Severity Status CVSS > — --- --- -- > CVE-2019-20444 io.netty_netty-codec 4.1.34.Final critical fixed in 4.1.44 9.1 > CVE-2019-20445 io.netty_netty-codec 4.1.34.Final critical fixed in 4.1.44 9.1 > CVE-2020-11612 io.netty_netty-codec 4.1.34.Final critical fixed in 4.1.46 9.8 > CVE-2019-16869 io.netty_netty-codec 4.1.34.Final high fixed in 4.1.42.Final > 7.5 > > CVE-2019-20444 and CVE-2019-20445 in theory was fixed in FLINK-16961 but I > still see it in my report. > > CVE-2020-11612 and CVE-2019-16869 I found it in FLINK-16356 but this one is > still open. > So I was just wondering if maybe FLINK-16961 fixed only some of the > components but some others are still being used? > If I searched in github I found this problematic versions in: > flink-connector-cassandra (io.netty:netty-codec:4.1.44.Final) > flink-connector-elasticsearch5 (io.netty:netty-codec:4.1.44.Final) > flink-python - (io.netty:netty-codec:4.1.42.Final) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19195) question on security vulnerabilities in flink
Miguel Costa created FLINK-19195: Summary: question on security vulnerabilities in flink Key: FLINK-19195 URL: https://issues.apache.org/jira/browse/FLINK-19195 Project: Flink Issue Type: Wish Components: flink-docker Affects Versions: docker-1.11.0.0 Reporter: Miguel Costa Fix For: 1.12.0 Hi All, Sorry if this is the wrong place but I was in github, website and other places and I could not find what I was looking for. I'm starting to learn about flink and I'm using this image for some of my explorations: docker pull amd64/flink:1.11-scala_2.11-java11 I'm using it in our development cluster in my company and when generating my image based on this I get some errors from the security report (from an external provider) that prevent me from generating an image (it's something on our side). I just wanted to know if this is indeed an error and it could be fixed in the future. This is what I got: CVE Package Version Severity Status CVSS — --- --- -- CVE-2019-20444 io.netty_netty-codec 4.1.34.Final critical fixed in 4.1.44 9.1 CVE-2019-20445 io.netty_netty-codec 4.1.34.Final critical fixed in 4.1.44 9.1 CVE-2020-11612 io.netty_netty-codec 4.1.34.Final critical fixed in 4.1.46 9.8 CVE-2019-16869 io.netty_netty-codec 4.1.34.Final high fixed in 4.1.42.Final 7.5 CVE-2019-20444 and CVE-2019-20445 in theory was fixed in FLINK-16961 but I still see it in my report. CVE-2020-11612 and CVE-2019-16869 I found it in FLINK-16356 but this one is still open. So I was just wondering if maybe FLINK-16961 fixed only some of the components but some others are still being used? If I searched in github I found this problematic versions in: flink-connector-cassandra (io.netty:netty-codec:4.1.44.Final) flink-connector-elasticsearch5 (io.netty:netty-codec:4.1.44.Final) flink-python - (io.netty:netty-codec:4.1.42.Final) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13344: [FLINK-10740] FLIP-27 File Source and various Updates on the Split Reader API
flinkbot edited a comment on pull request #13344: URL: https://github.com/apache/flink/pull/13344#issuecomment-688495541 ## CI report: * 15a81e82a4a42cd255baad4ffd3e7e20343b5389 UNKNOWN * 43239addb3380c94c2c3452557b8ecb1f5982e1b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6438) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] becketqin commented on a change in pull request #13344: [FLINK-10740] FLIP-27 File Source and various Updates on the Split Reader API
becketqin commented on a change in pull request #13344:
URL: https://github.com/apache/flink/pull/13344#discussion_r486316183
##
File path:
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/fetcher/SplitFetcherTask.java
##
@@ -32,7 +34,7 @@
* @return whether the runnable has successfully finished running.
* @throws InterruptedException when interrupted.
*/
Review comment:
nit: missing `@throws IOException`.
##
File path:
flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/source/reader/SourceReaderBaseUnitTest.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base.source.reader;
+
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.configuration.Configuration;
+import
org.apache.flink.connector.base.source.reader.mocks.PassThroughRecordEmitter;
+import
org.apache.flink.connector.base.source.reader.mocks.TestingReaderContext;
+import org.apache.flink.connector.base.source.reader.mocks.TestingReaderOutput;
+import
org.apache.flink.connector.base.source.reader.mocks.TestingRecordsWithSplitIds;
+import org.apache.flink.connector.base.source.reader.mocks.TestingSourceSplit;
+import org.apache.flink.connector.base.source.reader.mocks.TestingSplitReader;
+import
org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue;
+import
org.apache.flink.connector.base.source.reader.synchronization.FutureNotifier;
+
+import org.junit.Test;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Targeted unit tests for the {@link SourceReaderBase}.
+ */
+public class SourceReaderBaseUnitTest {
+
+ @Test
+ public void recordsWithSplitsNotRecycledWhenRecordsLeft() throws
Exception {
Review comment:
nit: it seems the test methods all start with "test". Should this be
`testRecordsWithSplitsNotRecycledWhenRecordsLeft`?
##
File path:
flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/source/reader/SourceReaderBaseUnitTest.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base.source.reader;
+
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.configuration.Configuration;
+import
org.apache.flink.connector.base.source.reader.mocks.PassThroughRecordEmitter;
+import
org.apache.flink.connector.base.source.reader.mocks.TestingReaderContext;
+import org.apache.flink.connector.base.source.reader.mocks.TestingReaderOutput;
+import
org.apache.flink.connector.base.source.reader.mocks.TestingRecordsWithSplitIds;
+import org.apache.flink.connector.base.source.reader.mocks.TestingSourceSplit;
+import org.apache.flink.connector.base.source.reader.mocks.TestingSplitReader;
+import
org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue;
+import
org.apache.flink.connector.base.source.reader.synchronization.FutureNotifier;
+
+import org.junit.Test;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+
+import static org.junit.Assert.assertFalse;
+import static
[jira] [Commented] (FLINK-19109) Split Reader eats chained periodic watermarks
[
https://issues.apache.org/jira/browse/FLINK-19109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193662#comment-17193662
]
Arvid Heise commented on FLINK-19109:
-
Yes, it's possible to contain the fix to just event time characteristics. I'll
prepare a PR soonish.
> Split Reader eats chained periodic watermarks
> -
>
> Key: FLINK-19109
> URL: https://issues.apache.org/jira/browse/FLINK-19109
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.10.2, 1.11.1
>Reporter: David Anderson
>Assignee: Roman Khachatryan
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2
>
>
> Attempting to generate watermarks chained to the Split Reader /
> ContinuousFileReaderOperator, as in
> {code:java}
> SingleOutputStreamOperator results = env
> .readTextFile(...)
> .map(...)
> .assignTimestampsAndWatermarks(bounded)
> .keyBy(...)
> .process(...);{code}
> leads to the Watermarks failing to be produced. Breaking the chain, via
> {{disableOperatorChaining()}} or a {{rebalance}}, works around the bug. Using
> punctuated watermarks also avoids the issue.
> Looking at this in the debugger reveals that timer service is being
> prematurely quiesced.
> In many respects this is FLINK-7666 brought back to life.
> The problem is not present in 1.9.3.
> There's a minimal reproducible example in
> [https://github.com/alpinegizmo/flink-question-001/tree/bug].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19135) (Stream)ExecutionEnvironment.execute() should not throw ExecutionException
[
https://issues.apache.org/jira/browse/FLINK-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-19135:
-
Fix Version/s: 1.12.0
> (Stream)ExecutionEnvironment.execute() should not throw ExecutionException
> --
>
> Key: FLINK-19135
> URL: https://issues.apache.org/jira/browse/FLINK-19135
> Project: Flink
> Issue Type: Bug
> Components: API / DataSet, API / DataStream
>Affects Versions: 1.10.0, 1.11.0, 1.12.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> In FLINK-14850 we changed the {{execute()}} method to be basically
> {code}
> final JobClient jobClient = executeAsync(...);
> return jobClient.getJobExecutionResult(userClassloader).get();
> {code}
> Unfortunately, this means that {{execute()}} now throws an
> {{ExecutionException}} instead of a {{ProgramInvocationException}} or
> {{JobExecutionException}} as before. The {{ExecutionException}} is wrapping
> the other exceptions that we were throwing before.
> We didn't notice this in tests because most tests use
> {{Test(Stream)Environment}} which overrides the {{execute()}} method and so
> doesn't go through the {{PipelineExecutor}} logic or the normal code path of
> delegating to {{executeAsync()}}.
> We should fix this to go back to the old behaviour.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19135) (Stream)ExecutionEnvironment.execute() should not throw ExecutionException
[
https://issues.apache.org/jira/browse/FLINK-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193659#comment-17193659
]
Aljoscha Krettek commented on FLINK-19135:
--
master: faca0e69d90162cc1b24b8f5865f1a8a604f355e
> (Stream)ExecutionEnvironment.execute() should not throw ExecutionException
> --
>
> Key: FLINK-19135
> URL: https://issues.apache.org/jira/browse/FLINK-19135
> Project: Flink
> Issue Type: Bug
> Components: API / DataSet, API / DataStream
>Affects Versions: 1.10.0, 1.11.0, 1.12.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Labels: pull-request-available
>
> In FLINK-14850 we changed the {{execute()}} method to be basically
> {code}
> final JobClient jobClient = executeAsync(...);
> return jobClient.getJobExecutionResult(userClassloader).get();
> {code}
> Unfortunately, this means that {{execute()}} now throws an
> {{ExecutionException}} instead of a {{ProgramInvocationException}} or
> {{JobExecutionException}} as before. The {{ExecutionException}} is wrapping
> the other exceptions that we were throwing before.
> We didn't notice this in tests because most tests use
> {{Test(Stream)Environment}} which overrides the {{execute()}} method and so
> doesn't go through the {{PipelineExecutor}} logic or the normal code path of
> delegating to {{executeAsync()}}.
> We should fix this to go back to the old behaviour.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18980) "Avro Confluent Schema Registry nightly end-to-end test" hangs
[
https://issues.apache.org/jira/browse/FLINK-18980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193656#comment-17193656
]
Robert Metzger commented on FLINK-18980:
Added debugging messages
https://github.com/apache/flink/commit/92e2f3b12165cfc0d0c5bc96656e87029ee5e694
> "Avro Confluent Schema Registry nightly end-to-end test" hangs
> --
>
> Key: FLINK-18980
> URL: https://issues.apache.org/jira/browse/FLINK-18980
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.0
>Reporter: Dian Fu
>Assignee: Robert Metzger
>Priority: Major
> Labels: pull-request-available, test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5629=logs=08866332-78f7-59e4-4f7e-49a56faa3179=3e8647c1-5a28-5917-dd93-bf78594ea994]
> {code}
> 2020-08-17T22:10:48.3400852Z Job has been submitted with JobID
> d9a2a95d9204d149165af0e0a1a7c488
> 2020-08-17T22:10:49.1383170Z [2020-08-17 22:10:49,137] INFO 127.0.0.1 - -
> [17/Aug/2020:22:10:49 +] "GET /schemas/ids/1 HTTP/1.1" 200 389 6
> (io.confluent.rest-utils.requests:77)
> 2020-08-17T22:10:51.8854897Z [2020-08-17 22:10:49,256] INFO Wait to catch up
> until the offset of the last message at 3
> (io.confluent.kafka.schemaregistry.storage.KafkaStore:356)
> 2020-08-17T22:10:51.8856950Z [2020-08-17 22:10:49,775] INFO 127.0.0.1 - -
> [17/Aug/2020:22:10:49 +] "POST /subjects/test-output-subject/versions
> HTTP/1.1" 500 61 524 (io.confluent.rest-utils.requests:77)
> 2020-08-18T00:20:27.5068516Z ##[error]The operation was canceled.
> 2020-08-18T00:20:27.5085465Z ##[section]Finishing: Run e2e tests
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rmetzger merged pull request #13359: [FLINK-18980][e2e] Add timeout to get logs from stalling test
rmetzger merged pull request #13359: URL: https://github.com/apache/flink/pull/13359 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #13359: [FLINK-18980][e2e] Add timeout to get logs from stalling test
rmetzger commented on pull request #13359: URL: https://github.com/apache/flink/pull/13359#issuecomment-690335967 Thanks a lot for your review! Merging ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-19109) Split Reader eats chained periodic watermarks
[
https://issues.apache.org/jira/browse/FLINK-19109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193526#comment-17193526
]
Piotr Nowojski edited comment on FLINK-19109 at 9/10/20, 2:32 PM:
--
As I wrote, I think we can not backport the proper fix. [~roman_khachatryan]'s
fix looks like can deadlock. Maybe it could be solved, but it's still hacky and
risky. If you insist on fixing it in 1.10, I would suggest to disable chaining
for the {{ContinoutFileReaderOperator}}.
was (Author: pnowojski):
As I wrote, I don't think we can not backport the proper fix.
[~roman_khachatryan]'s fix looks like can deadlock. Maybe it could be solved,
but it's still hacky and risky. If you insist on fixing it in 1.10, I would
suggest to disable chaining for the {{ContinoutFileReaderOperator}}.
> Split Reader eats chained periodic watermarks
> -
>
> Key: FLINK-19109
> URL: https://issues.apache.org/jira/browse/FLINK-19109
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.10.2, 1.11.1
>Reporter: David Anderson
>Assignee: Roman Khachatryan
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2
>
>
> Attempting to generate watermarks chained to the Split Reader /
> ContinuousFileReaderOperator, as in
> {code:java}
> SingleOutputStreamOperator results = env
> .readTextFile(...)
> .map(...)
> .assignTimestampsAndWatermarks(bounded)
> .keyBy(...)
> .process(...);{code}
> leads to the Watermarks failing to be produced. Breaking the chain, via
> {{disableOperatorChaining()}} or a {{rebalance}}, works around the bug. Using
> punctuated watermarks also avoids the issue.
> Looking at this in the debugger reveals that timer service is being
> prematurely quiesced.
> In many respects this is FLINK-7666 brought back to life.
> The problem is not present in 1.9.3.
> There's a minimal reproducible example in
> [https://github.com/alpinegizmo/flink-question-001/tree/bug].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-19192) Set higher limit on the HTTP connection pool
[ https://issues.apache.org/jira/browse/FLINK-19192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai closed FLINK-19192. --- Fix Version/s: statefun-2.2.0 Resolution: Fixed Fixed for statefun-master via b86daab2b2836a98fe8cd1fbd9288db50fe71811 > Set higher limit on the HTTP connection pool > > > Key: FLINK-19192 > URL: https://issues.apache.org/jira/browse/FLINK-19192 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Reporter: Igal Shilman >Assignee: Igal Shilman >Priority: Major > Labels: pull-request-available > Fix For: statefun-2.2.0 > > > The default size of the connection pool is too low, we should set it to a > higher value > and let servers to decide if they will keep the connection alive or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-19191) Reduce the default number for async operations
[ https://issues.apache.org/jira/browse/FLINK-19191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai closed FLINK-19191. --- Fix Version/s: statefun-2.2.0 Resolution: Fixed Fixed for statefun-master via 80aa4ae94ca74d83c36b383ab1d7342a48a96c12 > Reduce the default number for async operations > --- > > Key: FLINK-19191 > URL: https://issues.apache.org/jira/browse/FLINK-19191 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Reporter: Igal Shilman >Assignee: Igal Shilman >Priority: Major > Labels: pull-request-available > Fix For: statefun-2.2.0 > > > The default upper limit for async operations per task slot is currently set > to 10 million, > and it is unrealistically high, we should set it to a more realistic value. A > closer example would be the recommend value in Flink's AsyncWait operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-19190) Use camelCase for metric names
[ https://issues.apache.org/jira/browse/FLINK-19190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai closed FLINK-19190. --- Fix Version/s: statefun-2.2.0 Resolution: Fixed Merged for statefun-master via e35ba2b73270dae28247ccaee4fac9b5fc7b1528 > Use camelCase for metric names > -- > > Key: FLINK-19190 > URL: https://issues.apache.org/jira/browse/FLINK-19190 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Reporter: Igal Shilman >Assignee: Igal Shilman >Priority: Major > Labels: pull-request-available > Fix For: statefun-2.2.0 > > > Rename StateFun metrics to use camelCase instead of a snake-case, to conform > with Flink metrics convention. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-statefun] tzulitai closed pull request #146: [FLINK-19192] Set HTTP connection pool size to 1024
tzulitai closed pull request #146: URL: https://github.com/apache/flink-statefun/pull/146 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] tzulitai closed pull request #145: [FLINK-19191] Set the default maxInflightAsyncOperations to 1024
tzulitai closed pull request #145: URL: https://github.com/apache/flink-statefun/pull/145 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] tzulitai closed pull request #144: [FLINK-19190] Use camelCase for metric names
tzulitai closed pull request #144: URL: https://github.com/apache/flink-statefun/pull/144 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13228: [FLINK-19026][network] Improve threading model of CheckpointBarrierUnaligner
flinkbot edited a comment on pull request #13228: URL: https://github.com/apache/flink/pull/13228#issuecomment-679099456 ## CI report: * d141f10ef060063162dc073b1cd66729f5f75a3b UNKNOWN * 89b0550c7b223f304b67a6f64dfc4252509eb76e UNKNOWN * a6a2d36afcce1758e885f0751359ccf035ae2378 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6446) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13352: [FLINK-19092][sql-parser] Parse comment on computed column failed
flinkbot edited a comment on pull request #13352: URL: https://github.com/apache/flink/pull/13352#issuecomment-688773679 ## CI report: * 7074c013502a3b3d966e102bfdbc164cb3ba6ef1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6436) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13343: [FLINK-19083] Remove deprecated DataStream#split
flinkbot edited a comment on pull request #13343: URL: https://github.com/apache/flink/pull/13343#issuecomment-68843 ## CI report: * 3132c72413a7a8b41e06a1e93ffea94005a38493 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6329) * 83007c1bc19b10cf5369b0e4718979d71e674fac Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6448) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13284: [FLINK-17016][runtime] Integrate pipelined region scheduling
flinkbot edited a comment on pull request #13284: URL: https://github.com/apache/flink/pull/13284#issuecomment-683683000 ## CI report: * 6af35bfdfa14c049bf4abe1c88ee49a689f38085 UNKNOWN * acc8ba0856f79a5e7915e8612ef0789190a3806d UNKNOWN * b50f9615fa9ee90ce66d44ee70e14e9d4c780e0c UNKNOWN * e62b16868e32d87e90e0133fd7158132c408e615 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6435) * a687e9ac807a260103dc3249c962890571c4ecda Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6447) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] morsapaes edited a comment on pull request #378: Add xtsong to the committer list.
morsapaes edited a comment on pull request #378: URL: https://github.com/apache/flink-web/pull/378#issuecomment-690302055 Hey, @xintongsong (Tony!). This should only contain the markdown files you're touching and not any translated files, as outlined in the [contribution guidelines](https://flink.apache.org/contributing/improve-website.html#submit-your-contribution). Whoever takes care of the PR should then rebuild the website when merging! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] morsapaes edited a comment on pull request #378: Add xtsong to the committer list.
morsapaes edited a comment on pull request #378: URL: https://github.com/apache/flink-web/pull/378#issuecomment-690302055 Hey, @xintongsong (Tony!). This should only contain the markdown files you're touching and not any translated files, as outlined in the [contribution guidelines](https://flink.apache.org/contributing/improve-website.html#submit-your-contribution). Whoever takes care of the PR should then rebuild the website! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] morsapaes commented on pull request #378: Add xtsong to the committer list.
morsapaes commented on pull request #378: URL: https://github.com/apache/flink-web/pull/378#issuecomment-690302055 Hey, @xintongsong (Tony!). The PR should only contain the markdown files you're touching and not any translated files, as outlined in the [contribution guidelines](https://flink.apache.org/contributing/improve-website.html#submit-your-contribution). Whoever takes care of the commit should then rebuild the website! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19109) Split Reader eats chained periodic watermarks
[
https://issues.apache.org/jira/browse/FLINK-19109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193629#comment-17193629
]
David Anderson commented on FLINK-19109:
I like [~pnowojski]'s idea of disabling chaining for the
{{ContinousFileReaderOperator}} in 1.10. This strikes me as a pragmatic
solution, though it will impact performance for some users who aren't impacted
by the bug. Would it make sense to try to optimize this a bit, and only disable
chaining when the time characteristic is event time?
> Split Reader eats chained periodic watermarks
> -
>
> Key: FLINK-19109
> URL: https://issues.apache.org/jira/browse/FLINK-19109
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.10.2, 1.11.1
>Reporter: David Anderson
>Assignee: Roman Khachatryan
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2
>
>
> Attempting to generate watermarks chained to the Split Reader /
> ContinuousFileReaderOperator, as in
> {code:java}
> SingleOutputStreamOperator results = env
> .readTextFile(...)
> .map(...)
> .assignTimestampsAndWatermarks(bounded)
> .keyBy(...)
> .process(...);{code}
> leads to the Watermarks failing to be produced. Breaking the chain, via
> {{disableOperatorChaining()}} or a {{rebalance}}, works around the bug. Using
> punctuated watermarks also avoids the issue.
> Looking at this in the debugger reveals that timer service is being
> prematurely quiesced.
> In many respects this is FLINK-7666 brought back to life.
> The problem is not present in 1.9.3.
> There's a minimal reproducible example in
> [https://github.com/alpinegizmo/flink-question-001/tree/bug].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13343: [FLINK-19083] Remove deprecated DataStream#split
flinkbot edited a comment on pull request #13343: URL: https://github.com/apache/flink/pull/13343#issuecomment-68843 ## CI report: * 3132c72413a7a8b41e06a1e93ffea94005a38493 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6329) * 83007c1bc19b10cf5369b0e4718979d71e674fac UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13284: [FLINK-17016][runtime] Integrate pipelined region scheduling
flinkbot edited a comment on pull request #13284: URL: https://github.com/apache/flink/pull/13284#issuecomment-683683000 ## CI report: * 6af35bfdfa14c049bf4abe1c88ee49a689f38085 UNKNOWN * acc8ba0856f79a5e7915e8612ef0789190a3806d UNKNOWN * 95fb9ba421b50fc89e8f3e84f30386eed4082bcf Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6383) * b50f9615fa9ee90ce66d44ee70e14e9d4c780e0c UNKNOWN * e62b16868e32d87e90e0133fd7158132c408e615 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6435) * a687e9ac807a260103dc3249c962890571c4ecda Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6447) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13371: [FLINK-17879][python] update Python row serializer to support RowKind
flinkbot edited a comment on pull request #13371: URL: https://github.com/apache/flink/pull/13371#issuecomment-690084042 ## CI report: * a6e02f1a5e6fb8cc6dc6c62510df64af5b02d8e9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6431) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13284: [FLINK-17016][runtime] Integrate pipelined region scheduling
flinkbot edited a comment on pull request #13284: URL: https://github.com/apache/flink/pull/13284#issuecomment-683683000 ## CI report: * 6af35bfdfa14c049bf4abe1c88ee49a689f38085 UNKNOWN * acc8ba0856f79a5e7915e8612ef0789190a3806d UNKNOWN * 95fb9ba421b50fc89e8f3e84f30386eed4082bcf Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6383) * b50f9615fa9ee90ce66d44ee70e14e9d4c780e0c UNKNOWN * e62b16868e32d87e90e0133fd7158132c408e615 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6435) * a687e9ac807a260103dc3249c962890571c4ecda UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13228: [FLINK-19026][network] Improve threading model of CheckpointBarrierUnaligner
flinkbot edited a comment on pull request #13228: URL: https://github.com/apache/flink/pull/13228#issuecomment-679099456 ## CI report: * d141f10ef060063162dc073b1cd66729f5f75a3b UNKNOWN * 407169bc192b5d05bfba2c264849f1a6e7b81415 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6402) * 89b0550c7b223f304b67a6f64dfc4252509eb76e UNKNOWN * a6a2d36afcce1758e885f0751359ccf035ae2378 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6446) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13228: [FLINK-19026][network] Improve threading model of CheckpointBarrierUnaligner
flinkbot edited a comment on pull request #13228: URL: https://github.com/apache/flink/pull/13228#issuecomment-679099456 ## CI report: * d141f10ef060063162dc073b1cd66729f5f75a3b UNKNOWN * 407169bc192b5d05bfba2c264849f1a6e7b81415 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6402) * 89b0550c7b223f304b67a6f64dfc4252509eb76e UNKNOWN * a6a2d36afcce1758e885f0751359ccf035ae2378 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] zhuzhurk commented on pull request #378: Add xtsong to the committer list.
zhuzhurk commented on pull request #378: URL: https://github.com/apache/flink-web/pull/378#issuecomment-690248718 The rebuilt web should be in a separate commit I think. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19194) The UDF split and split_index get wrong result
[
https://issues.apache.org/jira/browse/FLINK-19194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fa zheng updated FLINK-19194:
-
Description:
If we run sql
{code:sql}
select split_index('ab-123"xyz\cd','-',1) ;
{code}
or
{code:sql}
select split('ab-123"xyz\cd','-')[2] ;
{code}
{noformat}
in sql-client, we should get result 123"xyz\cd, not 123\"xyz\\cd
{noformat}
was:
If we run sql
{code:sql}
select split_index('ab-123"xyz\cd','-',1) ;
{code}
or
{code:sql}
select split('ab-123"xyz\cd','-')[2] ;
{code}
in sql-client, we should get result 123"xyz\cd, not 123\"xyz\\cd
> The UDF split and split_index get wrong result
> ---
>
> Key: FLINK-19194
> URL: https://issues.apache.org/jira/browse/FLINK-19194
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.10.0, 1.10.1
>Reporter: fa zheng
>Priority: Major
>
> If we run sql
> {code:sql}
> select split_index('ab-123"xyz\cd','-',1) ;
> {code}
> or
> {code:sql}
> select split('ab-123"xyz\cd','-')[2] ;
> {code}
> {noformat}
> in sql-client, we should get result 123"xyz\cd, not 123\"xyz\\cd
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13311: [FLINK-18721] Migrate YarnResourceManager to the new YarnResourceManagerDriver
flinkbot edited a comment on pull request #13311: URL: https://github.com/apache/flink/pull/13311#issuecomment-686196434 ## CI report: * b7660009b48d48261b767855d8973c63741ab493 UNKNOWN * f16d78f22d3d66e763f5629a377936bcb20b80c4 UNKNOWN * 507f270bfbac225cafa7e25d4ab60b1c74b66c74 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6390) * 97cefe9ca646f66c5b0fa09dead666d1fe468029 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6443) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19193) Recommend stop-with-savepoint in upgrade guidelines
[ https://issues.apache.org/jira/browse/FLINK-19193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed FLINK-19193. Resolution: Fixed master: dc8dd211acdea05758050a1b8efd218aeddf402f release-1.11: 0d709393a4a0cbabc7367d085bb42f7ff5d338bf > Recommend stop-with-savepoint in upgrade guidelines > --- > > Key: FLINK-19193 > URL: https://issues.apache.org/jira/browse/FLINK-19193 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Major > Fix For: 1.12.0, 1.11.2 > > > This is about step one in the documentation here: > https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/upgrading.html#step-1-take-a-savepoint-in-the-old-flink-version > We currently advise users to take a savepoint, without telling them to stop > or cancel the job afterwards. We should update this to suggest stopping the > job with a savepoint in step one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19194) The UDF split and split_index get wrong result
fa zheng created FLINK-19194:
Summary: The UDF split and split_index get wrong result
Key: FLINK-19194
URL: https://issues.apache.org/jira/browse/FLINK-19194
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.10.1, 1.10.0
Reporter: fa zheng
If we run sql
{code:sql}
select split_index('ab-123"xyz\cd','-',1) ;
{code}
or
{code:sql}
select split('ab-123"xyz\cd','-')[2] ;
{code}
in sql-client, we should get result 123"xyz\cd, not 123\"xyz\\cd
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19193) Recommend stop-with-savepoint in upgrade guidelines
[ https://issues.apache.org/jira/browse/FLINK-19193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-19193: - Summary: Recommend stop-with-savepoint in upgrade guidelines (was: Update upgrade guidelines to use stop-with-savepoint) > Recommend stop-with-savepoint in upgrade guidelines > --- > > Key: FLINK-19193 > URL: https://issues.apache.org/jira/browse/FLINK-19193 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Major > Fix For: 1.12.0, 1.11.2 > > > This is about step one in the documentation here: > https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/upgrading.html#step-1-take-a-savepoint-in-the-old-flink-version > We currently advise users to take a savepoint, without telling them to stop > or cancel the job afterwards. We should update this to suggest stopping the > job with a savepoint in step one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19193) Update upgrade guidelines to use stop-with-savepoint
[ https://issues.apache.org/jira/browse/FLINK-19193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-19193: - Summary: Update upgrade guidelines to use stop-with-savepoint (was: Upgrade migration guidelines to use stop-with-savepoint) > Update upgrade guidelines to use stop-with-savepoint > > > Key: FLINK-19193 > URL: https://issues.apache.org/jira/browse/FLINK-19193 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Major > Fix For: 1.12.0, 1.11.2 > > > This is about step one in the documentation here: > https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/upgrading.html#step-1-take-a-savepoint-in-the-old-flink-version > We currently advise users to take a savepoint, without telling them to stop > or cancel the job afterwards. We should update this to suggest stopping the > job with a savepoint in step one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] Tartarus0zm commented on a change in pull request #13319: [FLINK-19022][runtime]Register the TerminationFuture of ResourceManager and Dispatcher with DispatcherResourceManagerComponen
Tartarus0zm commented on a change in pull request #13319:
URL: https://github.com/apache/flink/pull/13319#discussion_r486286684
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/component/DispatcherResourceManagerComponent.java
##
@@ -88,6 +95,18 @@
private void registerShutDownFuture() {
FutureUtils.forward(dispatcherRunner.getShutDownFuture(),
shutDownFuture);
+ BiConsumer terminateAction = (ignored,
throwable) -> {
+ if (throwable != null) {
+ shutDownFuture.completeExceptionally(throwable);
+ } else {
+
shutDownFuture.complete(ApplicationStatus.SUCCEEDED);
+ }
+ if (isRunning.get()) {
+ fatalErrorHandler.onFatalError(throwable);
+ }
+ };
+
dispatcherRunner.getTerminationFuture().whenComplete(terminateAction);
Review comment:
@rmetzger thanks, I try to solve it myself
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-19193) Upgrade migration guidelines to use stop-with-savepoint
[ https://issues.apache.org/jira/browse/FLINK-19193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-19193: - Fix Version/s: 1.11.2 1.12.0 > Upgrade migration guidelines to use stop-with-savepoint > --- > > Key: FLINK-19193 > URL: https://issues.apache.org/jira/browse/FLINK-19193 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Major > Fix For: 1.12.0, 1.11.2 > > > This is about step one in the documentation here: > https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/upgrading.html#step-1-take-a-savepoint-in-the-old-flink-version > We currently advise users to take a savepoint, without telling them to stop > or cancel the job afterwards. We should update this to suggest stopping the > job with a savepoint in step one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19193) Upgrade migration guidelines to use stop-with-savepoint
Aljoscha Krettek created FLINK-19193: Summary: Upgrade migration guidelines to use stop-with-savepoint Key: FLINK-19193 URL: https://issues.apache.org/jira/browse/FLINK-19193 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Aljoscha Krettek Assignee: Aljoscha Krettek This is about step one in the documentation here: https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/upgrading.html#step-1-take-a-savepoint-in-the-old-flink-version We currently advise users to take a savepoint, without telling them to stop or cancel the job afterwards. We should update this to suggest stopping the job with a savepoint in step one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] aljoscha closed pull request #13354: [FLINK-19135] Strip ExecutionException in (Stream)ExecutionEnvironment.execute()
aljoscha closed pull request #13354: URL: https://github.com/apache/flink/pull/13354 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13311: [FLINK-18721] Migrate YarnResourceManager to the new YarnResourceManagerDriver
flinkbot edited a comment on pull request #13311: URL: https://github.com/apache/flink/pull/13311#issuecomment-686196434 ## CI report: * b7660009b48d48261b767855d8973c63741ab493 UNKNOWN * f16d78f22d3d66e763f5629a377936bcb20b80c4 UNKNOWN * 507f270bfbac225cafa7e25d4ab60b1c74b66c74 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6390) * 97cefe9ca646f66c5b0fa09dead666d1fe468029 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13228: [FLINK-19026][network] Improve threading model of CheckpointBarrierUnaligner
flinkbot edited a comment on pull request #13228: URL: https://github.com/apache/flink/pull/13228#issuecomment-679099456 ## CI report: * d141f10ef060063162dc073b1cd66729f5f75a3b UNKNOWN * 407169bc192b5d05bfba2c264849f1a6e7b81415 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6402) * 89b0550c7b223f304b67a6f64dfc4252509eb76e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13040: [FLINK-17073] [checkpointing] checkpointing backpressure if there are too many checkpoints to clean
flinkbot edited a comment on pull request #13040: URL: https://github.com/apache/flink/pull/13040#issuecomment-667140037 ## CI report: * 31abcb25fdf72394dbfa91d0ba9e151968375ad7 UNKNOWN * db99952f8166cb4d6337dcb3815e72ff2b49604b UNKNOWN * 33af84f70ab92a76818f1a4392fecb0c4a8e62f9 UNKNOWN * 12d5698e9185b3b97e5323485ee72617c4e535f9 UNKNOWN * d189814365e219f4641f502be5c2b9f2d2afca1c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6441) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13355: [FLINK-17554] Allow to register release hooks for the classloader
azagrebin commented on a change in pull request #13355: URL: https://github.com/apache/flink/pull/13355#discussion_r486269832 ## File path: flink-connectors/flink-connector-kinesis/src/test/resources/log4j2-test.properties ## @@ -0,0 +1,29 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# Set root logger level to OFF to not flood build logs +# set manually to INFO for debugging purposes +rootLogger.level = OFF Review comment: also in `flink-table/flink-table-api-java-bridge` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13355: [FLINK-17554] Allow to register release hooks for the classloader
azagrebin commented on a change in pull request #13355:
URL: https://github.com/apache/flink/pull/13355#discussion_r485755685
##
File path:
flink-core/src/main/java/org/apache/flink/api/common/functions/RuntimeContext.java
##
@@ -123,6 +123,17 @@
*/
ClassLoader getUserCodeClassLoader();
+ /**
+* Registers a release hook for the user code class loader which is
executed just before the
+* user code class loader is being released.
+* Registration only happens if no hook has been registered under this
name already.
Review comment:
```suggestion
* Registers a custom hook for the user code class loader release.
*
* The release hook is executed just before the user code class
loader is being released.
* Registration only happens if no hook has been registered under this
name already.
```
Maybe not the final version. I think we usually keep the first
title-sentence short. Also for other such doc comments
##
File path:
flink-core/src/main/java/org/apache/flink/api/common/functions/RuntimeContext.java
##
@@ -123,6 +123,17 @@
*/
ClassLoader getUserCodeClassLoader();
+ /**
+* Registers a release hook for the user code class loader which is
executed just before the
+* user code class loader is being released.
+* Registration only happens if no hook has been registered under this
name already.
+*
+* @param releaseHookName name of the release hook.
+* @param releaseHook releaseHook which is executed just before the
user code class loader is being released
Review comment:
```suggestion
* @param releaseHook release hook which is executed just before the
user code class loader is being released
```
##
File path: flink-connectors/flink-connector-cassandra/pom.xml
##
@@ -187,6 +187,13 @@ under the License.
test
test-jar
+
+ org.apache.flink
+ flink-core
+ ${project.version}
+ test-jar
Review comment:
why do we need all these `flink-core/test-jar` changes? are they related
to the PR?
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/operators/chaining/ChainedDriver.java
##
@@ -63,8 +64,8 @@
public void setup(TaskConfig config, String taskName, Collector
outputCollector,
- AbstractInvokable parent, ClassLoader
userCodeClassLoader, ExecutionConfig executionConfig,
- Map> accumulatorMap)
+ AbstractInvokable parent,
UserCodeClassLoader userCodeClassLoader, ExecutionConfig executionConfig,
Review comment:
Why do we need to pass here `UserCodeClassLoader`?
I do not miss something, we only use it `asClassLoader` afterwards.
Then same for `BatchTask#initOutputs`.
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/execution/librarycache/BlobLibraryCacheManagerTest.java
##
@@ -504,6 +506,41 @@ public void closingLibraryCacheManager_closesClassLoader()
throws IOException {
assertTrue(classLoader.isClosed());
}
+ @Test
+ public void releaseUserCodeClassLoader_willRunReleaseHooks() throws
Exception {
+ final BlobLibraryCacheManager libraryCacheManager = new
TestingBlobLibraryCacheManagerBuilder().build();
+
+ final LibraryCacheManager.ClassLoaderLease classLoaderLease =
libraryCacheManager.registerClassLoaderLease(new JobID());
+ final UserCodeClassLoader userCodeClassLoader =
classLoaderLease.getOrResolveClassLoader(Collections.emptyList(),
Collections.emptyList());
+
+ final OneShotLatch releaseHookLatch = new OneShotLatch();
+ userCodeClassLoader.registerReleaseHookIfAbsent("test",
releaseHookLatch::trigger);
+
+ // this should trigger the release of the class loader
+ classLoaderLease.release();
+
+ releaseHookLatch.await();
+ }
+
+ @Test
+ public void releaseUserCodeClassLoader_willRegisterOnce() throws
Exception {
Review comment:
```suggestion
public void releaseUserCodeClassLoader_willRegisterOnce() throws
IOException, InterruptedException {
```
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/operators/chaining/ChainedDriver.java
##
@@ -63,8 +64,8 @@
public void setup(TaskConfig config, String taskName, Collector
outputCollector,
- AbstractInvokable parent, ClassLoader
userCodeClassLoader, ExecutionConfig executionConfig,
- Map> accumulatorMap)
+ AbstractInvokable parent,
[GitHub] [flink] flinkbot edited a comment on pull request #13369: [FLINK-19173][python] Add Pandas Batch Group Aggregation Function Operator
flinkbot edited a comment on pull request #13369: URL: https://github.com/apache/flink/pull/13369#issuecomment-690035137 ## CI report: * c07c5e36851207112dc67aad490005a3d26d39c5 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6424) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13368: [FLINK-19000] Forward initialization timestamp from Dispatcher to ExecGraph
flinkbot edited a comment on pull request #13368: URL: https://github.com/apache/flink/pull/13368#issuecomment-690035026 ## CI report: * 59deb5d9ae2ddf42b40ff4391d567daa8abef8db Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6423) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19192) Set higher limit on the HTTP connection pool
[ https://issues.apache.org/jira/browse/FLINK-19192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-19192: --- Labels: pull-request-available (was: ) > Set higher limit on the HTTP connection pool > > > Key: FLINK-19192 > URL: https://issues.apache.org/jira/browse/FLINK-19192 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Reporter: Igal Shilman >Assignee: Igal Shilman >Priority: Major > Labels: pull-request-available > > The default size of the connection pool is too low, we should set it to a > higher value > and let servers to decide if they will keep the connection alive or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-statefun] igalshilman opened a new pull request #146: [FLINK-19192] Set HTTP connection pool size to 1024
igalshilman opened a new pull request #146: URL: https://github.com/apache/flink-statefun/pull/146 This PR increase the max size of the http connection pool to 1024. The connection pool holds connections to the various endpoints and possibly various endpoint instances. The remote server can decide that they will not keep the connection alive. An idle connection would be evicted after 1 minute of inactivity. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13283: [FLINK-18759][tests] Add readme.md for TPC-DS tools
flinkbot edited a comment on pull request #13283: URL: https://github.com/apache/flink/pull/13283#issuecomment-683593514 ## CI report: * 62f40cd28a36147d84a77b499e63b6224df8b038 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6426) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19171) K8s Resource Manager may lead to resource leak after pod deleted
[
https://issues.apache.org/jira/browse/FLINK-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song closed FLINK-19171.
Resolution: Not A Problem
> K8s Resource Manager may lead to resource leak after pod deleted
>
>
> Key: FLINK-19171
> URL: https://issues.apache.org/jira/browse/FLINK-19171
> Project: Flink
> Issue Type: Bug
>Reporter: Yi Tang
>Priority: Minor
>
> {code:java}
> private void terminatedPodsInMainThread(List pods) {
>getMainThreadExecutor().execute(() -> {
> for (KubernetesPod pod : pods) {
> if (pod.isTerminated()) {
> ...
> }
> }
>});
> }
> {code}
> Looks like that the RM only remove the pod from ledger if the pod
> "isTerminated",
> and the pod has been taken accounted after being created.
> However, it is not complete by checking pod "isTerminated", e.g. a Pending
> pod is deleted manually.
> After that, a new job requires more resource can not trigger the allocation
> of a new pod.
>
> Pls let me know if i misunderstand, thanks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-19151) Flink does not normalize container resource with correct configurations when Yarn FairScheduler is used
[ https://issues.apache.org/jira/browse/FLINK-19151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song closed FLINK-19151. Resolution: Fixed Fixed via * master: 00bf41f33a10b59850f0ee4fe31c0271484d6d4c * release-1.11: bfff6b15ec7dd3a4415f6a5a9d8535ea7960e474 > Flink does not normalize container resource with correct configurations when > Yarn FairScheduler is used > > > Key: FLINK-19151 > URL: https://issues.apache.org/jira/browse/FLINK-19151 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.11.2 >Reporter: Xintong Song >Assignee: jinhai >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.11.3 > > > h3. Problem > It's a Yarn protocol that the requested container resource will be normalized > for allocation. That means, the allocated container may have different > resource (larger than or equal to) compared to what is requested. > Currently, Flink matches the allocated containers to the original requests by > reading the Yarn configurations and calculate how the requested resources > should be normalized. > What has been overlooked is that, Yarn FairScheduler (and its subclass > SLSFairScheduler) has overridden the normalization behavior. To be specific, > * By default, Yarn normalize container resources to integer multiple of > "yarn.scheduler.minimum-allocation-[mb|vcores]" > * FairScheduler normalize container resources to integer multiple of > "yarn.resource-types.[memory-mb|vcores].increment-allocation" (or the > deprecated keys "yarn.scheduler.increment-allocation-[mb|vcores]"), while > making sure the resource is no less than > "yarn.scheduler.minimum-allocation-[mb|vcores]" > h3. Proposal for short term solution > To fix this problem, a quick and easy way is to also read Yarn configuration > and learn which scheduler is used, and perform normalization calculations > accordingly. This should be good enough to cover behaviors of all the > schedulers that Yarn currently provides. The limitation is that, Flink will > not be able to deal with custom Yarn schedulers which override the > normalization behaviors. > h3. Proposal for long term solution > For long term, it would be good to use Yarn > ContainerRequest#allocationRequestId to match the allocated containers with > the original requests, so that Flink no longer needs to understand how Yarn > normalize container resources. > Yarn ContainerRequest#allocationRequestId is introduced in Hadoop 2.9, while > ATM Flink claims to be compatible with Hadoop 2.4+. Therefore, this solution > would not work at the moment. > Another idea is to support various Hadoop versions with different container > matching logics. We can abstract the container matching logics into a > dedicating component, and provide different implementations for it. This will > allow Flink to take advantages of the new versions (e.g., work well with > custom schedulers), while stay compatible with the old versions without those > advantages. > Given that we need the resource based matching anyway for the old Hadoop > versions, and the cost for maintaining two sets of matching logics, I tend to > think this approach as a back-up option to be worked on when we indeed see a > need for it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] xintongsong closed pull request #13347: [FLINK-19151][yarn]Fix the unit value according to different yarn scheduler
xintongsong closed pull request #13347: URL: https://github.com/apache/flink/pull/13347 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-19192) Set higher limit on the HTTP connection pool
Igal Shilman created FLINK-19192: Summary: Set higher limit on the HTTP connection pool Key: FLINK-19192 URL: https://issues.apache.org/jira/browse/FLINK-19192 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman The default size of the connection pool is too low, we should set it to a higher value and let servers to decide if they will keep the connection alive or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-19192) Set higher limit on the HTTP connection pool
[ https://issues.apache.org/jira/browse/FLINK-19192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igal Shilman reassigned FLINK-19192: Assignee: Igal Shilman > Set higher limit on the HTTP connection pool > > > Key: FLINK-19192 > URL: https://issues.apache.org/jira/browse/FLINK-19192 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Reporter: Igal Shilman >Assignee: Igal Shilman >Priority: Major > > The default size of the connection pool is too low, we should set it to a > higher value > and let servers to decide if they will keep the connection alive or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19178) Introduce the memory weights configuration option and interfaces
[ https://issues.apache.org/jira/browse/FLINK-19178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-19178: - Fix Version/s: (was: 1.11.3) > Introduce the memory weights configuration option and interfaces > > > Key: FLINK-19178 > URL: https://issues.apache.org/jira/browse/FLINK-19178 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Configuration >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Major > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-19159) Using Scalafmt to format scala source code
[ https://issues.apache.org/jira/browse/FLINK-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193544#comment-17193544 ] darion yaphet edited comment on FLINK-19159 at 9/10/20, 11:08 AM: -- [#aljoscha] Maybe you are right. I have use Scalafmt and try to adapt the current format. But I found it difficult to keep all of them. Sorry to say, something maybe need more research. thanks a lot :) was (Author: darion): [aljoscha|[https://issues.apache.org/jira/secure/ViewProfile.jspa?name=aljoscha]] Maybe you are right. I have use Scalafmt and try to adapt the current format. But I found it difficult to keep all of them. Sorry to say, something maybe need more research. thanks a lot :) > Using Scalafmt to format scala source code > -- > > Key: FLINK-19159 > URL: https://issues.apache.org/jira/browse/FLINK-19159 > Project: Flink > Issue Type: Improvement >Reporter: darion yaphet >Priority: Minor > > Scalafmt is a code formatter for Scala. It can help developer to avoid code > style conflict -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-19159) Using Scalafmt to format scala source code
[ https://issues.apache.org/jira/browse/FLINK-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193544#comment-17193544 ] darion yaphet edited comment on FLINK-19159 at 9/10/20, 11:08 AM: -- [aljoscha|[https://issues.apache.org/jira/secure/ViewProfile.jspa?name=aljoscha]] Maybe you are right. I have use Scalafmt and try to adapt the current format. But I found it difficult to keep all of them. Sorry to say, something maybe need more research. thanks a lot :) was (Author: darion): [https://issues.apache.org/jira/secure/ViewProfile.jspa?name=aljoscha](aljoscha) Maybe you are right. I have use Scalafmt and try to adapt the current format. But I found it difficult to keep all of them. Sorry to say, something maybe need more research. thanks a lot :) > Using Scalafmt to format scala source code > -- > > Key: FLINK-19159 > URL: https://issues.apache.org/jira/browse/FLINK-19159 > Project: Flink > Issue Type: Improvement >Reporter: darion yaphet >Priority: Minor > > Scalafmt is a code formatter for Scala. It can help developer to avoid code > style conflict -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-19159) Using Scalafmt to format scala source code
[ https://issues.apache.org/jira/browse/FLINK-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193544#comment-17193544 ] darion yaphet edited comment on FLINK-19159 at 9/10/20, 11:08 AM: -- [aljoscha |#aljoscha] Maybe you are right. I have use Scalafmt and try to adapt the current format. But I found it difficult to keep all of them. Sorry to say, something maybe need more research. thanks a lot :) was (Author: darion): [#aljoscha] Maybe you are right. I have use Scalafmt and try to adapt the current format. But I found it difficult to keep all of them. Sorry to say, something maybe need more research. thanks a lot :) > Using Scalafmt to format scala source code > -- > > Key: FLINK-19159 > URL: https://issues.apache.org/jira/browse/FLINK-19159 > Project: Flink > Issue Type: Improvement >Reporter: darion yaphet >Priority: Minor > > Scalafmt is a code formatter for Scala. It can help developer to avoid code > style conflict -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13344: [FLINK-10740] FLIP-27 File Source and various Updates on the Split Reader API
flinkbot edited a comment on pull request #13344: URL: https://github.com/apache/flink/pull/13344#issuecomment-688495541 ## CI report: * 15a81e82a4a42cd255baad4ffd3e7e20343b5389 UNKNOWN * d154ff654bd927988a42f2881e8f690a121ea536 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6341) * 43239addb3380c94c2c3452557b8ecb1f5982e1b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6438) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-19159) Using Scalafmt to format scala source code
[ https://issues.apache.org/jira/browse/FLINK-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193544#comment-17193544 ] darion yaphet edited comment on FLINK-19159 at 9/10/20, 11:06 AM: -- [https://issues.apache.org/jira/secure/ViewProfile.jspa?name=aljoscha](aljoscha) Maybe you are right. I have use Scalafmt and try to adapt the current format. But I found it difficult to keep all of them. Sorry to say, something maybe need more research. thanks a lot :) was (Author: darion): [aljoscha](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=aljoscha) Maybe you are right. I have use Scalafmt and try to adapt the current format. But I found it difficult to keep all of them. Sorry to say, something maybe need more research. thanks a lot :) > Using Scalafmt to format scala source code > -- > > Key: FLINK-19159 > URL: https://issues.apache.org/jira/browse/FLINK-19159 > Project: Flink > Issue Type: Improvement >Reporter: darion yaphet >Priority: Minor > > Scalafmt is a code formatter for Scala. It can help developer to avoid code > style conflict -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13040: [FLINK-17073] [checkpointing] checkpointing backpressure if there are too many checkpoints to clean
flinkbot edited a comment on pull request #13040: URL: https://github.com/apache/flink/pull/13040#issuecomment-667140037 ## CI report: * 31abcb25fdf72394dbfa91d0ba9e151968375ad7 UNKNOWN * db99952f8166cb4d6337dcb3815e72ff2b49604b UNKNOWN * 33af84f70ab92a76818f1a4392fecb0c4a8e62f9 UNKNOWN * 12d5698e9185b3b97e5323485ee72617c4e535f9 UNKNOWN * 6e61a9e7a9a18c96a74ec36ffdfa6d3765eb31fc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6171) * d189814365e219f4641f502be5c2b9f2d2afca1c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19171) K8s Resource Manager may lead to resource leak after pod deleted
[
https://issues.apache.org/jira/browse/FLINK-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193543#comment-17193543
]
Yi Tang commented on FLINK-19171:
-
bq. BTW, if you are interested in contributing to any of the mentioned 2
tickets, feel free to leave a message on the ticket.
I'll take a closer look
> K8s Resource Manager may lead to resource leak after pod deleted
>
>
> Key: FLINK-19171
> URL: https://issues.apache.org/jira/browse/FLINK-19171
> Project: Flink
> Issue Type: Bug
>Reporter: Yi Tang
>Priority: Minor
>
> {code:java}
> private void terminatedPodsInMainThread(List pods) {
>getMainThreadExecutor().execute(() -> {
> for (KubernetesPod pod : pods) {
> if (pod.isTerminated()) {
> ...
> }
> }
>});
> }
> {code}
> Looks like that the RM only remove the pod from ledger if the pod
> "isTerminated",
> and the pod has been taken accounted after being created.
> However, it is not complete by checking pod "isTerminated", e.g. a Pending
> pod is deleted manually.
> After that, a new job requires more resource can not trigger the allocation
> of a new pod.
>
> Pls let me know if i misunderstand, thanks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19159) Using Scalafmt to format scala source code
[ https://issues.apache.org/jira/browse/FLINK-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193544#comment-17193544 ] darion yaphet commented on FLINK-19159: --- [aljoscha](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=aljoscha) Maybe you are right. I have use Scalafmt and try to adapt the current format. But I found it difficult to keep all of them. Sorry to say, something maybe need more research. thanks a lot :) > Using Scalafmt to format scala source code > -- > > Key: FLINK-19159 > URL: https://issues.apache.org/jira/browse/FLINK-19159 > Project: Flink > Issue Type: Improvement >Reporter: darion yaphet >Priority: Minor > > Scalafmt is a code formatter for Scala. It can help developer to avoid code > style conflict -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19171) K8s Resource Manager may lead to resource leak after pod deleted
[
https://issues.apache.org/jira/browse/FLINK-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193542#comment-17193542
]
Yi Tang commented on FLINK-19171:
-
bq. So, if you don't mind, I'd like to close this ticket
It's ok. Maybe a doc still needed though, :)
> K8s Resource Manager may lead to resource leak after pod deleted
>
>
> Key: FLINK-19171
> URL: https://issues.apache.org/jira/browse/FLINK-19171
> Project: Flink
> Issue Type: Bug
>Reporter: Yi Tang
>Priority: Minor
>
> {code:java}
> private void terminatedPodsInMainThread(List pods) {
>getMainThreadExecutor().execute(() -> {
> for (KubernetesPod pod : pods) {
> if (pod.isTerminated()) {
> ...
> }
> }
>});
> }
> {code}
> Looks like that the RM only remove the pod from ledger if the pod
> "isTerminated",
> and the pod has been taken accounted after being created.
> However, it is not complete by checking pod "isTerminated", e.g. a Pending
> pod is deleted manually.
> After that, a new job requires more resource can not trigger the allocation
> of a new pod.
>
> Pls let me know if i misunderstand, thanks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
