[jira] [Commented] (FLINK-16917) "TPC-DS end-to-end test (Blink planner)" gets stuck
[
https://issues.apache.org/jira/browse/FLINK-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073431#comment-17073431
]
Kurt Young commented on FLINK-16917:
And I quickly went through other e2e tests between #20200331.7 and #20200331.8,
it's strange only tpc-h and tpc-ds tests are slowed down.**
> "TPC-DS end-to-end test (Blink planner)" gets stuck
> ---
>
> Key: FLINK-16917
> URL: https://issues.apache.org/jira/browse/FLINK-16917
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.11.0
>
> Attachments: Screenshot 2020-04-02 08.12.01.png, Screenshot
> 2020-04-02 08.24.28.png, image-2020-04-02-09-32-52-979.png
>
>
> The message you see from the CI system is
> {code}
> ##[error]The job running on agent Hosted Agent ran longer than the maximum
> time of 240 minutes. For more information, see
> https://go.microsoft.com/fwlink/?linkid=2077134
> {code}
> Example:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6899&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee
> The end of the log file looks as follows:
> {code}
> 2020-03-31T23:00:40.5416207Z [INFO]Run TPC-DS query 97 success.
> 2020-03-31T23:00:40.5439265Z [INFO]Run TPC-DS query 98 ...
> 2020-03-31T23:00:40.8269500Z Job has been submitted with JobID
> eec4759ae6d585ee9f8d9f84f1793c0e
> 2020-03-31T23:01:33.4757621Z Program execution finished
> 2020-03-31T23:01:33.4758328Z Job with JobID eec4759ae6d585ee9f8d9f84f1793c0e
> has finished.
> 2020-03-31T23:01:33.4758880Z Job Runtime: 51093 ms
> 2020-03-31T23:01:33.4759057Z
> 2020-03-31T23:01:33.4760999Z [INFO]Run TPC-DS query 98 success.
> 2020-03-31T23:01:33.4761612Z [INFO]Run TPC-DS query 99 ...
> 2020-03-31T23:01:33.7297686Z Job has been submitted with JobID
> f47efc4194df2e0ead677fff239f3dfd
> 2020-03-31T23:01:50.0037484Z ##[error]The operation was canceled.
> 2020-03-31T23:01:50.0091655Z ##[section]Finishing: Run e2e tests
> {code}
> Notice the time difference between "Job has been submitted" and "The
> operation was canceled.". There was nothing happening for 20 minutes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-16917) "TPC-DS end-to-end test (Blink planner)" gets stuck
[
https://issues.apache.org/jira/browse/FLINK-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073427#comment-17073427
]
Robert Metzger edited comment on FLINK-16917 at 4/2/20, 6:59 AM:
-
[~AHeise] Kurt is relating to the high build times of "#20200331.8
[FLINK-16245][tests] Adjust BatchFineGrainedRecoveryITCase" [1].
I have analyzed the e2e build time of "#20200331.15 [hotfix] Fix typo in
org.apache.flink.sql.parser.ddl.SqlCreateTable" [2].
Column J shows the durations of the e2e tests:
https://docs.google.com/spreadsheets/d/1VpA3wsOY88ezY8qTUFl8iztPc58mRa-1pIncIBAbUL8/edit#gid=117660203
As you can see in K, the e2e tests are not slower because of that (except for
"TPC-DS end-to-end test (Blink planner)", which doesn't finish :) )
[1] https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6877&view=result
[2]
https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6899&view=results
was (Author: rmetzger):
[~AHeise] Kurt is relating to the high build times of "#20200331.8
[FLINK-16245][tests] Adjust BatchFineGrainedRecoveryITCase" [1].
I have analyzed the e2e build time of "#20200331.15 [hotfix] Fix typo in
org.apache.flink.sql.parser.ddl.SqlCreateTable" [2].
Column J shows the durations of the e2e tests:
https://docs.google.com/spreadsheets/d/1VpA3wsOY88ezY8qTUFl8iztPc58mRa-1pIncIBAbUL8/edit#gid=117660203
As you can see in K, the e2e tests are not slower because of that.
[1] https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6877&view=result
[2]
https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6899&view=results
> "TPC-DS end-to-end test (Blink planner)" gets stuck
> ---
>
> Key: FLINK-16917
> URL: https://issues.apache.org/jira/browse/FLINK-16917
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.11.0
>
> Attachments: Screenshot 2020-04-02 08.12.01.png, Screenshot
> 2020-04-02 08.24.28.png, image-2020-04-02-09-32-52-979.png
>
>
> The message you see from the CI system is
> {code}
> ##[error]The job running on agent Hosted Agent ran longer than the maximum
> time of 240 minutes. For more information, see
> https://go.microsoft.com/fwlink/?linkid=2077134
> {code}
> Example:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6899&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee
> The end of the log file looks as follows:
> {code}
> 2020-03-31T23:00:40.5416207Z [INFO]Run TPC-DS query 97 success.
> 2020-03-31T23:00:40.5439265Z [INFO]Run TPC-DS query 98 ...
> 2020-03-31T23:00:40.8269500Z Job has been submitted with JobID
> eec4759ae6d585ee9f8d9f84f1793c0e
> 2020-03-31T23:01:33.4757621Z Program execution finished
> 2020-03-31T23:01:33.4758328Z Job with JobID eec4759ae6d585ee9f8d9f84f1793c0e
> has finished.
> 2020-03-31T23:01:33.4758880Z Job Runtime: 51093 ms
> 2020-03-31T23:01:33.4759057Z
> 2020-03-31T23:01:33.4760999Z [INFO]Run TPC-DS query 98 success.
> 2020-03-31T23:01:33.4761612Z [INFO]Run TPC-DS query 99 ...
> 2020-03-31T23:01:33.7297686Z Job has been submitted with JobID
> f47efc4194df2e0ead677fff239f3dfd
> 2020-03-31T23:01:50.0037484Z ##[error]The operation was canceled.
> 2020-03-31T23:01:50.0091655Z ##[section]Finishing: Run e2e tests
> {code}
> Notice the time difference between "Job has been submitted" and "The
> operation was canceled.". There was nothing happening for 20 minutes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16917) "TPC-DS end-to-end test (Blink planner)" gets stuck
[
https://issues.apache.org/jira/browse/FLINK-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073432#comment-17073432
]
Robert Metzger commented on FLINK-16917:
I would like to mention that the e2e test run of the pull request adding
FLINK-16245 [1] showed the issue that we are discussing here!
I know the Azure tests are not the most stable one, but they are sometimes
showing real issues :)
[1]https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6813&view=logs&j=e428c73e-5bb8-51ab-b1f4-da736192251d&t=fb85e9b1-9460-5259-9251-5d7b922523ea
> "TPC-DS end-to-end test (Blink planner)" gets stuck
> ---
>
> Key: FLINK-16917
> URL: https://issues.apache.org/jira/browse/FLINK-16917
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.11.0
>
> Attachments: Screenshot 2020-04-02 08.12.01.png, Screenshot
> 2020-04-02 08.24.28.png, image-2020-04-02-09-32-52-979.png
>
>
> The message you see from the CI system is
> {code}
> ##[error]The job running on agent Hosted Agent ran longer than the maximum
> time of 240 minutes. For more information, see
> https://go.microsoft.com/fwlink/?linkid=2077134
> {code}
> Example:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6899&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee
> The end of the log file looks as follows:
> {code}
> 2020-03-31T23:00:40.5416207Z [INFO]Run TPC-DS query 97 success.
> 2020-03-31T23:00:40.5439265Z [INFO]Run TPC-DS query 98 ...
> 2020-03-31T23:00:40.8269500Z Job has been submitted with JobID
> eec4759ae6d585ee9f8d9f84f1793c0e
> 2020-03-31T23:01:33.4757621Z Program execution finished
> 2020-03-31T23:01:33.4758328Z Job with JobID eec4759ae6d585ee9f8d9f84f1793c0e
> has finished.
> 2020-03-31T23:01:33.4758880Z Job Runtime: 51093 ms
> 2020-03-31T23:01:33.4759057Z
> 2020-03-31T23:01:33.4760999Z [INFO]Run TPC-DS query 98 success.
> 2020-03-31T23:01:33.4761612Z [INFO]Run TPC-DS query 99 ...
> 2020-03-31T23:01:33.7297686Z Job has been submitted with JobID
> f47efc4194df2e0ead677fff239f3dfd
> 2020-03-31T23:01:50.0037484Z ##[error]The operation was canceled.
> 2020-03-31T23:01:50.0091655Z ##[section]Finishing: Run e2e tests
> {code}
> Notice the time difference between "Job has been submitted" and "The
> operation was canceled.". There was nothing happening for 20 minutes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on a change in pull request #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
zhijiangW commented on a change in pull request #11515: [FLINK-16744][task]
implement channel state persistence for unaligned checkpoints
URL: https://github.com/apache/flink/pull/11515#discussion_r402094134
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriterImpl.java
##
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.CheckpointOptions;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.state.CheckpointStorageWorkerView;
+import org.apache.flink.runtime.state.CheckpointStreamFactory;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequest.completeInput;
+import static
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequest.completeOutput;
+import static
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequest.write;
+
+/**
+ * {@link ChannelStateWriter} implemented using
+ * {@link CheckpointStreamFactory.CheckpointStateOutputStream
CheckpointStateOutputStreams}. Internally, it has
+ *
+ * one stream per checkpoint; having multiple streams would mean more
files written and more connections opened
+ * (and more latency on restore)
+ * one thread; having multiple threads means more connections, couples
with the implementation and increases complexity
+ *
+ */
+@Internal
+@ThreadSafe
+public class ChannelStateWriterImpl implements ChannelStateWriter {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(ChannelStateWriterImpl.class);
+ private static final int DEFAULT_HANDOVER_CAPACITY = 10;
+ private static final int DEFAULT_MAX_CHECKPOINTS = 5; // currently,
only single in-flight checkpoint is supported
+
+ private class ProcessRequestsLoop implements Runnable {
+ private final ChannelStateWriteRequestProcessor
requestProcessor;
+
+ private ProcessRequestsLoop(int maxCheckpoints,
CheckpointStorageWorkerView streamFactoryFactory, ChannelStateSerializer
serializer) {
+ this.requestProcessor = new
ChannelStateWriteRequestProcessor(maxCheckpoints, streamFactoryFactory,
serializer);
+ }
+
+ @Override
+ public void run() {
+ try {
+ loop();
+ } catch (Exception ex) {
+ thrown = ex;
+ } finally {
+ handover.clear();
+ requestProcessor.cleanup(thrown != null ?
thrown : new RuntimeException("loop terminated, isRunning: " + isRunning));
+ }
+ LOG.debug("loop terminated");
+ }
+
+ private void loop() throws Exception {
+ while (isRunning || !handover.isEmpty()) {
+ try {
+
requestProcessor.processRequest(handover.take());
+ } catch (InterruptedException e) {
+ if (isRunning || !handover.isEmpty()) {
+ LOG.info("interrupted while
waiting for an item (continue waiting)", e);
+ } else {
+
Thread.currentThread().interrupt();
+ return;
+ }
+ }
+ }
+ }
+ }
+
+ private final Thread asyncWriter;
+ private final BlockingQueue handover;
+ private final Map results;
Review
[jira] [Comment Edited] (FLINK-16917) "TPC-DS end-to-end test (Blink planner)" gets stuck
[
https://issues.apache.org/jira/browse/FLINK-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073432#comment-17073432
]
Robert Metzger edited comment on FLINK-16917 at 4/2/20, 7:04 AM:
-
I would like to mention that the e2e test run of the pull request adding
FLINK-16245 [1] showed the issue that we are discussing here!
I know the Azure tests are not the most stable one, but they are sometimes
showing real issues :)
[1]
https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6813&view=logs&j=e428c73e-5bb8-51ab-b1f4-da736192251d&t=fb85e9b1-9460-5259-9251-5d7b922523ea
was (Author: rmetzger):
I would like to mention that the e2e test run of the pull request adding
FLINK-16245 [1] showed the issue that we are discussing here!
I know the Azure tests are not the most stable one, but they are sometimes
showing real issues :)
[1]https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6813&view=logs&j=e428c73e-5bb8-51ab-b1f4-da736192251d&t=fb85e9b1-9460-5259-9251-5d7b922523ea
> "TPC-DS end-to-end test (Blink planner)" gets stuck
> ---
>
> Key: FLINK-16917
> URL: https://issues.apache.org/jira/browse/FLINK-16917
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.11.0
>
> Attachments: Screenshot 2020-04-02 08.12.01.png, Screenshot
> 2020-04-02 08.24.28.png, image-2020-04-02-09-32-52-979.png
>
>
> The message you see from the CI system is
> {code}
> ##[error]The job running on agent Hosted Agent ran longer than the maximum
> time of 240 minutes. For more information, see
> https://go.microsoft.com/fwlink/?linkid=2077134
> {code}
> Example:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6899&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee
> The end of the log file looks as follows:
> {code}
> 2020-03-31T23:00:40.5416207Z [INFO]Run TPC-DS query 97 success.
> 2020-03-31T23:00:40.5439265Z [INFO]Run TPC-DS query 98 ...
> 2020-03-31T23:00:40.8269500Z Job has been submitted with JobID
> eec4759ae6d585ee9f8d9f84f1793c0e
> 2020-03-31T23:01:33.4757621Z Program execution finished
> 2020-03-31T23:01:33.4758328Z Job with JobID eec4759ae6d585ee9f8d9f84f1793c0e
> has finished.
> 2020-03-31T23:01:33.4758880Z Job Runtime: 51093 ms
> 2020-03-31T23:01:33.4759057Z
> 2020-03-31T23:01:33.4760999Z [INFO]Run TPC-DS query 98 success.
> 2020-03-31T23:01:33.4761612Z [INFO]Run TPC-DS query 99 ...
> 2020-03-31T23:01:33.7297686Z Job has been submitted with JobID
> f47efc4194df2e0ead677fff239f3dfd
> 2020-03-31T23:01:50.0037484Z ##[error]The operation was canceled.
> 2020-03-31T23:01:50.0091655Z ##[section]Finishing: Run e2e tests
> {code}
> Notice the time difference between "Job has been submitted" and "The
> operation was canceled.". There was nothing happening for 20 minutes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11276: [FLINK-16029][table-planner-blink] Remove register source and sink in test cases of blink planner
flinkbot edited a comment on issue #11276: [FLINK-16029][table-planner-blink] Remove register source and sink in test cases of blink planner URL: https://github.com/apache/flink/pull/11276#issuecomment-593234780 ## CI report: * f3897666444b1e28f60e4855e5b1e18dd3849dcf Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/157501659) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6934) * 1eee3c75e8c4619e988a2886c725c277b723e71c Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/157700077) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6961) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11598: [FLINK-16914][python] Support ArrayType in vectorized Python UDF
flinkbot edited a comment on issue #11598: [FLINK-16914][python] Support ArrayType in vectorized Python UDF URL: https://github.com/apache/flink/pull/11598#issuecomment-607168470 ## CI report: * d81b0e98dc23cf9370cdfdd20b39dd8cfdab3f84 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/15747) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6929) * 18822435d028d86859d2b2becbb4e63ae2843f1c Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/157700295) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6963) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor
flinkbot edited a comment on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor URL: https://github.com/apache/flink/pull/11571#issuecomment-606377768 ## CI report: * 91507d011419afe848d1a90ae0027cd2511817e5 UNKNOWN * fd838b6268e0233589afed81f4f1fd9a0ec77bf9 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/156948705) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6859) * 235a3dad633fd7a3654f0fa57be79787da769334 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11610: [FLINK-16772][hive] Move derby dependency to test scope
flinkbot edited a comment on issue #11610: [FLINK-16772][hive] Move derby dependency to test scope URL: https://github.com/apache/flink/pull/11610#issuecomment-607601133 ## CI report: * eaffdd64f5013434c6cdbde1dc3d14cdad4f4b80 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/157687240) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6958) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11542: [FLINK-16303][rest] Enable retrieval of custom JobManager log files
flinkbot edited a comment on issue #11542: [FLINK-16303][rest] Enable retrieval of custom JobManager log files URL: https://github.com/apache/flink/pull/11542#issuecomment-604812411 ## CI report: * c45cb2a911fbd9a775fee2725370081f5d15b764 UNKNOWN * 05f2fcf630501899ec9b4f34bef99b98e6e57f72 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/157465348) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6924) * 74a2a88b887fda108d960c238ecab9411c3308a8 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/157700236) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6962) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] JingsongLi commented on issue #11594: [FLINK-16860][orc] Distinguish empty filters from no pushed down in expainSource
JingsongLi commented on issue #11594: [FLINK-16860][orc] Distinguish empty filters from no pushed down in expainSource URL: https://github.com/apache/flink/pull/11594#issuecomment-607663312 > Thanks @JingsongLi for contribution, LGTM > just one minor comment that should we check `(predicates.length == 0) ` in `ParquetTableSource#predicateString()` too ? Thanks @leonardBang for the review, i created https://issues.apache.org/jira/browse/FLINK-16937 to track parquet. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16937) ParquetTableSource should generate correct isFilterPushedDown
[
https://issues.apache.org/jira/browse/FLINK-16937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee updated FLINK-16937:
-
Affects Version/s: (was: 1.10.1)
1.10.0
> ParquetTableSource should generate correct isFilterPushedDown
> -
>
> Key: FLINK-16937
> URL: https://issues.apache.org/jira/browse/FLINK-16937
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.9.2, 1.10.0
>Reporter: Jingsong Lee
>Priority: Major
>
>
> {code:java}
> if (predicate != null) {
>this.isFilterPushedDown = true;
> }
> {code}
> If all filters can not be converted to parquet filter, the predicate will be
> null, this will lead to false isFilterPushdedDown, which is wrong.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16937) ParquetTableSource should generate correct isFilterPushedDown
[
https://issues.apache.org/jira/browse/FLINK-16937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee updated FLINK-16937:
-
Affects Version/s: 1.10.1
1.9.2
> ParquetTableSource should generate correct isFilterPushedDown
> -
>
> Key: FLINK-16937
> URL: https://issues.apache.org/jira/browse/FLINK-16937
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.9.2, 1.10.1
>Reporter: Jingsong Lee
>Priority: Major
>
>
> {code:java}
> if (predicate != null) {
>this.isFilterPushedDown = true;
> }
> {code}
> If all filters can not be converted to parquet filter, the predicate will be
> null, this will lead to false isFilterPushdedDown, which is wrong.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi merged pull request #11594: [FLINK-16860][orc] Distinguish empty filters from no pushed down in expainSource
JingsongLi merged pull request #11594: [FLINK-16860][orc] Distinguish empty filters from no pushed down in expainSource URL: https://github.com/apache/flink/pull/11594 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-16937) ParquetTableSource should generate correct isFilterPushedDown
Jingsong Lee created FLINK-16937:
Summary: ParquetTableSource should generate correct
isFilterPushedDown
Key: FLINK-16937
URL: https://issues.apache.org/jira/browse/FLINK-16937
Project: Flink
Issue Type: Bug
Reporter: Jingsong Lee
{code:java}
if (predicate != null) {
this.isFilterPushedDown = true;
}
{code}
If all filters can not be converted to parquet filter, the predicate will be
null, this will lead to false isFilterPushdedDown, which is wrong.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on a change in pull request #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
zhijiangW commented on a change in pull request #11515: [FLINK-16744][task]
implement channel state persistence for unaligned checkpoints
URL: https://github.com/apache/flink/pull/11515#discussion_r402096143
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriteRequestProcessor.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.runtime.state.CheckpointStorageWorkerView;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Maintains a set of {@link ChannelStateCheckpointWriter writers} per
checkpoint and translates incoming
+ * {@link ChannelStateWriteRequest requests} to their corresponding methods.
+ */
+final class ChannelStateWriteRequestProcessor {
+ private static final Logger LOG =
LoggerFactory.getLogger(ChannelStateWriteRequestProcessor.class);
+
+ private final Map writers; //
limited indirectly by results max size
+ private final CheckpointStorageWorkerView streamFactoryFactory;
+ private final ChannelStateSerializer serializer;
+
+ ChannelStateWriteRequestProcessor(int maxCheckpoints,
CheckpointStorageWorkerView streamFactoryFactory, ChannelStateSerializer
serializer) {
+ this.writers = new HashMap<>(maxCheckpoints);
+ this.streamFactoryFactory = streamFactoryFactory;
+ this.serializer = serializer;
Review comment:
nit: checkNotNull for two arguments
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] JingsongLi merged pull request #11596: [FLINK-16860][orc] Distinguish empty filters from no pushed down in expainSource
JingsongLi merged pull request #11596: [FLINK-16860][orc] Distinguish empty filters from no pushed down in expainSource URL: https://github.com/apache/flink/pull/11596 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] JingsongLi merged pull request #11597: [FLINK-16860][orc] Distinguish empty filters from no pushed down in expainSource
JingsongLi merged pull request #11597: [FLINK-16860][orc] Distinguish empty filters from no pushed down in expainSource URL: https://github.com/apache/flink/pull/11597 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16917) "TPC-DS end-to-end test (Blink planner)" gets stuck
[
https://issues.apache.org/jira/browse/FLINK-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073437#comment-17073437
]
Arvid Heise commented on FLINK-16917:
-
I'm assuming that something in Table API is leaking the classloader and hence
we only see the issue there. [~chesnay] , seems like failing fast would have
been the better option after all.
> "TPC-DS end-to-end test (Blink planner)" gets stuck
> ---
>
> Key: FLINK-16917
> URL: https://issues.apache.org/jira/browse/FLINK-16917
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.11.0
>
> Attachments: Screenshot 2020-04-02 08.12.01.png, Screenshot
> 2020-04-02 08.24.28.png, image-2020-04-02-09-32-52-979.png
>
>
> The message you see from the CI system is
> {code}
> ##[error]The job running on agent Hosted Agent ran longer than the maximum
> time of 240 minutes. For more information, see
> https://go.microsoft.com/fwlink/?linkid=2077134
> {code}
> Example:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6899&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee
> The end of the log file looks as follows:
> {code}
> 2020-03-31T23:00:40.5416207Z [INFO]Run TPC-DS query 97 success.
> 2020-03-31T23:00:40.5439265Z [INFO]Run TPC-DS query 98 ...
> 2020-03-31T23:00:40.8269500Z Job has been submitted with JobID
> eec4759ae6d585ee9f8d9f84f1793c0e
> 2020-03-31T23:01:33.4757621Z Program execution finished
> 2020-03-31T23:01:33.4758328Z Job with JobID eec4759ae6d585ee9f8d9f84f1793c0e
> has finished.
> 2020-03-31T23:01:33.4758880Z Job Runtime: 51093 ms
> 2020-03-31T23:01:33.4759057Z
> 2020-03-31T23:01:33.4760999Z [INFO]Run TPC-DS query 98 success.
> 2020-03-31T23:01:33.4761612Z [INFO]Run TPC-DS query 99 ...
> 2020-03-31T23:01:33.7297686Z Job has been submitted with JobID
> f47efc4194df2e0ead677fff239f3dfd
> 2020-03-31T23:01:50.0037484Z ##[error]The operation was canceled.
> 2020-03-31T23:01:50.0091655Z ##[section]Finishing: Run e2e tests
> {code}
> Notice the time difference between "Job has been submitted" and "The
> operation was canceled.". There was nothing happening for 20 minutes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-16561) Resuming Externalized Checkpoint (rocks, incremental, no parallelism change) end-to-end test fails on Azure
[
https://issues.apache.org/jira/browse/FLINK-16561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biao Liu closed FLINK-16561.
Resolution: Duplicate
> Resuming Externalized Checkpoint (rocks, incremental, no parallelism change)
> end-to-end test fails on Azure
> ---
>
> Key: FLINK-16561
> URL: https://issues.apache.org/jira/browse/FLINK-16561
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Tests
>Affects Versions: 1.11.0
>Reporter: Biao Liu
>Assignee: Yun Tang
>Priority: Critical
> Labels: test-stability
> Fix For: 1.11.0
>
>
> {quote}Caused by: java.io.IOException: Cannot access file system for
> checkpoint/savepoint path 'file://.'.
> at
> org.apache.flink.runtime.state.filesystem.AbstractFsCheckpointStorage.resolveCheckpointPointer(AbstractFsCheckpointStorage.java:233)
> at
> org.apache.flink.runtime.state.filesystem.AbstractFsCheckpointStorage.resolveCheckpoint(AbstractFsCheckpointStorage.java:110)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1332)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.tryRestoreExecutionGraphFromSavepoint(SchedulerBase.java:314)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:247)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.(SchedulerBase.java:223)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.(DefaultScheduler.java:118)
> at
> org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:103)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:281)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:269)
> at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98)
> at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40)
> at
> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.(JobManagerRunnerImpl.java:146)
> ... 10 more
> Caused by: java.io.IOException: Found local file path with authority '.' in
> path 'file://.'. Hint: Did you forget a slash? (correct path would be
> 'file:///.')
> at
> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:441)
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389)
> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)
> at
> org.apache.flink.runtime.state.filesystem.AbstractFsCheckpointStorage.resolveCheckpointPointer(AbstractFsCheckpointStorage.java:230)
> ... 22 more
> {quote}
> The original log is here,
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6073&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=2b7514ee-e706-5046-657b-3430666e7bd9
> There are some similar tickets about this case, but the stack here looks
> different.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] hequn8128 commented on a change in pull request #11598: [FLINK-16914][python] Support ArrayType in vectorized Python UDF
hequn8128 commented on a change in pull request #11598: [FLINK-16914][python]
Support ArrayType in vectorized Python UDF
URL: https://github.com/apache/flink/pull/11598#discussion_r402095415
##
File path:
flink-python/src/main/java/org/apache/flink/table/runtime/arrow/ArrowUtils.java
##
@@ -223,39 +239,43 @@ private static Field toArrowField(RowType.RowField
rowField) {
private static ArrowFieldWriter
createBaseRowArrowFieldWriter(FieldVector vector, LogicalType fieldType) {
Review comment:
The return type now may not be a BaseRow. Maybe also add a test in
`BaseRowArrowReaderWriterTest` to cover this case？
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (FLINK-16860) TableException: Failed to push filter into table source! when upgrading flink to 1.9.2
[
https://issues.apache.org/jira/browse/FLINK-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee resolved FLINK-16860.
--
Resolution: Fixed
master: c8a23c74e618b752bbdc58dca62d997ddd303d40
release-1.10: 9d21a4a5539e6fe0b253def255d6641335da18df
release-1.9: 78e2c0de6ef21e497e82de4a180759a960d11eab
> TableException: Failed to push filter into table source! when upgrading flink
> to 1.9.2
> --
>
> Key: FLINK-16860
> URL: https://issues.apache.org/jira/browse/FLINK-16860
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ORC, Table SQL / API
>Affects Versions: 1.9.2, 1.10.0
> Environment: flink 1.8.2
> flink 1.9.2
>Reporter: Nikola
>Assignee: Jingsong Lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.3, 1.10.1
>
> Attachments: flink-1.8.2.txt, flink-1.9.2.txt
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> We have a batch job which we currently have on a flink cluster running 1.8.2
> The job runs fine. We wanted to upgrade to flink 1.10, but that yielded
> errors, so we started downgrading until we found that the issue is in flink
> 1.9.2
> The job on 1.9.2 fails with:
> {code:java}
> Caused by: org.apache.flink.table.api.TableException: Failed to push filter
> into table source! table source with pushdown capability must override and
> change explainSource() API to explain the pushdown applied!{code}
> Which is not happening on flink 1.8.2. You can check the logs for the exactly
> same job, just running on different cluster versions: [^flink-1.8.2.txt]
> [^flink-1.9.2.txt]
>
> I tried to narrow it down and it seems that this exception has been added in
> FLINK-12399 and there was a small discussion regarding the exception:
> [https://github.com/apache/flink/pull/8468#discussion_r329876088]
> Our code looks something like this:
>
> {code:java}
> String tempTableName = "tempTable";
> String sql = SqlBuilder.buildSql(tempTableName);
> BatchTableEnvironment tableEnv = BatchTableEnvironment.create(env);
> OrcTableSource orcTableSource = OrcTableSource.builder()
> .path(hdfsFolder, true)
> .forOrcSchema(ORC.getSchema())
> .withConfiguration(config)
> .build();
> tableEnv.registerTableSource(tempTableName, orcTableSource);
> Table tempTable = tableEnv.sqlQuery(sql);
> return tableEnv.toDataSet(tempTable, Row.class);
> {code}
> Where the sql build is nothing more than
> {code:java}
> SELECT * FROM table WHERE id IN (1,2,3) AND mid IN(4,5,6){code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on a change in pull request #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
zhijiangW commented on a change in pull request #11515: [FLINK-16744][task]
implement channel state persistence for unaligned checkpoints
URL: https://github.com/apache/flink/pull/11515#discussion_r402097005
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriteRequestProcessor.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import org.apache.flink.runtime.state.CheckpointStorageWorkerView;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Maintains a set of {@link ChannelStateCheckpointWriter writers} per
checkpoint and translates incoming
+ * {@link ChannelStateWriteRequest requests} to their corresponding methods.
+ */
+final class ChannelStateWriteRequestProcessor {
+ private static final Logger LOG =
LoggerFactory.getLogger(ChannelStateWriteRequestProcessor.class);
+
+ private final Map writers; //
limited indirectly by results max size
+ private final CheckpointStorageWorkerView streamFactoryFactory;
+ private final ChannelStateSerializer serializer;
+
+ ChannelStateWriteRequestProcessor(int maxCheckpoints,
CheckpointStorageWorkerView streamFactoryFactory, ChannelStateSerializer
serializer) {
+ this.writers = new HashMap<>(maxCheckpoints);
+ this.streamFactoryFactory = streamFactoryFactory;
Review comment:
ditto: weird naming for me
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-16561) Resuming Externalized Checkpoint (rocks, incremental, no parallelism change) end-to-end test fails on Azure
[
https://issues.apache.org/jira/browse/FLINK-16561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073441#comment-17073441
]
Biao Liu commented on FLINK-16561:
--
[~rmetzger], I have closed this ticket, let's discuss under FLINK-16770.
> Resuming Externalized Checkpoint (rocks, incremental, no parallelism change)
> end-to-end test fails on Azure
> ---
>
> Key: FLINK-16561
> URL: https://issues.apache.org/jira/browse/FLINK-16561
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Tests
>Affects Versions: 1.11.0
>Reporter: Biao Liu
>Assignee: Yun Tang
>Priority: Critical
> Labels: test-stability
> Fix For: 1.11.0
>
>
> {quote}Caused by: java.io.IOException: Cannot access file system for
> checkpoint/savepoint path 'file://.'.
> at
> org.apache.flink.runtime.state.filesystem.AbstractFsCheckpointStorage.resolveCheckpointPointer(AbstractFsCheckpointStorage.java:233)
> at
> org.apache.flink.runtime.state.filesystem.AbstractFsCheckpointStorage.resolveCheckpoint(AbstractFsCheckpointStorage.java:110)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1332)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.tryRestoreExecutionGraphFromSavepoint(SchedulerBase.java:314)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:247)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.(SchedulerBase.java:223)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.(DefaultScheduler.java:118)
> at
> org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:103)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:281)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:269)
> at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98)
> at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40)
> at
> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.(JobManagerRunnerImpl.java:146)
> ... 10 more
> Caused by: java.io.IOException: Found local file path with authority '.' in
> path 'file://.'. Hint: Did you forget a slash? (correct path would be
> 'file:///.')
> at
> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:441)
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389)
> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)
> at
> org.apache.flink.runtime.state.filesystem.AbstractFsCheckpointStorage.resolveCheckpointPointer(AbstractFsCheckpointStorage.java:230)
> ... 22 more
> {quote}
> The original log is here,
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6073&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=2b7514ee-e706-5046-657b-3430666e7bd9
> There are some similar tickets about this case, but the stack here looks
> different.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16924) TableEnvironment#sqlUpdate throw NPE when called in async thread
[ https://issues.apache.org/jira/browse/FLINK-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073443#comment-17073443 ] Terry Wang commented on FLINK-16924: hi [~twalthr] My use case is to explain the execute plan in async thread to avoid potential compile too much time or other unknown struck in main thread. > TableEnvironment#sqlUpdate throw NPE when called in async thread > > > Key: FLINK-16924 > URL: https://issues.apache.org/jira/browse/FLINK-16924 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.11.0 >Reporter: Terry Wang >Priority: Major > Attachments: 7871C046-4D67-49C8-AC8A-A4030A7981CC.png > > > Now in the latest code of flink master branch, we encountered unexpected npe > exception as the picture like attachments. > It seems that I can reproduce this problem by creating tableEnv and doing > some operations in main thread and then calling sqlQuery or sqlUpdate in > another async thread. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] gaoyunhaii commented on a change in pull request #11474: FLINK-10114: Add ORC BulkWriter support for StreamingFileSink
gaoyunhaii commented on a change in pull request #11474: FLINK-10114: Add ORC
BulkWriter support for StreamingFileSink
URL: https://github.com/apache/flink/pull/11474#discussion_r40209
##
File path:
flink-formats/flink-orc-compress/src/main/java/org/apache/flink/formats/orc/vectorizer/Vectorizer.java
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.orc.vectorizer;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+import org.apache.orc.storage.ql.exec.vector.VectorizedRowBatch;
+
+import java.io.IOException;
+import java.io.Serializable;
+
+/**
+ * Implementors of this interface provide the logic to transform their data to
{@link VectorizedRowBatch}.
+ *
+ * @param The type of the element
+ */
+@PublicEvolving
+public interface Vectorizer extends Serializable {
+
+ /**
+* Creates a VectorizedRowBatch containing an array of ColumnVectors
+* from the provided element.
+*
+* @param element The input element
+* @return The VectorizedRowBatch containing the ColumnVectors of the
input element
+* @throws IOException if there is an error while transforming the
input.
+*/
+ VectorizedRowBatch vectorize(T element) throws IOException;
Review comment:
As the [ORC's
document](https://orc.apache.org/docs/core-java.html#writing-orc-files) shows,
`VectorizedRowBatch` should be the buffer to maintain the data to write to the
file, therefore, it seems that the framework should mange the lifecycle of the
row batch, and interface could be defined to let users write each record into
the batch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-16913) ReadableConfigToConfigurationAdapter#getEnum throws UnsupportedOperationException
[
https://issues.apache.org/jira/browse/FLINK-16913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Wysakowicz updated FLINK-16913:
-
Priority: Blocker (was: Critical)
> ReadableConfigToConfigurationAdapter#getEnum throws
> UnsupportedOperationException
> -
>
> Key: FLINK-16913
> URL: https://issues.apache.org/jira/browse/FLINK-16913
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.10.0
>Reporter: Canbin Zheng
>Priority: Blocker
> Fix For: 1.10.1, 1.11.0
>
> Attachments: image-2020-04-01-16-46-13-122.png
>
>
> Steps to reproduce the issue:
> # Set flink-conf.yaml
> ** state.backend: rocksdb
> ** state.checkpoints.dir: hdfs:///flink-checkpoints
> ** state.savepoints.dir: hdfs:///flink-checkpoints
> # Start a Kubernetes session cluster
> # Submit a job to the session cluster, unfortunately a
> UnsupportedOperationException occurs.
> {code:java}
> The program finished with the following
> exception:org.apache.flink.client.program.ProgramInvocationException: The
> main method caused an error: The adapter does not support this method
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:143)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:659)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:890)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:963)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:963)
> Caused by: java.lang.UnsupportedOperationException: The adapter does not
> support this method
> at
> org.apache.flink.configuration.ReadableConfigToConfigurationAdapter.getEnum(ReadableConfigToConfigurationAdapter.java:258)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.(RocksDBStateBackend.java:336)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.configure(RocksDBStateBackend.java:394)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory.createFromConfig(RocksDBStateBackendFactory.java:47)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory.createFromConfig(RocksDBStateBackendFactory.java:32)
> at
> org.apache.flink.runtime.state.StateBackendLoader.loadStateBackendFromConfig(StateBackendLoader.java:154)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.loadStateBackend(StreamExecutionEnvironment.java:792)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.configure(StreamExecutionEnvironment.java:761)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.(StreamExecutionEnvironment.java:217)
> at
> org.apache.flink.client.program.StreamContextEnvironment.(StreamContextEnvironment.java:53)
> at
> org.apache.flink.client.program.StreamContextEnvironment.lambda$setAsContext$2(StreamContextEnvironment.java:103)
> at java.util.Optional.map(Optional.java:215)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getExecutionEnvironment(StreamExecutionEnvironment.java:1882)
> at
> org.apache.flink.streaming.examples.socket.SocketWindowWordCount.main(SocketWindowWordCount.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
> ... 8 more
> {code}
> I am wondering why we introduce {{ReadableConfigToConfigurationAdapter}} to
> wrap the {{Configuration}} but leave many of the getter methods in it to
> throw UnsupportedOperationException that causes potential problems.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-16913) ReadableConfigToConfigurationAdapter#getEnum throws UnsupportedOperationException
[
https://issues.apache.org/jira/browse/FLINK-16913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Wysakowicz reassigned FLINK-16913:
Assignee: Dawid Wysakowicz
> ReadableConfigToConfigurationAdapter#getEnum throws
> UnsupportedOperationException
> -
>
> Key: FLINK-16913
> URL: https://issues.apache.org/jira/browse/FLINK-16913
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.10.0
>Reporter: Canbin Zheng
>Assignee: Dawid Wysakowicz
>Priority: Blocker
> Fix For: 1.10.1, 1.11.0
>
> Attachments: image-2020-04-01-16-46-13-122.png
>
>
> Steps to reproduce the issue:
> # Set flink-conf.yaml
> ** state.backend: rocksdb
> ** state.checkpoints.dir: hdfs:///flink-checkpoints
> ** state.savepoints.dir: hdfs:///flink-checkpoints
> # Start a Kubernetes session cluster
> # Submit a job to the session cluster, unfortunately a
> UnsupportedOperationException occurs.
> {code:java}
> The program finished with the following
> exception:org.apache.flink.client.program.ProgramInvocationException: The
> main method caused an error: The adapter does not support this method
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:143)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:659)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:890)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:963)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:963)
> Caused by: java.lang.UnsupportedOperationException: The adapter does not
> support this method
> at
> org.apache.flink.configuration.ReadableConfigToConfigurationAdapter.getEnum(ReadableConfigToConfigurationAdapter.java:258)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.(RocksDBStateBackend.java:336)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.configure(RocksDBStateBackend.java:394)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory.createFromConfig(RocksDBStateBackendFactory.java:47)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory.createFromConfig(RocksDBStateBackendFactory.java:32)
> at
> org.apache.flink.runtime.state.StateBackendLoader.loadStateBackendFromConfig(StateBackendLoader.java:154)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.loadStateBackend(StreamExecutionEnvironment.java:792)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.configure(StreamExecutionEnvironment.java:761)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.(StreamExecutionEnvironment.java:217)
> at
> org.apache.flink.client.program.StreamContextEnvironment.(StreamContextEnvironment.java:53)
> at
> org.apache.flink.client.program.StreamContextEnvironment.lambda$setAsContext$2(StreamContextEnvironment.java:103)
> at java.util.Optional.map(Optional.java:215)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getExecutionEnvironment(StreamExecutionEnvironment.java:1882)
> at
> org.apache.flink.streaming.examples.socket.SocketWindowWordCount.main(SocketWindowWordCount.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
> ... 8 more
> {code}
> I am wondering why we introduce {{ReadableConfigToConfigurationAdapter}} to
> wrap the {{Configuration}} but leave many of the getter methods in it to
> throw UnsupportedOperationException that causes potential problems.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] gaoyunhaii commented on a change in pull request #11474: FLINK-10114: Add ORC BulkWriter support for StreamingFileSink
gaoyunhaii commented on a change in pull request #11474: FLINK-10114: Add ORC
BulkWriter support for StreamingFileSink
URL: https://github.com/apache/flink/pull/11474#discussion_r402098387
##
File path:
flink-formats/flink-orc-compress/src/main/java/org/apache/flink/formats/orc/writers/OrcBulkWriterFactory.java
##
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.orc.writers;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.formats.orc.vectorizer.Vectorizer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.orc.OrcFile;
+import org.apache.orc.TypeDescription;
+import org.apache.orc.impl.WriterImpl;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A factory that creates an ORC {@link BulkWriter}. The factory takes a user
+ * supplied{@link Vectorizer} implementation to convert the element into an
+ * {@link org.apache.orc.storage.ql.exec.vector.VectorizedRowBatch}.
+ *
+ * @param The type of element to write.
+ */
+@PublicEvolving
+public class OrcBulkWriterFactory implements BulkWriter.Factory {
+
+ /*
+ A dummy Hadoop Path to work around the current implementation of ORC
WriterImpl which
+ works on the basis of a Hadoop FileSystem and Hadoop Path but since we
use a customised
+ ORC PhysicalWriter implementation that uses Flink's own
FSDataOutputStream as the
+ underlying/internal stream instead of Hadoop's FSDataOutputStream, we
don't have to worry
+ about this usage.
+*/
+ private static final Path FIXED_PATH = new Path(".");
+
+ private final Vectorizer vectorizer;
+ private final Properties writerProperties;
+ private final Map confMap;
+ private final TypeDescription schema;
+ private OrcFile.WriterOptions writerOptions;
Review comment:
It seems that we could create the `writerOptions` in the constructor and
make this field also `final`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-16917) "TPC-DS end-to-end test (Blink planner)" gets stuck
[
https://issues.apache.org/jira/browse/FLINK-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073447#comment-17073447
]
Robert Metzger commented on FLINK-16917:
Who's going to revert?
> "TPC-DS end-to-end test (Blink planner)" gets stuck
> ---
>
> Key: FLINK-16917
> URL: https://issues.apache.org/jira/browse/FLINK-16917
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.11.0
>
> Attachments: Screenshot 2020-04-02 08.12.01.png, Screenshot
> 2020-04-02 08.24.28.png, image-2020-04-02-09-32-52-979.png
>
>
> The message you see from the CI system is
> {code}
> ##[error]The job running on agent Hosted Agent ran longer than the maximum
> time of 240 minutes. For more information, see
> https://go.microsoft.com/fwlink/?linkid=2077134
> {code}
> Example:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6899&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee
> The end of the log file looks as follows:
> {code}
> 2020-03-31T23:00:40.5416207Z [INFO]Run TPC-DS query 97 success.
> 2020-03-31T23:00:40.5439265Z [INFO]Run TPC-DS query 98 ...
> 2020-03-31T23:00:40.8269500Z Job has been submitted with JobID
> eec4759ae6d585ee9f8d9f84f1793c0e
> 2020-03-31T23:01:33.4757621Z Program execution finished
> 2020-03-31T23:01:33.4758328Z Job with JobID eec4759ae6d585ee9f8d9f84f1793c0e
> has finished.
> 2020-03-31T23:01:33.4758880Z Job Runtime: 51093 ms
> 2020-03-31T23:01:33.4759057Z
> 2020-03-31T23:01:33.4760999Z [INFO]Run TPC-DS query 98 success.
> 2020-03-31T23:01:33.4761612Z [INFO]Run TPC-DS query 99 ...
> 2020-03-31T23:01:33.7297686Z Job has been submitted with JobID
> f47efc4194df2e0ead677fff239f3dfd
> 2020-03-31T23:01:50.0037484Z ##[error]The operation was canceled.
> 2020-03-31T23:01:50.0091655Z ##[section]Finishing: Run e2e tests
> {code}
> Notice the time difference between "Job has been submitted" and "The
> operation was canceled.". There was nothing happening for 20 minutes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] gaoyunhaii commented on a change in pull request #11474: FLINK-10114: Add ORC BulkWriter support for StreamingFileSink
gaoyunhaii commented on a change in pull request #11474: FLINK-10114: Add ORC
BulkWriter support for StreamingFileSink
URL: https://github.com/apache/flink/pull/11474#discussion_r402101810
##
File path:
flink-formats/flink-orc-compress/src/main/java/org/apache/flink/formats/orc/vectorizer/Vectorizer.java
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.orc.vectorizer;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+import org.apache.orc.storage.ql.exec.vector.VectorizedRowBatch;
+
+import java.io.IOException;
+import java.io.Serializable;
+
+/**
+ * Implementors of this interface provide the logic to transform their data to
{@link VectorizedRowBatch}.
+ *
+ * @param The type of the element
+ */
+@PublicEvolving
+public interface Vectorizer extends Serializable {
+
+ /**
+* Creates a VectorizedRowBatch containing an array of ColumnVectors
+* from the provided element.
+*
+* @param element The input element
+* @return The VectorizedRowBatch containing the ColumnVectors of the
input element
+* @throws IOException if there is an error while transforming the
input.
+*/
+ VectorizedRowBatch vectorize(T element) throws IOException;
Review comment:
Ignore me, I did not refresh the page and do not see the above comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
zhijiangW commented on a change in pull request #11515: [FLINK-16744][task]
implement channel state persistence for unaligned checkpoints
URL: https://github.com/apache/flink/pull/11515#discussion_r402102112
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateCheckpointWriter.java
##
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter.ChannelStateWriteResult;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.state.AbstractChannelStateHandle;
+import org.apache.flink.runtime.state.CheckpointStreamFactory;
+import
org.apache.flink.runtime.state.CheckpointStreamFactory.CheckpointStateOutputStream;
+import org.apache.flink.runtime.state.InputChannelStateHandle;
+import org.apache.flink.runtime.state.ResultSubpartitionStateHandle;
+import org.apache.flink.runtime.state.StreamStateHandle;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.RunnableWithException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BiFunction;
+
+import static org.apache.flink.runtime.state.CheckpointedStateScope.EXCLUSIVE;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Writes channel state for a specific checkpoint-subtask-attempt triple.
+ */
+@NotThreadSafe
+class ChannelStateCheckpointWriter {
+ private static final Logger LOG =
LoggerFactory.getLogger(ChannelStateCheckpointWriter.class);
+
+ private final DataOutputStream dataStream;
+ private final CheckpointStateOutputStream checkpointStream;
+ private final ChannelStateWriteResult result;
+ private final Map> inputChannelOffsets =
new HashMap<>();
+ private final Map>
resultSubpartitionOffsets = new HashMap<>();
+ private final ChannelStateSerializer serializer;
+ private final long checkpointId;
+ private boolean inputCompleted = false;
+ private boolean outputCompleted = false;
+ private final Runnable onComplete;
+
+ ChannelStateCheckpointWriter(
Review comment:
I suggest removing this constructor to avoid introducing multiple
constructors to maintain, except for easing tests purpose. But here is not that
case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11610: [FLINK-16772][hive] Move derby dependency to test scope
flinkbot edited a comment on issue #11610: [FLINK-16772][hive] Move derby dependency to test scope URL: https://github.com/apache/flink/pull/11610#issuecomment-607601133 ## CI report: * eaffdd64f5013434c6cdbde1dc3d14cdad4f4b80 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/157687240) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6958) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor
flinkbot edited a comment on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor URL: https://github.com/apache/flink/pull/11571#issuecomment-606377768 ## CI report: * 91507d011419afe848d1a90ae0027cd2511817e5 UNKNOWN * 235a3dad633fd7a3654f0fa57be79787da769334 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/157703537) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6964) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11574: [FLINK-16859][table-runtime] Introduce file system table factory
flinkbot edited a comment on issue #11574: [FLINK-16859][table-runtime] Introduce file system table factory URL: https://github.com/apache/flink/pull/11574#issuecomment-606453374 ## CI report: * 5b17b69ae7a5aab2fff50545abff32f1bc0ce8ff Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/157698652) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6960) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
zhijiangW commented on a change in pull request #11515: [FLINK-16744][task]
implement channel state persistence for unaligned checkpoints
URL: https://github.com/apache/flink/pull/11515#discussion_r402103925
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateCheckpointWriter.java
##
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter.ChannelStateWriteResult;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.state.AbstractChannelStateHandle;
+import org.apache.flink.runtime.state.CheckpointStreamFactory;
+import
org.apache.flink.runtime.state.CheckpointStreamFactory.CheckpointStateOutputStream;
+import org.apache.flink.runtime.state.InputChannelStateHandle;
+import org.apache.flink.runtime.state.ResultSubpartitionStateHandle;
+import org.apache.flink.runtime.state.StreamStateHandle;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.RunnableWithException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BiFunction;
+
+import static org.apache.flink.runtime.state.CheckpointedStateScope.EXCLUSIVE;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Writes channel state for a specific checkpoint-subtask-attempt triple.
+ */
+@NotThreadSafe
+class ChannelStateCheckpointWriter {
+ private static final Logger LOG =
LoggerFactory.getLogger(ChannelStateCheckpointWriter.class);
+
+ private final DataOutputStream dataStream;
+ private final CheckpointStateOutputStream checkpointStream;
+ private final ChannelStateWriteResult result;
+ private final Map> inputChannelOffsets =
new HashMap<>();
+ private final Map>
resultSubpartitionOffsets = new HashMap<>();
+ private final ChannelStateSerializer serializer;
+ private final long checkpointId;
+ private boolean inputCompleted = false;
+ private boolean outputCompleted = false;
+ private final Runnable onComplete;
+
+ ChannelStateCheckpointWriter(
+ CheckpointStartRequest startCheckpointItem,
+ CheckpointStreamFactory streamFactory,
+ ChannelStateSerializer serializer,
+ Runnable onComplete) throws Exception {
+ this(startCheckpointItem.checkpointId,
startCheckpointItem.targetResult,
streamFactory.createCheckpointStateOutputStream(EXCLUSIVE), serializer,
onComplete);
+ }
+
+ ChannelStateCheckpointWriter(
+ long checkpointId,
+ ChannelStateWriteResult result,
+ CheckpointStateOutputStream stream,
+ ChannelStateSerializer serializer,
+ Runnable onComplete) throws Exception {
+ this(checkpointId, result, serializer, onComplete, stream, new
DataOutputStream(stream));
+ }
+
+ ChannelStateCheckpointWriter(
+ long checkpointId,
+ ChannelStateWriteResult result,
+ ChannelStateSerializer serializer,
+ Runnable onComplete,
+ CheckpointStateOutputStream checkpointStateOutputStream,
+ DataOutputStream dataStream) throws Exception {
+ this.checkpointId = checkpointId;
+ this.result = checkNotNull(result);
+ this.checkpointStream =
checkNotNull(checkpointStateOutputStream);
Review comment:
nit: better to keep the sequence as above arguments
This is an automated message from the Apache Git Service.
To respond to the message
[jira] [Commented] (FLINK-16924) TableEnvironment#sqlUpdate throw NPE when called in async thread
[
https://issues.apache.org/jira/browse/FLINK-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073449#comment-17073449
]
Jark Wu commented on FLINK-16924:
-
I think it's hard to throw a proper exception for multithreads. But we can add
a description on {{TableEnvironment}} that it is not thread-safe.
Hi [~Terry1897], is it possible to hold an independent TableEnv instance for
each thread?
> TableEnvironment#sqlUpdate throw NPE when called in async thread
>
>
> Key: FLINK-16924
> URL: https://issues.apache.org/jira/browse/FLINK-16924
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: Terry Wang
>Priority: Major
> Attachments: 7871C046-4D67-49C8-AC8A-A4030A7981CC.png
>
>
> Now in the latest code of flink master branch, we encountered unexpected npe
> exception as the picture like attachments.
> It seems that I can reproduce this problem by creating tableEnv and doing
> some operations in main thread and then calling sqlQuery or sqlUpdate in
> another async thread.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16915) Cannot find compatible factory for specified execution.target (=local)
[
https://issues.apache.org/jira/browse/FLINK-16915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073451#comment-17073451
]
Jaryzhen commented on FLINK-16915:
--
Yeah, it's work, thx
> Cannot find compatible factory for specified execution.target (=local)
> --
>
> Key: FLINK-16915
> URL: https://issues.apache.org/jira/browse/FLINK-16915
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Affects Versions: 1.11.0
>Reporter: Jaryzhen
>Priority: Minor
>
> It occurred when I run from newly version.
> {code:java}
> Exception in thread "main" java.lang.NullPointerException: Cannot find
> compatible factory for specified execution.target (=local)
> at
> org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:104)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1750)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1658)
> at
> org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:74)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1644)
> at
> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:684)
> at
> org.apache.flink.streaming.scala.examples.wordcount.WordCount$.main(WordCount.scala:89)
> at
> org.apache.flink.streaming.scala.examples.wordcount.WordCount.main(WordCount.scala)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on a change in pull request #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
zhijiangW commented on a change in pull request #11515: [FLINK-16744][task]
implement channel state persistence for unaligned checkpoints
URL: https://github.com/apache/flink/pull/11515#discussion_r402107896
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriteRequest.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter.ChannelStateWriteResult;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.state.CheckpointStorageLocationReference;
+import org.apache.flink.util.function.ThrowingConsumer;
+
+abstract class ChannelStateWriteRequest {
+ final long checkpointId;
+
+ ChannelStateWriteRequest(long checkpointId) {
+ this.checkpointId = checkpointId;
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + ", checkpointId=" +
checkpointId;
+ }
+
+ static CheckpointInProgressRequest completeInput(long checkpointId) {
+ return new CheckpointInProgressRequest(checkpointId,
ChannelStateCheckpointWriter::completeInput);
+ }
+
+ static CheckpointInProgressRequest completeOutput(long checkpointId) {
+ return new CheckpointInProgressRequest(checkpointId,
ChannelStateCheckpointWriter::completeOutput);
+ }
+
+ static ChannelStateWriteRequest write(long checkpointId,
InputChannelInfo info, Buffer... flinkBuffers) {
Review comment:
nit: it is weird for me the emphasis `flinkBuffers`, `buffers` should be ok.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
zhijiangW commented on a change in pull request #11515: [FLINK-16744][task]
implement channel state persistence for unaligned checkpoints
URL: https://github.com/apache/flink/pull/11515#discussion_r402107896
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriteRequest.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter.ChannelStateWriteResult;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.state.CheckpointStorageLocationReference;
+import org.apache.flink.util.function.ThrowingConsumer;
+
+abstract class ChannelStateWriteRequest {
+ final long checkpointId;
+
+ ChannelStateWriteRequest(long checkpointId) {
+ this.checkpointId = checkpointId;
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + ", checkpointId=" +
checkpointId;
+ }
+
+ static CheckpointInProgressRequest completeInput(long checkpointId) {
+ return new CheckpointInProgressRequest(checkpointId,
ChannelStateCheckpointWriter::completeInput);
+ }
+
+ static CheckpointInProgressRequest completeOutput(long checkpointId) {
+ return new CheckpointInProgressRequest(checkpointId,
ChannelStateCheckpointWriter::completeOutput);
+ }
+
+ static ChannelStateWriteRequest write(long checkpointId,
InputChannelInfo info, Buffer... flinkBuffers) {
Review comment:
nit: it is weird for me to emphasis `flinkBuffers`, `buffers` should be ok.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
zhijiangW commented on a change in pull request #11515: [FLINK-16744][task]
implement channel state persistence for unaligned checkpoints
URL: https://github.com/apache/flink/pull/11515#discussion_r402108225
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriteRequest.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter.ChannelStateWriteResult;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.state.CheckpointStorageLocationReference;
+import org.apache.flink.util.function.ThrowingConsumer;
+
+abstract class ChannelStateWriteRequest {
+ final long checkpointId;
+
+ ChannelStateWriteRequest(long checkpointId) {
+ this.checkpointId = checkpointId;
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + ", checkpointId=" +
checkpointId;
+ }
+
+ static CheckpointInProgressRequest completeInput(long checkpointId) {
+ return new CheckpointInProgressRequest(checkpointId,
ChannelStateCheckpointWriter::completeInput);
+ }
+
+ static CheckpointInProgressRequest completeOutput(long checkpointId) {
+ return new CheckpointInProgressRequest(checkpointId,
ChannelStateCheckpointWriter::completeOutput);
+ }
+
+ static ChannelStateWriteRequest write(long checkpointId,
InputChannelInfo info, Buffer... flinkBuffers) {
+ return new CheckpointInProgressRequest(checkpointId, writer ->
writer.writeInput(info, flinkBuffers));
+ }
+
+ static ChannelStateWriteRequest write(long checkpointId,
ResultSubpartitionInfo info, Buffer... flinkBuffers) {
+ return new CheckpointInProgressRequest(checkpointId, writer ->
writer.writeOutput(info, flinkBuffers));
+ }
+}
+
+final class CheckpointStartRequest extends ChannelStateWriteRequest {
+ final ChannelStateWriteResult targetResult;
+ final CheckpointStorageLocationReference locationReference;
+
+ CheckpointStartRequest(long checkpointId, ChannelStateWriteResult
targetResult, CheckpointStorageLocationReference locationReference) {
+ super(checkpointId);
+ this.targetResult = targetResult;
+ this.locationReference = locationReference;
+ }
+}
+
+final class CheckpointInProgressRequest extends ChannelStateWriteRequest {
+ final ThrowingConsumer action;
+
+ CheckpointInProgressRequest(long checkpointId,
ThrowingConsumer action) {
+ super(checkpointId);
+ this.action = action;
Review comment:
ditto: checkNotNull
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
zhijiangW commented on a change in pull request #11515: [FLINK-16744][task]
implement channel state persistence for unaligned checkpoints
URL: https://github.com/apache/flink/pull/11515#discussion_r402108103
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriteRequest.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter.ChannelStateWriteResult;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.state.CheckpointStorageLocationReference;
+import org.apache.flink.util.function.ThrowingConsumer;
+
+abstract class ChannelStateWriteRequest {
+ final long checkpointId;
+
+ ChannelStateWriteRequest(long checkpointId) {
+ this.checkpointId = checkpointId;
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + ", checkpointId=" +
checkpointId;
+ }
+
+ static CheckpointInProgressRequest completeInput(long checkpointId) {
+ return new CheckpointInProgressRequest(checkpointId,
ChannelStateCheckpointWriter::completeInput);
+ }
+
+ static CheckpointInProgressRequest completeOutput(long checkpointId) {
+ return new CheckpointInProgressRequest(checkpointId,
ChannelStateCheckpointWriter::completeOutput);
+ }
+
+ static ChannelStateWriteRequest write(long checkpointId,
InputChannelInfo info, Buffer... flinkBuffers) {
+ return new CheckpointInProgressRequest(checkpointId, writer ->
writer.writeInput(info, flinkBuffers));
+ }
+
+ static ChannelStateWriteRequest write(long checkpointId,
ResultSubpartitionInfo info, Buffer... flinkBuffers) {
+ return new CheckpointInProgressRequest(checkpointId, writer ->
writer.writeOutput(info, flinkBuffers));
+ }
+}
+
+final class CheckpointStartRequest extends ChannelStateWriteRequest {
+ final ChannelStateWriteResult targetResult;
+ final CheckpointStorageLocationReference locationReference;
+
+ CheckpointStartRequest(long checkpointId, ChannelStateWriteResult
targetResult, CheckpointStorageLocationReference locationReference) {
+ super(checkpointId);
+ this.targetResult = targetResult;
+ this.locationReference = locationReference;
Review comment:
nit: checkNotNull
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-16735) FlinkKafkaProducer should check that it is not null before sending a record
[ https://issues.apache.org/jira/browse/FLINK-16735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073453#comment-17073453 ] Aljoscha Krettek commented on FLINK-16735: -- But do you want to change the behaviour? I'm open to suggestions. 😅 > FlinkKafkaProducer should check that it is not null before sending a record > --- > > Key: FLINK-16735 > URL: https://issues.apache.org/jira/browse/FLINK-16735 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.10.0 >Reporter: Shangwen Tang >Assignee: Shangwen Tang >Priority: Major > Attachments: image-2020-03-24-11-40-22-143.png > > > In our user scenario, some users implemented the KafkaSerializationSchema and > sometimes returned a null record, resulting in a null pointer exception > !image-2020-03-24-11-40-22-143.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-16916) The logic of NullableSerializer#copy is wrong
[
https://issues.apache.org/jira/browse/FLINK-16916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek reassigned FLINK-16916:
Assignee: Congxian Qiu(klion26)
> The logic of NullableSerializer#copy is wrong
> -
>
> Key: FLINK-16916
> URL: https://issues.apache.org/jira/browse/FLINK-16916
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Affects Versions: 1.8.3, 1.9.2, 1.10.0
>Reporter: Congxian Qiu(klion26)
>Assignee: Congxian Qiu(klion26)
>Priority: Blocker
> Fix For: 1.8.4, 1.9.3, 1.11.0, 1.10.2
>
>
> When debugging the problem reported by FLINK-16724, Found that the logic of
> {{NullableSerializer#copy}} is wrong. currently, the logic is such as below:
> {code:java}
> public void copy(DataInputView source, DataOutputView target) throws
> IOException {
>boolean isNull = source.readBoolean();
>target.writeBoolean(isNull);
>if (isNull) {
> target.write(padding);
>}
>else {
> originalSerializer.copy(source, target);
>}
> }
> {code}
> we forgot to skip {{paddings.length}} bytes when if the {{padding}}'s length
> is not 0.
> We can correct the logic such as below
> {code:java}
> public void copy(DataInputView source, DataOutputView target) throws
> IOException {
>boolean isNull = deserializeNull(source); // this will skip the padding
> values.
>target.writeBoolean(isNull);
>if (isNull) {
> target.write(padding);
>}
>else {
> originalSerializer.copy(source, target);
>}
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16916) The logic of NullableSerializer#copy is wrong
[
https://issues.apache.org/jira/browse/FLINK-16916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073456#comment-17073456
]
Aljoscha Krettek commented on FLINK-16916:
--
Good analysis!
> The logic of NullableSerializer#copy is wrong
> -
>
> Key: FLINK-16916
> URL: https://issues.apache.org/jira/browse/FLINK-16916
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Affects Versions: 1.8.3, 1.9.2, 1.10.0
>Reporter: Congxian Qiu(klion26)
>Assignee: Congxian Qiu(klion26)
>Priority: Blocker
> Fix For: 1.8.4, 1.9.3, 1.11.0, 1.10.2
>
>
> When debugging the problem reported by FLINK-16724, Found that the logic of
> {{NullableSerializer#copy}} is wrong. currently, the logic is such as below:
> {code:java}
> public void copy(DataInputView source, DataOutputView target) throws
> IOException {
>boolean isNull = source.readBoolean();
>target.writeBoolean(isNull);
>if (isNull) {
> target.write(padding);
>}
>else {
> originalSerializer.copy(source, target);
>}
> }
> {code}
> we forgot to skip {{paddings.length}} bytes when if the {{padding}}'s length
> is not 0.
> We can correct the logic such as below
> {code:java}
> public void copy(DataInputView source, DataOutputView target) throws
> IOException {
>boolean isNull = deserializeNull(source); // this will skip the padding
> values.
>target.writeBoolean(isNull);
>if (isNull) {
> target.write(padding);
>}
>else {
> originalSerializer.copy(source, target);
>}
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-16915) Cannot find compatible factory for specified execution.target (=local)
[
https://issues.apache.org/jira/browse/FLINK-16915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-16915.
Resolution: Duplicate
> Cannot find compatible factory for specified execution.target (=local)
> --
>
> Key: FLINK-16915
> URL: https://issues.apache.org/jira/browse/FLINK-16915
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Affects Versions: 1.11.0
>Reporter: Jaryzhen
>Priority: Minor
>
> It occurred when I run from newly version.
> {code:java}
> Exception in thread "main" java.lang.NullPointerException: Cannot find
> compatible factory for specified execution.target (=local)
> at
> org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:104)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1750)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1658)
> at
> org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:74)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1644)
> at
> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:684)
> at
> org.apache.flink.streaming.scala.examples.wordcount.WordCount$.main(WordCount.scala:89)
> at
> org.apache.flink.streaming.scala.examples.wordcount.WordCount.main(WordCount.scala)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] AHeise opened a new pull request #11611: [FLINK-16917][runtime] Revert FLINK-16245
AHeise opened a new pull request #11611: [FLINK-16917][runtime] Revert FLINK-16245 URL: https://github.com/apache/flink/pull/11611 ## What is the purpose of the change Reverts FLINK-16245 to improve/stabilize table API e2e tests. ## Brief change log ## Verifying this change ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16917) "TPC-DS end-to-end test (Blink planner)" gets stuck
[
https://issues.apache.org/jira/browse/FLINK-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-16917:
---
Labels: pull-request-available test-stability (was: test-stability)
> "TPC-DS end-to-end test (Blink planner)" gets stuck
> ---
>
> Key: FLINK-16917
> URL: https://issues.apache.org/jira/browse/FLINK-16917
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.11.0
>
> Attachments: Screenshot 2020-04-02 08.12.01.png, Screenshot
> 2020-04-02 08.24.28.png, image-2020-04-02-09-32-52-979.png
>
>
> The message you see from the CI system is
> {code}
> ##[error]The job running on agent Hosted Agent ran longer than the maximum
> time of 240 minutes. For more information, see
> https://go.microsoft.com/fwlink/?linkid=2077134
> {code}
> Example:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6899&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee
> The end of the log file looks as follows:
> {code}
> 2020-03-31T23:00:40.5416207Z [INFO]Run TPC-DS query 97 success.
> 2020-03-31T23:00:40.5439265Z [INFO]Run TPC-DS query 98 ...
> 2020-03-31T23:00:40.8269500Z Job has been submitted with JobID
> eec4759ae6d585ee9f8d9f84f1793c0e
> 2020-03-31T23:01:33.4757621Z Program execution finished
> 2020-03-31T23:01:33.4758328Z Job with JobID eec4759ae6d585ee9f8d9f84f1793c0e
> has finished.
> 2020-03-31T23:01:33.4758880Z Job Runtime: 51093 ms
> 2020-03-31T23:01:33.4759057Z
> 2020-03-31T23:01:33.4760999Z [INFO]Run TPC-DS query 98 success.
> 2020-03-31T23:01:33.4761612Z [INFO]Run TPC-DS query 99 ...
> 2020-03-31T23:01:33.7297686Z Job has been submitted with JobID
> f47efc4194df2e0ead677fff239f3dfd
> 2020-03-31T23:01:50.0037484Z ##[error]The operation was canceled.
> 2020-03-31T23:01:50.0091655Z ##[section]Finishing: Run e2e tests
> {code}
> Notice the time difference between "Job has been submitted" and "The
> operation was canceled.". There was nothing happening for 20 minutes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] kl0u commented on a change in pull request #11474: FLINK-10114: Add ORC BulkWriter support for StreamingFileSink
kl0u commented on a change in pull request #11474: FLINK-10114: Add ORC
BulkWriter support for StreamingFileSink
URL: https://github.com/apache/flink/pull/11474#discussion_r402110707
##
File path:
flink-formats/flink-orc-compress/src/main/java/org/apache/flink/formats/orc/vectorizer/Vectorizer.java
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.orc.vectorizer;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+import org.apache.orc.storage.ql.exec.vector.VectorizedRowBatch;
+
+import java.io.IOException;
+import java.io.Serializable;
+
+/**
+ * Implementors of this interface provide the logic to transform their data to
{@link VectorizedRowBatch}.
+ *
+ * @param The type of the element
+ */
+@PublicEvolving
+public interface Vectorizer extends Serializable {
+
+ /**
+* Creates a VectorizedRowBatch containing an array of ColumnVectors
+* from the provided element.
+*
+* @param element The input element
+* @return The VectorizedRowBatch containing the ColumnVectors of the
input element
+* @throws IOException if there is an error while transforming the
input.
+*/
+ VectorizedRowBatch vectorize(T element) throws IOException;
Review comment:
@zenfenan I think it would be really helpful if you could reach out to the
ORC community to help us understand the tradeoffs in this. Thanks for taking
the initiative!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (FLINK-16938) SqlTimestamp has lag when convert long to Timestamp
YufeiLiu created FLINK-16938:
Summary: SqlTimestamp has lag when convert long to Timestamp
Key: FLINK-16938
URL: https://issues.apache.org/jira/browse/FLINK-16938
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.10.0
Reporter: YufeiLiu
When I set rowtime attribute by using expression 'column.rowtime' , and result
type is sql.Timestamp, the result will have lag which is equals with default
timezone offset.
{code:java}
tEnv.fromDataStream(stream, "user_action_time.rowtime, user_name, data");
{code}
I look into the conversion logic, the field was go through 'long ->
SqlTimestamp -> Timestamp' conversion.
{code:java}
long from = System.currentTimeMillis();
long to = SqlTimestamp
.fromEpochMillis(from)
.toTimestamp()
.getTime();
{code}
The result is {{from!=to}}. In {{SqlTimestamp.toTimestamp()}} using
{{Timestamp.valueOf(LocalDateTime dateTime)}} which is contain timezone
infomation, will casue time lag.
>From Timestamp to Timestamp not have this issue, but convert Datastream to
>Table is use StreamRecord.timestamp as rowtime field.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #11611: [FLINK-16917][runtime] Revert FLINK-16245
flinkbot commented on issue #11611: [FLINK-16917][runtime] Revert FLINK-16245 URL: https://github.com/apache/flink/pull/11611#issuecomment-607677105 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit fc3458e7e2861e48b8af1a404dec443a44467488 (Thu Apr 02 07:41:33 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16938) SqlTimestamp has lag when convert long to Timestamp
[
https://issues.apache.org/jira/browse/FLINK-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073461#comment-17073461
]
YufeiLiu commented on FLINK-16938:
--
cc [~jark]
> SqlTimestamp has lag when convert long to Timestamp
> ---
>
> Key: FLINK-16938
> URL: https://issues.apache.org/jira/browse/FLINK-16938
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.10.0
>Reporter: YufeiLiu
>Priority: Major
>
> When I set rowtime attribute by using expression 'column.rowtime' , and
> result type is sql.Timestamp, the result will have lag which is equals with
> default timezone offset.
> {code:java}
> tEnv.fromDataStream(stream, "user_action_time.rowtime, user_name, data");
> {code}
> I look into the conversion logic, the field was go through 'long ->
> SqlTimestamp -> Timestamp' conversion.
> {code:java}
> long from = System.currentTimeMillis();
> long to = SqlTimestamp
> .fromEpochMillis(from)
> .toTimestamp()
> .getTime();
> {code}
> The result is {{from!=to}}. In {{SqlTimestamp.toTimestamp()}} using
> {{Timestamp.valueOf(LocalDateTime dateTime)}} which is contain timezone
> infomation, will casue time lag.
> From Timestamp to Timestamp not have this issue, but convert Datastream to
> Table is use StreamRecord.timestamp as rowtime field.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-16915) Cannot find compatible factory for specified execution.target (=local)
[
https://issues.apache.org/jira/browse/FLINK-16915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073451#comment-17073451
]
Jaryzhen edited comment on FLINK-16915 at 4/2/20, 7:44 AM:
---
Yeah, it's works, thx
was (Author: jaryzhen):
Yeah, it's work, thx
> Cannot find compatible factory for specified execution.target (=local)
> --
>
> Key: FLINK-16915
> URL: https://issues.apache.org/jira/browse/FLINK-16915
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Affects Versions: 1.11.0
>Reporter: Jaryzhen
>Priority: Minor
>
> It occurred when I run from newly version.
> {code:java}
> Exception in thread "main" java.lang.NullPointerException: Cannot find
> compatible factory for specified execution.target (=local)
> at
> org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:104)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1750)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1658)
> at
> org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:74)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1644)
> at
> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:684)
> at
> org.apache.flink.streaming.scala.examples.wordcount.WordCount$.main(WordCount.scala:89)
> at
> org.apache.flink.streaming.scala.examples.wordcount.WordCount.main(WordCount.scala)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16903) Add sink.parallelism for file system factory
[
https://issues.apache.org/jira/browse/FLINK-16903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073465#comment-17073465
]
Jark Wu commented on FLINK-16903:
-
I think in the long term, {{SupportsCustomParallelism}} is needed to inference
a default parallelism of source and sink, e.g. use the number of Kafka
partitions. So I'm +1 to this way. But I would like to call it
{{SupportsReportSplits}} or {{SupportsReportPartitions}}, so it can be used not
only for specifying a parallelism, but can also be used as a max-parallelism.
> Add sink.parallelism for file system factory
>
>
> Key: FLINK-16903
> URL: https://issues.apache.org/jira/browse/FLINK-16903
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Runtime
>Reporter: Jingsong Lee
>Priority: Major
> Fix For: 1.11.0
>
>
> A single task may be writing multiple files at the same time. If the
> parallelism is too high, it may lead to a large number of small files. If the
> parallelism is too small, the performance is not enough. This requires that
> the user can specify parallelism.
> * Default is the same as upstream transformation
> * Users can specify parallelism too.
> |‘connector.sink.parallelism’ = ...|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11574: [FLINK-16859][table-runtime] Introduce file system table factory
flinkbot edited a comment on issue #11574: [FLINK-16859][table-runtime] Introduce file system table factory URL: https://github.com/apache/flink/pull/11574#issuecomment-606453374 ## CI report: * 5b17b69ae7a5aab2fff50545abff32f1bc0ce8ff Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/157698652) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6960) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11542: [FLINK-16303][rest] Enable retrieval of custom JobManager log files
flinkbot edited a comment on issue #11542: [FLINK-16303][rest] Enable retrieval of custom JobManager log files URL: https://github.com/apache/flink/pull/11542#issuecomment-604812411 ## CI report: * c45cb2a911fbd9a775fee2725370081f5d15b764 UNKNOWN * 74a2a88b887fda108d960c238ecab9411c3308a8 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/157700236) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6962) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-12071) HadoopRecoverableWriterTest fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-12071.
Resolution: Duplicate
Closing as per [~liyu]'s suggestion.
> HadoopRecoverableWriterTest fails on Travis
> ---
>
> Key: FLINK-12071
> URL: https://issues.apache.org/jira/browse/FLINK-12071
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
>Affects Versions: 1.9.0
>Reporter: Chesnay Schepler
>Assignee: leesf
>Priority: Critical
> Labels: test-stability
>
> https://travis-ci.org/apache/flink/jobs/513373067
> {code}
> testExceptionWritingAfterCloseForCommit(org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriterTest)
> Time elapsed: 0.031 s <<< ERROR!
> java.lang.Exception: Unexpected exception, expected but
> was
> Caused by: java.lang.IllegalArgumentException: Self-suppression not permitted
> Caused by: java.io.IOException: The stream is closed
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16931) Large _metadata file lead to JobManager not responding when restart
[
https://issues.apache.org/jira/browse/FLINK-16931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Zagrebin updated FLINK-16931:
Component/s: Runtime / Coordination
Runtime / Checkpointing
> Large _metadata file lead to JobManager not responding when restart
> ---
>
> Key: FLINK-16931
> URL: https://issues.apache.org/jira/browse/FLINK-16931
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Reporter: Lu Niu
>Priority: Major
>
> When _metadata file is big, JobManager could never recover from checkpoint.
> It fall into a loop that fetch checkpoint -> JM timeout -> restart. Here is
> related log:
> {code:java}
> 2020-04-01 17:08:25,689 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Recovering checkpoints from ZooKeeper.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found
> 3 checkpoints in ZooKeeper.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to fetch 3 checkpoints from storage.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to retrieve checkpoint 50.
> 2020-04-01 17:08:48,589 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to retrieve checkpoint 51.
> 2020-04-01 17:09:12,775 INFO org.apache.flink.yarn.YarnResourceManager - The
> heartbeat of JobManager with id 02500708baf0bb976891c391afd3d7d5 timed out.
> {code}
> Digging into the code, looks like ExecutionGraph::restart runs in JobMaster
> main thread and finally calls
> ZooKeeperCompletedCheckpointStore::retrieveCompletedCheckpoint which download
> file form DFS. The main thread is basically blocked for a while because of
> this. One possible solution is to making the downloading part async. More
> things might need to consider as the original change tries to make it
> single-threaded. [https://github.com/apache/flink/pull/7568]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on a change in pull request #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
zhijiangW commented on a change in pull request #11515: [FLINK-16744][task]
implement channel state persistence for unaligned checkpoints
URL: https://github.com/apache/flink/pull/11515#discussion_r402115061
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateCheckpointWriter.java
##
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter.ChannelStateWriteResult;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.state.AbstractChannelStateHandle;
+import org.apache.flink.runtime.state.CheckpointStreamFactory;
+import
org.apache.flink.runtime.state.CheckpointStreamFactory.CheckpointStateOutputStream;
+import org.apache.flink.runtime.state.InputChannelStateHandle;
+import org.apache.flink.runtime.state.ResultSubpartitionStateHandle;
+import org.apache.flink.runtime.state.StreamStateHandle;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.RunnableWithException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BiFunction;
+
+import static org.apache.flink.runtime.state.CheckpointedStateScope.EXCLUSIVE;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Writes channel state for a specific checkpoint-subtask-attempt triple.
+ */
+@NotThreadSafe
+class ChannelStateCheckpointWriter {
+ private static final Logger LOG =
LoggerFactory.getLogger(ChannelStateCheckpointWriter.class);
+
+ private final DataOutputStream dataStream;
+ private final CheckpointStateOutputStream checkpointStream;
+ private final ChannelStateWriteResult result;
+ private final Map> inputChannelOffsets =
new HashMap<>();
+ private final Map>
resultSubpartitionOffsets = new HashMap<>();
+ private final ChannelStateSerializer serializer;
+ private final long checkpointId;
+ private boolean inputCompleted = false;
+ private boolean outputCompleted = false;
+ private final Runnable onComplete;
+
+ ChannelStateCheckpointWriter(
+ CheckpointStartRequest startCheckpointItem,
+ CheckpointStreamFactory streamFactory,
+ ChannelStateSerializer serializer,
+ Runnable onComplete) throws Exception {
+ this(startCheckpointItem.checkpointId,
startCheckpointItem.targetResult,
streamFactory.createCheckpointStateOutputStream(EXCLUSIVE), serializer,
onComplete);
+ }
+
+ ChannelStateCheckpointWriter(
+ long checkpointId,
+ ChannelStateWriteResult result,
+ CheckpointStateOutputStream stream,
+ ChannelStateSerializer serializer,
+ Runnable onComplete) throws Exception {
+ this(checkpointId, result, serializer, onComplete, stream, new
DataOutputStream(stream));
+ }
+
+ ChannelStateCheckpointWriter(
+ long checkpointId,
+ ChannelStateWriteResult result,
+ ChannelStateSerializer serializer,
+ Runnable onComplete,
+ CheckpointStateOutputStream checkpointStateOutputStream,
+ DataOutputStream dataStream) throws Exception {
+ this.checkpointId = checkpointId;
+ this.result = checkNotNull(result);
+ this.checkpointStream =
checkNotNull(checkpointStateOutputStream);
+ this.serializer = checkNotNull(serializer);
+ this.dataStream = checkNotNull(dataStream);
+ this.onComplete = checkNotNull(onComplete);
+ runWithChecks(() -> seriali
[jira] [Commented] (FLINK-16931) Large _metadata file lead to JobManager not responding when restart
[
https://issues.apache.org/jira/browse/FLINK-16931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073470#comment-17073470
]
Andrey Zagrebin commented on FLINK-16931:
-
Thanks for reporting this [~qqibrow]
Have you checked what makes the _metadata so big?
> Large _metadata file lead to JobManager not responding when restart
> ---
>
> Key: FLINK-16931
> URL: https://issues.apache.org/jira/browse/FLINK-16931
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Reporter: Lu Niu
>Priority: Major
>
> When _metadata file is big, JobManager could never recover from checkpoint.
> It fall into a loop that fetch checkpoint -> JM timeout -> restart. Here is
> related log:
> {code:java}
> 2020-04-01 17:08:25,689 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Recovering checkpoints from ZooKeeper.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found
> 3 checkpoints in ZooKeeper.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to fetch 3 checkpoints from storage.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to retrieve checkpoint 50.
> 2020-04-01 17:08:48,589 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to retrieve checkpoint 51.
> 2020-04-01 17:09:12,775 INFO org.apache.flink.yarn.YarnResourceManager - The
> heartbeat of JobManager with id 02500708baf0bb976891c391afd3d7d5 timed out.
> {code}
> Digging into the code, looks like ExecutionGraph::restart runs in JobMaster
> main thread and finally calls
> ZooKeeperCompletedCheckpointStore::retrieveCompletedCheckpoint which download
> file form DFS. The main thread is basically blocked for a while because of
> this. One possible solution is to making the downloading part async. More
> things might need to consider as the original change tries to make it
> single-threaded. [https://github.com/apache/flink/pull/7568]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] kl0u commented on issue #11474: FLINK-10114: Add ORC BulkWriter support for StreamingFileSink
kl0u commented on issue #11474: FLINK-10114: Add ORC BulkWriter support for StreamingFileSink URL: https://github.com/apache/flink/pull/11474#issuecomment-607682169 @zenfenan I understand that time is of an issue these days so there is no reason to hurry. The PR can wait and there is no reason for you to get stressed about it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
zhijiangW commented on a change in pull request #11515: [FLINK-16744][task]
implement channel state persistence for unaligned checkpoints
URL: https://github.com/apache/flink/pull/11515#discussion_r402120167
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateCheckpointWriter.java
##
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter.ChannelStateWriteResult;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.state.AbstractChannelStateHandle;
+import org.apache.flink.runtime.state.CheckpointStreamFactory;
+import
org.apache.flink.runtime.state.CheckpointStreamFactory.CheckpointStateOutputStream;
+import org.apache.flink.runtime.state.InputChannelStateHandle;
+import org.apache.flink.runtime.state.ResultSubpartitionStateHandle;
+import org.apache.flink.runtime.state.StreamStateHandle;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.RunnableWithException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BiFunction;
+
+import static org.apache.flink.runtime.state.CheckpointedStateScope.EXCLUSIVE;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Writes channel state for a specific checkpoint-subtask-attempt triple.
+ */
+@NotThreadSafe
+class ChannelStateCheckpointWriter {
+ private static final Logger LOG =
LoggerFactory.getLogger(ChannelStateCheckpointWriter.class);
+
+ private final DataOutputStream dataStream;
+ private final CheckpointStateOutputStream checkpointStream;
+ private final ChannelStateWriteResult result;
+ private final Map> inputChannelOffsets =
new HashMap<>();
+ private final Map>
resultSubpartitionOffsets = new HashMap<>();
+ private final ChannelStateSerializer serializer;
+ private final long checkpointId;
+ private boolean inputCompleted = false;
+ private boolean outputCompleted = false;
+ private final Runnable onComplete;
+
+ ChannelStateCheckpointWriter(
+ CheckpointStartRequest startCheckpointItem,
+ CheckpointStreamFactory streamFactory,
+ ChannelStateSerializer serializer,
+ Runnable onComplete) throws Exception {
+ this(startCheckpointItem.checkpointId,
startCheckpointItem.targetResult,
streamFactory.createCheckpointStateOutputStream(EXCLUSIVE), serializer,
onComplete);
+ }
+
+ ChannelStateCheckpointWriter(
+ long checkpointId,
+ ChannelStateWriteResult result,
+ CheckpointStateOutputStream stream,
+ ChannelStateSerializer serializer,
+ Runnable onComplete) throws Exception {
+ this(checkpointId, result, serializer, onComplete, stream, new
DataOutputStream(stream));
+ }
+
+ ChannelStateCheckpointWriter(
+ long checkpointId,
+ ChannelStateWriteResult result,
+ ChannelStateSerializer serializer,
+ Runnable onComplete,
+ CheckpointStateOutputStream checkpointStateOutputStream,
+ DataOutputStream dataStream) throws Exception {
+ this.checkpointId = checkpointId;
+ this.result = checkNotNull(result);
+ this.checkpointStream =
checkNotNull(checkpointStateOutputStream);
+ this.serializer = checkNotNull(serializer);
+ this.dataStream = checkNotNull(dataStream);
+ this.onComplete = checkNotNull(onComplete);
+ runWithChecks(() -> seriali
[jira] [Commented] (FLINK-16938) SqlTimestamp has lag when convert long to Timestamp
[
https://issues.apache.org/jira/browse/FLINK-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073473#comment-17073473
]
Jark Wu commented on FLINK-16938:
-
Hi [~liuyufei], could you give an example which can re-produce your problem?
Btw, {{java.sql.Timestamp}} is not a semantically clear definition, the
conversion between SqlTimestamp and Timestamp requires the default time zone.
That's why you will see the lag. {{Instant}} and {{LocalDateTime}} are the
suggested class to use if you want to have a clear semantic.
> SqlTimestamp has lag when convert long to Timestamp
> ---
>
> Key: FLINK-16938
> URL: https://issues.apache.org/jira/browse/FLINK-16938
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.10.0
>Reporter: YufeiLiu
>Priority: Major
>
> When I set rowtime attribute by using expression 'column.rowtime' , and
> result type is sql.Timestamp, the result will have lag which is equals with
> default timezone offset.
> {code:java}
> tEnv.fromDataStream(stream, "user_action_time.rowtime, user_name, data");
> {code}
> I look into the conversion logic, the field was go through 'long ->
> SqlTimestamp -> Timestamp' conversion.
> {code:java}
> long from = System.currentTimeMillis();
> long to = SqlTimestamp
> .fromEpochMillis(from)
> .toTimestamp()
> .getTime();
> {code}
> The result is {{from!=to}}. In {{SqlTimestamp.toTimestamp()}} using
> {{Timestamp.valueOf(LocalDateTime dateTime)}} which is contain timezone
> infomation, will casue time lag.
> From Timestamp to Timestamp not have this issue, but convert Datastream to
> Table is use StreamRecord.timestamp as rowtime field.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rmetzger commented on issue #11611: [FLINK-16917][runtime] Revert FLINK-16245
rmetzger commented on issue #11611: [FLINK-16917][runtime] Revert FLINK-16245 URL: https://github.com/apache/flink/pull/11611#issuecomment-607686700 Thanks a lot! Since this is somewhat time-critical, I triggered two runs in a private testing azure instance: - https://dev.azure.com/georgeryan1322/Flink/_build/results?buildId=260&view=results - https://dev.azure.com/georgeryan1322/Flink/_build/results?buildId=261&view=results This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor
flinkbot edited a comment on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor URL: https://github.com/apache/flink/pull/11571#issuecomment-606377768 ## CI report: * 91507d011419afe848d1a90ae0027cd2511817e5 UNKNOWN * 235a3dad633fd7a3654f0fa57be79787da769334 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/157703537) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6964) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11611: [FLINK-16917][runtime] Revert FLINK-16245
flinkbot commented on issue #11611: [FLINK-16917][runtime] Revert FLINK-16245 URL: https://github.com/apache/flink/pull/11611#issuecomment-607686865 ## CI report: * fc3458e7e2861e48b8af1a404dec443a44467488 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16876) Make TtlTimeProvider configurable when creating keyed state backend
[ https://issues.apache.org/jira/browse/FLINK-16876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073484#comment-17073484 ] dalongliu commented on FLINK-16876: --- [~yunta] As you say, you help to implement customized mock StreamTaskStateInitializerImpl or StateBackend with TtlTimeProvider in constructor for unit tests? > Make TtlTimeProvider configurable when creating keyed state backend > --- > > Key: FLINK-16876 > URL: https://issues.apache.org/jira/browse/FLINK-16876 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.10.0 >Reporter: Yun Tang >Priority: Major > > Currently, we would always use TtlTimeProvider.DEFAULT to create keyed state > backend. This is somehow acceptable since we only support processing time for > TTL now. However, this would make UT tests which would verify TTL logic not > so convenient like FLINK-16581. > I propose to let TtlTimeProvider configurable when creating keyed state > backend to not block other feature development. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] aljoscha commented on issue #11607: [FLINK-16560] Forward Configuration in PackagedProgramUtils#getPipelineFromProgram
aljoscha commented on issue #11607: [FLINK-16560] Forward Configuration in PackagedProgramUtils#getPipelineFromProgram URL: https://github.com/apache/flink/pull/11607#issuecomment-607689442 Thanks for your review! I addressed your comments and merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16929) Session Window produces sessions randomly
[
https://issues.apache.org/jira/browse/FLINK-16929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073485#comment-17073485
]
Aljoscha Krettek commented on FLINK-16929:
--
Hmm, that seems quite weird. Just for debugging, could you try running a
representative workload with both RocksDB and the FileStateBackend to see if
there's a difference?
> Session Window produces sessions randomly
> -
>
> Key: FLINK-16929
> URL: https://issues.apache.org/jira/browse/FLINK-16929
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.9.1
>Reporter: Ori Popowski
>Priority: Major
> Attachments: image-2020-04-01-19-56-00-239.png,
> image-2020-04-01-19-56-27-720.png
>
>
> We have a Flink job which keyBys session ID (sId), and uses a session window
> with 30 minutes gap:
> {code:java}
> inputStream
> .keyBy(keySelector)
> .window(EventTimeSessionWindows.withGap(Time.minutes(30)))
> .allowedLateness(Time.seconds(0L))
> {code}
> This Flink job reads from Kinesis stream.
> Lately (I suspect after upgrading from 1.5.4 to 1.9.1) we get too many
> sessions, with gaps of several seconds (instead of 30 minutes).
> We have no idea why it's happening and suspect a Flink bug or a state backend
> bug (we use RocksDB).
> I haven't found any indication in the logs except for some read throughput
> warnings which were resolved by a backoff.
> Attached is a table of derived sessions, and then the raw events
> *Sessions*
> !image-2020-04-01-19-56-00-239.png|width=753,height=406!
>
> *Events*
>
> !image-2020-04-01-19-56-27-720.png|width=312,height=383!
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] aljoscha closed pull request #11607: [FLINK-16560] Forward Configuration in PackagedProgramUtils#getPipelineFromProgram
aljoscha closed pull request #11607: [FLINK-16560] Forward Configuration in PackagedProgramUtils#getPipelineFromProgram URL: https://github.com/apache/flink/pull/11607 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16917) "TPC-DS end-to-end test (Blink planner)" gets stuck
[
https://issues.apache.org/jira/browse/FLINK-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073488#comment-17073488
]
Robert Metzger commented on FLINK-16917:
[~AHeise] is. I've assinged you to the ticket.
> "TPC-DS end-to-end test (Blink planner)" gets stuck
> ---
>
> Key: FLINK-16917
> URL: https://issues.apache.org/jira/browse/FLINK-16917
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.11.0
>
> Attachments: Screenshot 2020-04-02 08.12.01.png, Screenshot
> 2020-04-02 08.24.28.png, image-2020-04-02-09-32-52-979.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The message you see from the CI system is
> {code}
> ##[error]The job running on agent Hosted Agent ran longer than the maximum
> time of 240 minutes. For more information, see
> https://go.microsoft.com/fwlink/?linkid=2077134
> {code}
> Example:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6899&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee
> The end of the log file looks as follows:
> {code}
> 2020-03-31T23:00:40.5416207Z [INFO]Run TPC-DS query 97 success.
> 2020-03-31T23:00:40.5439265Z [INFO]Run TPC-DS query 98 ...
> 2020-03-31T23:00:40.8269500Z Job has been submitted with JobID
> eec4759ae6d585ee9f8d9f84f1793c0e
> 2020-03-31T23:01:33.4757621Z Program execution finished
> 2020-03-31T23:01:33.4758328Z Job with JobID eec4759ae6d585ee9f8d9f84f1793c0e
> has finished.
> 2020-03-31T23:01:33.4758880Z Job Runtime: 51093 ms
> 2020-03-31T23:01:33.4759057Z
> 2020-03-31T23:01:33.4760999Z [INFO]Run TPC-DS query 98 success.
> 2020-03-31T23:01:33.4761612Z [INFO]Run TPC-DS query 99 ...
> 2020-03-31T23:01:33.7297686Z Job has been submitted with JobID
> f47efc4194df2e0ead677fff239f3dfd
> 2020-03-31T23:01:50.0037484Z ##[error]The operation was canceled.
> 2020-03-31T23:01:50.0091655Z ##[section]Finishing: Run e2e tests
> {code}
> Notice the time difference between "Job has been submitted" and "The
> operation was canceled.". There was nothing happening for 20 minutes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-16917) "TPC-DS end-to-end test (Blink planner)" gets stuck
[
https://issues.apache.org/jira/browse/FLINK-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger reassigned FLINK-16917:
--
Assignee: Arvid Heise
> "TPC-DS end-to-end test (Blink planner)" gets stuck
> ---
>
> Key: FLINK-16917
> URL: https://issues.apache.org/jira/browse/FLINK-16917
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Assignee: Arvid Heise
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.11.0
>
> Attachments: Screenshot 2020-04-02 08.12.01.png, Screenshot
> 2020-04-02 08.24.28.png, image-2020-04-02-09-32-52-979.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The message you see from the CI system is
> {code}
> ##[error]The job running on agent Hosted Agent ran longer than the maximum
> time of 240 minutes. For more information, see
> https://go.microsoft.com/fwlink/?linkid=2077134
> {code}
> Example:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6899&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee
> The end of the log file looks as follows:
> {code}
> 2020-03-31T23:00:40.5416207Z [INFO]Run TPC-DS query 97 success.
> 2020-03-31T23:00:40.5439265Z [INFO]Run TPC-DS query 98 ...
> 2020-03-31T23:00:40.8269500Z Job has been submitted with JobID
> eec4759ae6d585ee9f8d9f84f1793c0e
> 2020-03-31T23:01:33.4757621Z Program execution finished
> 2020-03-31T23:01:33.4758328Z Job with JobID eec4759ae6d585ee9f8d9f84f1793c0e
> has finished.
> 2020-03-31T23:01:33.4758880Z Job Runtime: 51093 ms
> 2020-03-31T23:01:33.4759057Z
> 2020-03-31T23:01:33.4760999Z [INFO]Run TPC-DS query 98 success.
> 2020-03-31T23:01:33.4761612Z [INFO]Run TPC-DS query 99 ...
> 2020-03-31T23:01:33.7297686Z Job has been submitted with JobID
> f47efc4194df2e0ead677fff239f3dfd
> 2020-03-31T23:01:50.0037484Z ##[error]The operation was canceled.
> 2020-03-31T23:01:50.0091655Z ##[section]Finishing: Run e2e tests
> {code}
> Notice the time difference between "Job has been submitted" and "The
> operation was canceled.". There was nothing happening for 20 minutes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-16560) Forward Configuration in PackagedProgramUtils#getPipelineFromProgram
[ https://issues.apache.org/jira/browse/FLINK-16560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed FLINK-16560. Resolution: Fixed master: a24734ea339872763306b44770678c4ace6a369f release-1.10: bf3c84370748e444257b49c6cdd52663f7379436 > Forward Configuration in PackagedProgramUtils#getPipelineFromProgram > > > Key: FLINK-16560 > URL: https://issues.apache.org/jira/browse/FLINK-16560 > Project: Flink > Issue Type: Bug > Components: API / DataStream, Runtime / Configuration >Affects Versions: 1.10.0 >Reporter: Zhu Zhu >Assignee: Aljoscha Krettek >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.1, 1.11.0 > > Time Spent: 40m > Remaining Estimate: 0h > > PackagedProgramUtils#createJobGraph(...) is used to generate JobGraph in k8s > job mode. > The problem is that the configuration field of StreamExecutionEnvironment is > a newly created one when building the job program. This is because > StreamPlanEnvironment ctor will base on the no param version ctor of > StreamExecutionEnvironment. > This may lead to an unexpected result when invoking > StreamExecutionEnvironment#configure(...) which relies on the configuration. > Many configurations in the flink conf file will not be respected, like > pipeline.time-characteristic, pipeline.operator-chaining, > execution.buffer-timeout, and state backend configs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] aljoscha commented on issue #11608: [FLINK-16560] Forward Configuration in PackagedProgramUtils#getPipelineFromProgram
aljoscha commented on issue #11608: [FLINK-16560] Forward Configuration in PackagedProgramUtils#getPipelineFromProgram URL: https://github.com/apache/flink/pull/11608#issuecomment-607692494 Merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] aljoscha closed pull request #11608: [FLINK-16560] Forward Configuration in PackagedProgramUtils#getPipelineFromProgram
aljoscha closed pull request #11608: [FLINK-16560] Forward Configuration in PackagedProgramUtils#getPipelineFromProgram URL: https://github.com/apache/flink/pull/11608 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
zhijiangW commented on a change in pull request #11515: [FLINK-16744][task]
implement channel state persistence for unaligned checkpoints
URL: https://github.com/apache/flink/pull/11515#discussion_r402131847
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateCheckpointWriter.java
##
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint.channel;
+
+import
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter.ChannelStateWriteResult;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.state.AbstractChannelStateHandle;
+import org.apache.flink.runtime.state.CheckpointStreamFactory;
+import
org.apache.flink.runtime.state.CheckpointStreamFactory.CheckpointStateOutputStream;
+import org.apache.flink.runtime.state.InputChannelStateHandle;
+import org.apache.flink.runtime.state.ResultSubpartitionStateHandle;
+import org.apache.flink.runtime.state.StreamStateHandle;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.function.RunnableWithException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BiFunction;
+
+import static org.apache.flink.runtime.state.CheckpointedStateScope.EXCLUSIVE;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Writes channel state for a specific checkpoint-subtask-attempt triple.
+ */
+@NotThreadSafe
+class ChannelStateCheckpointWriter {
+ private static final Logger LOG =
LoggerFactory.getLogger(ChannelStateCheckpointWriter.class);
+
+ private final DataOutputStream dataStream;
+ private final CheckpointStateOutputStream checkpointStream;
+ private final ChannelStateWriteResult result;
+ private final Map> inputChannelOffsets =
new HashMap<>();
+ private final Map>
resultSubpartitionOffsets = new HashMap<>();
+ private final ChannelStateSerializer serializer;
+ private final long checkpointId;
+ private boolean inputCompleted = false;
+ private boolean outputCompleted = false;
+ private final Runnable onComplete;
+
+ ChannelStateCheckpointWriter(
+ CheckpointStartRequest startCheckpointItem,
+ CheckpointStreamFactory streamFactory,
+ ChannelStateSerializer serializer,
+ Runnable onComplete) throws Exception {
+ this(startCheckpointItem.checkpointId,
startCheckpointItem.targetResult,
streamFactory.createCheckpointStateOutputStream(EXCLUSIVE), serializer,
onComplete);
+ }
+
+ ChannelStateCheckpointWriter(
+ long checkpointId,
+ ChannelStateWriteResult result,
+ CheckpointStateOutputStream stream,
+ ChannelStateSerializer serializer,
+ Runnable onComplete) throws Exception {
+ this(checkpointId, result, serializer, onComplete, stream, new
DataOutputStream(stream));
+ }
+
+ ChannelStateCheckpointWriter(
+ long checkpointId,
+ ChannelStateWriteResult result,
+ ChannelStateSerializer serializer,
+ Runnable onComplete,
+ CheckpointStateOutputStream checkpointStateOutputStream,
+ DataOutputStream dataStream) throws Exception {
+ this.checkpointId = checkpointId;
+ this.result = checkNotNull(result);
+ this.checkpointStream =
checkNotNull(checkpointStateOutputStream);
+ this.serializer = checkNotNull(serializer);
+ this.dataStream = checkNotNull(dataStream);
+ this.onComplete = checkNotNull(onComplete);
+ runWithChecks(() -> seriali
[jira] [Commented] (FLINK-16906) Various tests failing with: Could not register mbeans javax.management.InstanceAlreadyExistsException
[
https://issues.apache.org/jira/browse/FLINK-16906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073495#comment-17073495
]
Robert Metzger commented on FLINK-16906:
Since so many issues are potentially related to this one, I would really like
to understand what's going on.
[~danny0405] how are you progressing on this ticket? Please let me know if you
don't have time to work on this. I can spend some time looking into disabling
the JMX integration, but I don't want to work on it if somebody else is
potentially working on it.
> Various tests failing with: Could not register mbeans
> javax.management.InstanceAlreadyExistsException
> -
>
> Key: FLINK-16906
> URL: https://issues.apache.org/jira/browse/FLINK-16906
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: Piotr Nowojski
>Assignee: Danny Chen
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *Probably it's better to analyse the issue not from this initially reported
> E2E test, but from some ITCase posted as "another instance" in the comment
> below.*
> Watch out, logs are very large and on my machine didn't load in the browser
> (I had to wget the raw log):
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6889&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> raw log:
> https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_apis/build/builds/6889/logs/145
> {noformat}
> [FAIL] 'TPC-DS end-to-end test (Blink planner)' failed after 80 minutes and
> 59 seconds! Test exited with exit code 0 but the logs contained errors,
> exceptions or non-empty .out files
> {noformat}
> The exception tha probably caused the test to fail:
> {noformat}
> 2020-03-31T18:06:08.7602474Z 2020-03-31 16:47:03,397
> HashJoin(joinType=[InnerJoin], where=[(ss_sold_date_sk = d_date_sk)],
> select=[ss_sold_date_sk, ss_customer_sk, ss_ext_discount_amt,
> ss_ext_sales_price, ss_ext_wholesale_cost, ss_ex
> t_list_price, d_date_sk], isBroadcast=[true], build=[right]) ->
> Calc(select=[ss_customer_sk, ss_ext_discount_amt, ss_ext_sales_price,
> ss_ext_wholesale_cost, ss_ext_list_price]) (4/4) ERROR Could not register
> mbeans javax.management.I
> nstanceAlreadyExistsException:
> org.apache.logging.log4j2:type=791f33ab,component=Loggers,name=
> 2020-03-31T18:06:08.7604158Zat
> com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> 2020-03-31T18:06:08.7604762Zat
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> 2020-03-31T18:06:08.7605457Zat
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> 2020-03-31T18:06:08.7606145Zat
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> 2020-03-31T18:06:08.7607040Zat
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> 2020-03-31T18:06:08.7607589Zat
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> 2020-03-31T18:06:08.7608064Zat
> org.apache.logging.log4j.core.jmx.Server.register(Server.java:393)
> 2020-03-31T18:06:08.7608523Zat
> org.apache.logging.log4j.core.jmx.Server.registerLoggerConfigs(Server.java:362)
> 2020-03-31T18:06:08.7609052Zat
> org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:186)
> 2020-03-31T18:06:08.7609689Zat
> org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:141)
> 2020-03-31T18:06:08.7610211Zat
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:590)
> 2020-03-31T18:06:08.7610726Zat
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
> 2020-03-31T18:06:08.7611211Zat
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
> 2020-03-31T18:06:08.7611753Zat
> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
> 2020-03-31T18:06:08.7612279Zat
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
> 2020-03-31T18:06:08.7612832Zat
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
> 2020-03-31T18:06:08.7613468Zat
> org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
> 2020-03-31T18:06:08.7614156Zat
> org.apache.logging.log4j.spi.AbstractLogg
[jira] [Updated] (FLINK-16939) TaskManagerMessageParameters#taskManagerIdParameter is not declared final
[
https://issues.apache.org/jira/browse/FLINK-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-16939:
-
Priority: Minor (was: Major)
> TaskManagerMessageParameters#taskManagerIdParameter is not declared final
> -
>
> Key: FLINK-16939
> URL: https://issues.apache.org/jira/browse/FLINK-16939
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST
>Affects Versions: 1.11.0
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Minor
> Fix For: 1.11.0
>
>
> The field {{TaskManagerMessageParameters#taskManagerIdParameter}} is not
> declared final but it should be.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16939) TaskManagerMessageParameters#taskManagerIdParameter is not declared final
Gary Yao created FLINK-16939:
Summary: TaskManagerMessageParameters#taskManagerIdParameter is
not declared final
Key: FLINK-16939
URL: https://issues.apache.org/jira/browse/FLINK-16939
Project: Flink
Issue Type: Bug
Components: Runtime / REST
Affects Versions: 1.11.0
Reporter: Gary Yao
Assignee: Gary Yao
Fix For: 1.11.0
The field {{TaskManagerMessageParameters#taskManagerIdParameter}} is not
declared final but it should be.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on a change in pull request #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
zhijiangW commented on a change in pull request #11515: [FLINK-16744][task]
implement channel state persistence for unaligned checkpoints
URL: https://github.com/apache/flink/pull/11515#discussion_r402133516
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriter.java
##
@@ -18,19 +18,51 @@
package org.apache.flink.runtime.checkpoint.channel;
import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.checkpoint.CheckpointOptions;
import org.apache.flink.runtime.io.network.buffer.Buffer;
-import org.apache.flink.runtime.state.StateObject;
+import org.apache.flink.runtime.state.InputChannelStateHandle;
+import org.apache.flink.runtime.state.ResultSubpartitionStateHandle;
+import java.io.Closeable;
import java.util.Collection;
import java.util.Collections;
import java.util.concurrent.CompletableFuture;
-import java.util.concurrent.Future;
/**
* Writes channel state during checkpoint/savepoint.
*/
@Internal
-public interface ChannelStateWriter extends AutoCloseable {
+public interface ChannelStateWriter extends Closeable {
+
+ /**
+* Channel state write result.
+*/
+ class ChannelStateWriteResult {
+ final CompletableFuture>
inputChannelStateHandles;
+ final
CompletableFuture>
resultSubpartitionStateHandles;
+
+ ChannelStateWriteResult() {
+ this(new CompletableFuture<>(), new
CompletableFuture<>());
+ }
+
+
ChannelStateWriteResult(CompletableFuture>
i, CompletableFuture> r) {
Review comment:
nit: `i, r` give some meaningful names and split the arguments in separate
line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11542: [FLINK-16303][rest] Enable retrieval of custom JobManager log files
flinkbot edited a comment on issue #11542: [FLINK-16303][rest] Enable retrieval of custom JobManager log files URL: https://github.com/apache/flink/pull/11542#issuecomment-604812411 ## CI report: * c45cb2a911fbd9a775fee2725370081f5d15b764 UNKNOWN * 74a2a88b887fda108d960c238ecab9411c3308a8 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/157700236) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6962) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11602: [FLINK-16912][parquet] Introduce table row write support for parquet writer
flinkbot edited a comment on issue #11602: [FLINK-16912][parquet] Introduce table row write support for parquet writer URL: https://github.com/apache/flink/pull/11602#issuecomment-607241516 ## CI report: * b28a0ffe70a414addafd0d3b481292a2831d4e10 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/157506831) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6937) * 3f7c4f6dd3c6f9cdb66d9d5a1dcb456b567436c3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11611: [FLINK-16917][runtime] Revert FLINK-16245
flinkbot edited a comment on issue #11611: [FLINK-16917][runtime] Revert FLINK-16245 URL: https://github.com/apache/flink/pull/11611#issuecomment-607686865 ## CI report: * fc3458e7e2861e48b8af1a404dec443a44467488 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/157709956) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6966) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] GJL opened a new pull request #11612: [FLINK-16939][rest] Declare taskManagerIdParameter final
GJL opened a new pull request #11612: [FLINK-16939][rest] Declare taskManagerIdParameter final URL: https://github.com/apache/flink/pull/11612 ## What is the purpose of the change *Declare `taskManagerIdParameter` final* ## Brief change log - *See commit* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16939) TaskManagerMessageParameters#taskManagerIdParameter is not declared final
[
https://issues.apache.org/jira/browse/FLINK-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-16939:
---
Labels: pull-request-available (was: )
> TaskManagerMessageParameters#taskManagerIdParameter is not declared final
> -
>
> Key: FLINK-16939
> URL: https://issues.apache.org/jira/browse/FLINK-16939
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST
>Affects Versions: 1.11.0
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> The field {{TaskManagerMessageParameters#taskManagerIdParameter}} is not
> declared final but it should be.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #11612: [FLINK-16939][rest] Declare taskManagerIdParameter final
flinkbot commented on issue #11612: [FLINK-16939][rest] Declare taskManagerIdParameter final URL: https://github.com/apache/flink/pull/11612#issuecomment-607699087 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit c1d8f0621261d54ddf07e55f10f6d6d2acf5c099 (Thu Apr 02 08:26:59 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16938) SqlTimestamp has lag when convert long to Timestamp
[
https://issues.apache.org/jira/browse/FLINK-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073504#comment-17073504
]
YufeiLiu commented on FLINK-16938:
--
[~jark] Here is the test demo. I know use {{Instant}} maybe would avoid the
issue, but we have lots of legacy task and udf is defined as Timestamp type.
{code:java}
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.createLocalEnvironment(1);
final EnvironmentSettings settings =
EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
final StreamTableEnvironment tEnv =
StreamTableEnvironment.create(env, settings);
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
DataStreamSource source = env.fromElements(
Row.of("first", System.currentTimeMillis()),
Row.of("second", System.currentTimeMillis())
);
long outOfOrderness = 1000;
DataStream input =
source.assignTimestampsAndWatermarks(new
BoundedOutOfOrdernessTimestampExtractor(Time.milliseconds(outOfOrderness))
{
@Override
public long extractTimestamp(Row element) {
return ((long) element.getField(1));
}
}).returns(Types.ROW_NAMED(new String[]{"name", "ts"},
Types.STRING, Types.LONG));
Table sourceTable = tEnv.fromDataStream(input, "name,
ts.rowtime");
tEnv.createTemporaryView("source_table", sourceTable);
tEnv
.connect(new FileSystem().path("/tmp/sink.csv"))
.withSchema(
new Schema()
.field("name", DataTypes.STRING())
.field("ts", DataTypes.TIMESTAMP(3))
)
.withFormat(new Csv())
.inAppendMode()
.createTemporaryTable("sink_table");
tEnv.insertInto("sink_table", sourceTable);
tEnv.execute("");
{code}
> SqlTimestamp has lag when convert long to Timestamp
> ---
>
> Key: FLINK-16938
> URL: https://issues.apache.org/jira/browse/FLINK-16938
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.10.0
>Reporter: YufeiLiu
>Priority: Major
>
> When I set rowtime attribute by using expression 'column.rowtime' , and
> result type is sql.Timestamp, the result will have lag which is equals with
> default timezone offset.
> {code:java}
> tEnv.fromDataStream(stream, "user_action_time.rowtime, user_name, data");
> {code}
> I look into the conversion logic, the field was go through 'long ->
> SqlTimestamp -> Timestamp' conversion.
> {code:java}
> long from = System.currentTimeMillis();
> long to = SqlTimestamp
> .fromEpochMillis(from)
> .toTimestamp()
> .getTime();
> {code}
> The result is {{from!=to}}. In {{SqlTimestamp.toTimestamp()}} using
> {{Timestamp.valueOf(LocalDateTime dateTime)}} which is contain timezone
> infomation, will casue time lag.
> From Timestamp to Timestamp not have this issue, but convert Datastream to
> Table is use StreamRecord.timestamp as rowtime field.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhangjun888 commented on a change in pull request #11457: [FLINK-14255][WIP][table] Introduce FileSystemStreamingSinkV2
zhangjun888 commented on a change in pull request #11457:
[FLINK-14255][WIP][table] Introduce FileSystemStreamingSinkV2
URL: https://github.com/apache/flink/pull/11457#discussion_r402139052
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/HiveTableSink.java
##
@@ -82,59 +95,134 @@ public HiveTableSink(JobConf jobConf, ObjectPath
tablePath, CatalogTable table)
tableSchema =
TableSchemaUtils.getPhysicalSchema(table.getSchema());
}
+ private boolean isStreaming() {
+ return Boolean.parseBoolean(catalogTable.getProperties()
+
.get(FileSystemValidator.CONNECTOR_SINK_STREAMING_ENABLE));
+ }
+
@Override
- public OutputFormat getOutputFormat() {
+ public final DataStreamSink consumeDataStream(DataStream dataStream) {
+ int parallelism = dataStream.getParallelism();
+ String sinkParallelism = catalogTable.getProperties()
+
.get(ConnectorDescriptorValidator.CONNECTOR_SINK_PARALLELISM);
+ if (sinkParallelism != null) {
+ parallelism = Integer.parseInt(sinkParallelism);
+ }
+
String[] partitionColumns =
getPartitionFieldNames().toArray(new String[0]);
String dbName = tablePath.getDatabaseName();
String tableName = tablePath.getObjectName();
try (HiveMetastoreClientWrapper client =
HiveMetastoreClientFactory.create(
new HiveConf(jobConf, HiveConf.class),
hiveVersion)) {
Table table = client.getTable(dbName, tableName);
StorageDescriptor sd = table.getSd();
+ HiveTableMetaStoreFactory msFactory = new
HiveTableMetaStoreFactory(
+ jobConf, hiveVersion, dbName,
tableName);
+
+ String defaultPartName = jobConf.get(
+
HiveConf.ConfVars.DEFAULTPARTITIONNAME.varname,
+
HiveConf.ConfVars.DEFAULTPARTITIONNAME.defaultStrVal);
+
+ if (isStreaming()) {
+ if (overwrite) {
+ throw new
IllegalStateException("Streaming mode not support overwrite.");
+ }
+
+ if (!isParquet(sd)) {
Review comment:
only parquet ? no ORC ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-16411) Maven central connection timeouts on Azure Pipelines
[
https://issues.apache.org/jira/browse/FLINK-16411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073505#comment-17073505
]
Robert Metzger commented on FLINK-16411:
I have set up a Maven Cache, my results:
- Out of 8 builds (each build has 11 jobs), 0 jobs failed
- I analyzed the time of one build, it is 22% faster (93 minutes)
I have two concerns against introducing a Maven Cache:
1. we introduce a single point of failure
2. Because the cache is only accessible for the CI machines, we will need to
add a few conditions to our build logic (disable the cache on ali machines +
configure a different maven repository). This will make the build config files
more complicated.
[~chesnay] what do you think? I tend to give the maven cache a try, because of
the much improved build times.
> Maven central connection timeouts on Azure Pipelines
>
>
> Key: FLINK-16411
> URL: https://issues.apache.org/jira/browse/FLINK-16411
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Some test stages invoke maven again, where additional dependencies are
> downloaded, sometimes failing the build.
> This ticket is about using the Google mirror wherever possible.
> Examples of failing tests:
> -
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5882&view=logs&j=636f54dd-dda5-5b4b-f495-2d92ec493b6c&t=6c30efdf-a92a-5da3-9a6a-004c8552b2df
> A failure looks like this:
> {code}
> [ERROR] Failed to execute goal on project flink-hadoop-fs: Could not resolve
> dependencies for project org.apache.flink:flink-hadoop-fs:jar:1.11-SNAPSHOT:
> Could not transfer artifact
> org.apache.flink:flink-shaded-hadoop-2:jar:2.8.3-10.0 from/to central
> (https://repo.maven.apache.org/maven2): GET request of:
> org/apache/flink/flink-shaded-hadoop-2/2.8.3-10.0/flink-shaded-hadoop-2-2.8.3-10.0.jar
> from central failed: Connection reset -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] KarmaGYZ commented on issue #11577: [FLINK-16874] Respect the dynamic options when calculating memory options in taskmanager.sh
KarmaGYZ commented on issue #11577: [FLINK-16874] Respect the dynamic options when calculating memory options in taskmanager.sh URL: https://github.com/apache/flink/pull/11577#issuecomment-607701909 cc @azagrebin This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-16940) Avoid creating currentRegion HashSet with manually set initialCapacity
Gary Yao created FLINK-16940:
Summary: Avoid creating currentRegion HashSet with manually set
initialCapacity
Key: FLINK-16940
URL: https://issues.apache.org/jira/browse/FLINK-16940
Project: Flink
Issue Type: Bug
Components: Runtime / REST
Affects Versions: 1.11.0
Reporter: Gary Yao
Assignee: Gary Yao
Fix For: 1.11.0
The {{currentRegion}} HashSet in {{PipelinedRegionComputeUtil}} is created with
an initialCapacity of 1. This is wrong because when we add the first element,
the sets capacity will be already increased. From the style guidelines:
{quote}
Set the initial capacity for a collection only if there is a good proven reason
for that, otherwise do not clutter the code. In case of Maps it can be even
deluding because the Map’s load factor effectively reduces the capacity.
{quote}
https://flink.apache.org/contributing/code-style-and-quality-java.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] GJL opened a new pull request #11613: [FLINK-16940][runtime] Create currentRegion HashSet with default capacity
GJL opened a new pull request #11613: [FLINK-16940][runtime] Create currentRegion HashSet with default capacity URL: https://github.com/apache/flink/pull/11613 ## What is the purpose of the change *Create currentRegion HashSet with default capacity* ## Brief change log - *See commit* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16938) SqlTimestamp has lag when convert long to Timestamp
[
https://issues.apache.org/jira/browse/FLINK-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073511#comment-17073511
]
YufeiLiu commented on FLINK-16938:
--
org.apache.flink.table.api.ValidationException: Field types of query result and
registered TableSink default_catalog.default_database.sink_table do not match.
Query schema: [name: STRING, ts: TIMESTAMP(3) *ROWTIME*]
Sink schema: [name: STRING, ts: TIMESTAMP(3) WITH LOCAL TIME ZONE]
And seems rowtime attribute DataType default is {{TIMESTAMP(3) *ROWTIME*}}
> SqlTimestamp has lag when convert long to Timestamp
> ---
>
> Key: FLINK-16938
> URL: https://issues.apache.org/jira/browse/FLINK-16938
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.10.0
>Reporter: YufeiLiu
>Priority: Major
>
> When I set rowtime attribute by using expression 'column.rowtime' , and
> result type is sql.Timestamp, the result will have lag which is equals with
> default timezone offset.
> {code:java}
> tEnv.fromDataStream(stream, "user_action_time.rowtime, user_name, data");
> {code}
> I look into the conversion logic, the field was go through 'long ->
> SqlTimestamp -> Timestamp' conversion.
> {code:java}
> long from = System.currentTimeMillis();
> long to = SqlTimestamp
> .fromEpochMillis(from)
> .toTimestamp()
> .getTime();
> {code}
> The result is {{from!=to}}. In {{SqlTimestamp.toTimestamp()}} using
> {{Timestamp.valueOf(LocalDateTime dateTime)}} which is contain timezone
> infomation, will casue time lag.
> From Timestamp to Timestamp not have this issue, but convert Datastream to
> Table is use StreamRecord.timestamp as rowtime field.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16940) Avoid creating currentRegion HashSet with manually set initialCapacity
[
https://issues.apache.org/jira/browse/FLINK-16940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-16940:
---
Labels: pull-request-available (was: )
> Avoid creating currentRegion HashSet with manually set initialCapacity
> --
>
> Key: FLINK-16940
> URL: https://issues.apache.org/jira/browse/FLINK-16940
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST
>Affects Versions: 1.11.0
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> The {{currentRegion}} HashSet in {{PipelinedRegionComputeUtil}} is created
> with an initialCapacity of 1. This is wrong because when we add the first
> element, the sets capacity will be already increased. From the style
> guidelines:
> {quote}
> Set the initial capacity for a collection only if there is a good proven
> reason for that, otherwise do not clutter the code. In case of Maps it can be
> even deluding because the Map’s load factor effectively reduces the capacity.
> {quote}
> https://flink.apache.org/contributing/code-style-and-quality-java.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16931) Large _metadata file lead to JobManager not responding when restart
[
https://issues.apache.org/jira/browse/FLINK-16931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-16931:
--
Priority: Critical (was: Major)
> Large _metadata file lead to JobManager not responding when restart
> ---
>
> Key: FLINK-16931
> URL: https://issues.apache.org/jira/browse/FLINK-16931
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Reporter: Lu Niu
>Priority: Critical
>
> When _metadata file is big, JobManager could never recover from checkpoint.
> It fall into a loop that fetch checkpoint -> JM timeout -> restart. Here is
> related log:
> {code:java}
> 2020-04-01 17:08:25,689 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Recovering checkpoints from ZooKeeper.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found
> 3 checkpoints in ZooKeeper.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to fetch 3 checkpoints from storage.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to retrieve checkpoint 50.
> 2020-04-01 17:08:48,589 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to retrieve checkpoint 51.
> 2020-04-01 17:09:12,775 INFO org.apache.flink.yarn.YarnResourceManager - The
> heartbeat of JobManager with id 02500708baf0bb976891c391afd3d7d5 timed out.
> {code}
> Digging into the code, looks like ExecutionGraph::restart runs in JobMaster
> main thread and finally calls
> ZooKeeperCompletedCheckpointStore::retrieveCompletedCheckpoint which download
> file form DFS. The main thread is basically blocked for a while because of
> this. One possible solution is to making the downloading part async. More
> things might need to consider as the original change tries to make it
> single-threaded. [https://github.com/apache/flink/pull/7568]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16931) Large _metadata file lead to JobManager not responding when restart
[
https://issues.apache.org/jira/browse/FLINK-16931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-16931:
--
Fix Version/s: 1.11.0
> Large _metadata file lead to JobManager not responding when restart
> ---
>
> Key: FLINK-16931
> URL: https://issues.apache.org/jira/browse/FLINK-16931
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.9.2, 1.10.0, 1.11.0
>Reporter: Lu Niu
>Priority: Critical
> Fix For: 1.11.0
>
>
> When _metadata file is big, JobManager could never recover from checkpoint.
> It fall into a loop that fetch checkpoint -> JM timeout -> restart. Here is
> related log:
> {code:java}
> 2020-04-01 17:08:25,689 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Recovering checkpoints from ZooKeeper.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found
> 3 checkpoints in ZooKeeper.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to fetch 3 checkpoints from storage.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to retrieve checkpoint 50.
> 2020-04-01 17:08:48,589 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to retrieve checkpoint 51.
> 2020-04-01 17:09:12,775 INFO org.apache.flink.yarn.YarnResourceManager - The
> heartbeat of JobManager with id 02500708baf0bb976891c391afd3d7d5 timed out.
> {code}
> Digging into the code, looks like ExecutionGraph::restart runs in JobMaster
> main thread and finally calls
> ZooKeeperCompletedCheckpointStore::retrieveCompletedCheckpoint which download
> file form DFS. The main thread is basically blocked for a while because of
> this. One possible solution is to making the downloading part async. More
> things might need to consider as the original change tries to make it
> single-threaded. [https://github.com/apache/flink/pull/7568]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16931) Large _metadata file lead to JobManager not responding when restart
[
https://issues.apache.org/jira/browse/FLINK-16931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-16931:
--
Affects Version/s: 1.11.0
1.9.2
1.10.0
> Large _metadata file lead to JobManager not responding when restart
> ---
>
> Key: FLINK-16931
> URL: https://issues.apache.org/jira/browse/FLINK-16931
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.9.2, 1.10.0, 1.11.0
>Reporter: Lu Niu
>Priority: Critical
>
> When _metadata file is big, JobManager could never recover from checkpoint.
> It fall into a loop that fetch checkpoint -> JM timeout -> restart. Here is
> related log:
> {code:java}
> 2020-04-01 17:08:25,689 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Recovering checkpoints from ZooKeeper.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found
> 3 checkpoints in ZooKeeper.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to fetch 3 checkpoints from storage.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to retrieve checkpoint 50.
> 2020-04-01 17:08:48,589 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to retrieve checkpoint 51.
> 2020-04-01 17:09:12,775 INFO org.apache.flink.yarn.YarnResourceManager - The
> heartbeat of JobManager with id 02500708baf0bb976891c391afd3d7d5 timed out.
> {code}
> Digging into the code, looks like ExecutionGraph::restart runs in JobMaster
> main thread and finally calls
> ZooKeeperCompletedCheckpointStore::retrieveCompletedCheckpoint which download
> file form DFS. The main thread is basically blocked for a while because of
> this. One possible solution is to making the downloading part async. More
> things might need to consider as the original change tries to make it
> single-threaded. [https://github.com/apache/flink/pull/7568]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16931) Large _metadata file lead to JobManager not responding when restart
[
https://issues.apache.org/jira/browse/FLINK-16931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073514#comment-17073514
]
Till Rohrmann commented on FLINK-16931:
---
Thanks for reporting this issue [~qqibrow]. You are right that we should not
run blocking operations in the rpc endpoint's main thread. We should fix this.
> Large _metadata file lead to JobManager not responding when restart
> ---
>
> Key: FLINK-16931
> URL: https://issues.apache.org/jira/browse/FLINK-16931
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Reporter: Lu Niu
>Priority: Critical
>
> When _metadata file is big, JobManager could never recover from checkpoint.
> It fall into a loop that fetch checkpoint -> JM timeout -> restart. Here is
> related log:
> {code:java}
> 2020-04-01 17:08:25,689 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Recovering checkpoints from ZooKeeper.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found
> 3 checkpoints in ZooKeeper.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to fetch 3 checkpoints from storage.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to retrieve checkpoint 50.
> 2020-04-01 17:08:48,589 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to retrieve checkpoint 51.
> 2020-04-01 17:09:12,775 INFO org.apache.flink.yarn.YarnResourceManager - The
> heartbeat of JobManager with id 02500708baf0bb976891c391afd3d7d5 timed out.
> {code}
> Digging into the code, looks like ExecutionGraph::restart runs in JobMaster
> main thread and finally calls
> ZooKeeperCompletedCheckpointStore::retrieveCompletedCheckpoint which download
> file form DFS. The main thread is basically blocked for a while because of
> this. One possible solution is to making the downloading part async. More
> things might need to consider as the original change tries to make it
> single-threaded. [https://github.com/apache/flink/pull/7568]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11276: [FLINK-16029][table-planner-blink] Remove register source and sink in test cases of blink planner
flinkbot edited a comment on issue #11276: [FLINK-16029][table-planner-blink] Remove register source and sink in test cases of blink planner URL: https://github.com/apache/flink/pull/11276#issuecomment-593234780 ## CI report: * 1eee3c75e8c4619e988a2886c725c277b723e71c Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/157700077) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6961) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhangjun888 commented on a change in pull request #11457: [FLINK-14255][WIP][table] Introduce FileSystemStreamingSinkV2
zhangjun888 commented on a change in pull request #11457:
[FLINK-14255][WIP][table] Introduce FileSystemStreamingSinkV2
URL: https://github.com/apache/flink/pull/11457#discussion_r402144332
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/HiveTablePartition.java
##
@@ -56,4 +56,14 @@ public StorageDescriptor getStorageDescriptor() {
public Map getPartitionSpec() {
return partitionSpec;
}
+
+ public static boolean isParquet(StorageDescriptor sd) {
+ return sd.getSerdeInfo().getSerializationLib()
+ .toLowerCase().contains("parquet");
+ }
+
+ public static boolean isOrc(StorageDescriptor sd) {
+ return sd.getSerdeInfo().getSerializationLib()
+ .toLowerCase().contains("parquet");
Review comment:
maybe a word mistake? it is orc instead of parquet
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
