[jira] [Comment Edited] (FLINK-17961) Create an Elasticsearch source
[ https://issues.apache.org/jira/browse/FLINK-17961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119635#comment-17119635 ] Etienne Chauchot edited comment on FLINK-17961 at 6/4/20, 1:34 PM: --- Thanks Aljoscha for commenting. ES has data streams features but only for time series data; the aim of this source is to read all kind of data. Apart from data streams, it behaves like a database. You read the content of an index (similar to a table) corresponding to the given query (similar to SQL). So, regarding streaming changes, if there are changes between 2 read requests, at the second the whole index (containing the change) will be read another time. Regarding failover: I guess exactly once semantics cannot be guaranteed only at least once. Indeed there is no ack mechanism on already read data. Under those circumstances, I guess an ES source cannot get into ES. So what should a user do to read from ES? Should he send ES requests manually from a Map ? was (Author: echauchot): Thanks Aljoscha for commenting. ES has data streams features but only for time series data; the aim of this source is to read all kind of data. Apart from data streams it behaves like a database. You read the content of an index (similar to a table) corresponding to the given query (similar to SQL). So, regarding streaming changes, if there are changes between 2 read requests, at the second the whole index (containing the change) will be read another time. Regarding failover: I guess exactly once semantics cannot be guaranteed only at least once. Indeed there is no ack mechanism on already read data. Under those circumstances, I guess an ES source cannot get into ES. So what should a user do to read from ES? Should he send ES requests manually from a Map ? > Create an Elasticsearch source > -- > > Key: FLINK-17961 > URL: https://issues.apache.org/jira/browse/FLINK-17961 > Project: Flink > Issue Type: New Feature > Components: Connectors / ElasticSearch >Reporter: Etienne Chauchot >Priority: Minor > > There is only an Elasticsearch sink available. There are opensource github > repos such as [this > one|[https://github.com/mnubo/flink-elasticsearch-source-connector]]. Also > the apache bahir project does not provide an Elasticsearch source connector > for flink either. IMHO I think the project would benefit from having an > bundled source connector for ES alongside with the available sink connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] rkhachatryan commented on a change in pull request #12478: [FLINK-17869][task][checkpointing] Fix race condition when caling ChannelStateWriter.abort
rkhachatryan commented on a change in pull request #12478:
URL: https://github.com/apache/flink/pull/12478#discussion_r435266519
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/SubtaskCheckpointCoordinatorImpl.java
##
@@ -158,6 +159,16 @@ private ChannelStateWriter openChannelStateWriter() {
@Override
public void abortCheckpointOnBarrier(long checkpointId, Throwable
cause, OperatorChain operatorChain) throws IOException {
LOG.debug("Aborting checkpoint via cancel-barrier {} for task
{}", checkpointId, taskName);
+ lastCheckpointId = Math.max(lastCheckpointId, checkpointId);
+ Iterator iterator = abortedCheckpointIds.iterator();
+ while (iterator.hasNext()) {
+ long next = iterator.next();
+ if (next < lastCheckpointId) {
+ iterator.remove();
+ } else {
+ break;
+ }
+ }
Review comment:
This is just an optimization.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #12478: [FLINK-17869][task][checkpointing] Fix race condition when caling ChannelStateWriter.abort
rkhachatryan commented on a change in pull request #12478:
URL: https://github.com/apache/flink/pull/12478#discussion_r435272756
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/SubtaskCheckpointCoordinatorImpl.java
##
@@ -281,6 +281,8 @@ public void notifyCheckpointAborted(long checkpointId,
OperatorChain opera
}
}
+ channelStateWriter.abort(checkpointId, new
CancellationException("checkpoint aborted via notification"), false);
Review comment:
This is just an optimization (possible now with `abort` and
`cleanup=false`).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #12478: [FLINK-17869][task][checkpointing] Fix race condition when caling ChannelStateWriter.abort
rkhachatryan commented on a change in pull request #12478:
URL: https://github.com/apache/flink/pull/12478#discussion_r435273439
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/SubtaskCheckpointCoordinatorImpl.java
##
@@ -281,6 +281,8 @@ public void notifyCheckpointAborted(long checkpointId,
OperatorChain opera
}
}
+ channelStateWriter.abort(checkpointId, new
CancellationException("checkpoint aborted via notification"), false);
Review comment:
This is just an optimization (possible now with `abort` and
`cleanup=false`).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12352: [FLINK-17717][sql-parser] Throws for DDL create temporary system func…
flinkbot edited a comment on pull request #12352: URL: https://github.com/apache/flink/pull/12352#issuecomment-634517986 ## CI report: * 7dc817125d0684987814d45192eab87a41ebc5b5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2716) * 6d12fc558cc4c24403fc39884d4d7fea41dfffcf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2735) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12486: [hotfix][docs] Modify the lateness definition according to the descri…
flinkbot edited a comment on pull request #12486: URL: https://github.com/apache/flink/pull/12486#issuecomment-638848235 ## CI report: * 21bab4df76688fccc953d68398fc2cad54b0a26a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2738) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12436: [FLINK-17847][table sql / planner] ArrayIndexOutOfBoundsException happens in StreamExecCalc operator
flinkbot edited a comment on pull request #12436: URL: https://github.com/apache/flink/pull/12436#issuecomment-637399261 ## CI report: * 4d1f156fd758a44f4c394a478b6f37bc2de61c86 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2711) * 63831f0ba3dc2a6ec31acc7fb1cfd1b51288945b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2721) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12479: [FLINK-17260] Fix StreamingKafkaITCase instabilities
flinkbot edited a comment on pull request #12479: URL: https://github.com/apache/flink/pull/12479#issuecomment-638721700 ## CI report: * 1bef9a6921ec7d86739b7d6821b6f23f8602bd1b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2704) * 2c7be46e975aaaef574cc3620169a3b3178b310e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12484: [FLINK-18127][tests] Streamline manual execution of Java E2E tests
flinkbot edited a comment on pull request #12484: URL: https://github.com/apache/flink/pull/12484#issuecomment-638836075 ## CI report: * 30b3c3dfdc501e2f8c54877d6a9c950e688dfd09 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2734) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12456: [FLINK-17113][sql-cli] Refactor view support in SQL Client
flinkbot edited a comment on pull request #12456: URL: https://github.com/apache/flink/pull/12456#issuecomment-638032908 ## CI report: * 12e3932ed8d74a59c0d4c28441b336284a2dffeb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2713) * 028c7804132598c982cdc82506781df5a4fb753a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2736) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12485: [FLINK-18130][hive][fs-connector] File name conflict for different jobs in filesystem/hive sink
flinkbot edited a comment on pull request #12485: URL: https://github.com/apache/flink/pull/12485#issuecomment-638848123 ## CI report: * 7fe7332d4767bbfb996e86f1b59d99c8c4713192 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2737) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17961) Create an Elasticsearch source
[ https://issues.apache.org/jira/browse/FLINK-17961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125944#comment-17125944 ] Etienne Chauchot commented on FLINK-17961: -- I just commented in the existing design discussion thread: [https://lists.apache.org/thread.html/r33cd907cecfd125ab1164ddc8a4d8e45d6bd3afd332fbb034881b1ff%40%3Cdev.flink.apache.org%3E] > Create an Elasticsearch source > -- > > Key: FLINK-17961 > URL: https://issues.apache.org/jira/browse/FLINK-17961 > Project: Flink > Issue Type: New Feature > Components: Connectors / ElasticSearch >Reporter: Etienne Chauchot >Priority: Minor > > There is only an Elasticsearch sink available. There are opensource github > repos such as [this > one|[https://github.com/mnubo/flink-elasticsearch-source-connector]]. Also > the apache bahir project does not provide an Elasticsearch source connector > for flink either. IMHO I think the project would benefit from having an > bundled source connector for ES alongside with the available sink connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16713) Support source mode of elasticsearch connector
[ https://issues.apache.org/jira/browse/FLINK-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125945#comment-17125945 ] Etienne Chauchot commented on FLINK-16713: -- I just commented in the existing design discussion thread: [https://lists.apache.org/thread.html/r33cd907cecfd125ab1164ddc8a4d8e45d6bd3afd332fbb034881b1ff%40%3Cdev.flink.apache.org%3E] > Support source mode of elasticsearch connector > -- > > Key: FLINK-16713 > URL: https://issues.apache.org/jira/browse/FLINK-16713 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.10.0 >Reporter: jackray wang >Priority: Major > > [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#elasticsearch-connector] > For append-only queries, the connector can also operate in [append > mode|https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#update-modes] > for exchanging only INSERT messages with the external system. If no key is > defined by the query, a key is automatically generated by Elasticsearch. > I want to know ,why the connector of flink with ES just support sink but > doesn't support source .Which version could add this feature to ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18070) Time attribute been materialized after sub graph optimize
[
https://issues.apache.org/jira/browse/FLINK-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125952#comment-17125952
]
YufeiLiu commented on FLINK-18070:
--
[~libenchao]
process time have already been materialized at begining of convert.
{code:java}
val converter = new RelTimeIndicatorConverter(rexBuilder)
val convertedRoot = rootRel.accept(converter)
{code}
> Time attribute been materialized after sub graph optimize
> -
>
> Key: FLINK-18070
> URL: https://issues.apache.org/jira/browse/FLINK-18070
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.10.0
>Reporter: YufeiLiu
>Assignee: YufeiLiu
>Priority: Major
> Fix For: 1.11.0
>
>
> Hi, I want to use window aggregate after create temporary, and has multiple
> sinks. But throw exception:
> {code:java}
> java.lang.AssertionError: type mismatch:
> ref:
> TIME ATTRIBUTE(PROCTIME) NOT NULL
> input:
> TIMESTAMP(3) NOT NULL
> {code}
> I look into the optimizer logic, there is comment at
> {{CommonSubGraphBasedOptimizer}}:
> "1. In general, for multi-sinks users tend to use VIEW which is a natural
> common sub-graph."
> After sub graph optimize, time attribute from source have been convert to
> basic TIMESTAMP type according to {{FlinkRelTimeIndicatorProgram}}. But my
> create view sql is simple query, I think didn't need to materialized time
> attribute in theory.
> Here is my code:
> {code:java}
> // connector.type COLLECTION is for debug use
> tableEnv.sqlUpdate("CREATE TABLE source (\n" +
> "`ts` AS PROCTIME(),\n" +
> "`order_type` INT\n" +
> ") WITH (\n" +
> "'connector.type' = 'COLLECTION',\n" +
> "'format.type' = 'json'\n" +
> ")\n");
> tableEnv.createTemporaryView("source_view", tableEnv.sqlQuery("SELECT * FROM
> source"));
> tableEnv.sqlUpdate("CREATE TABLE sink (\n" +
> "`result` BIGINT\n" +
> ") WITH (\n" +
> "'connector.type' = 'COLLECTION',\n" +
> "'format.type' = 'json'\n" +
> ")\n");
> tableEnv.sqlUpdate("INSERT INTO sink \n" +
> "SELECT\n" +
> "COUNT(1)\n" +
> "FROM\n" +
> "`source_view`\n" +
> "WHERE\n" +
> " `order_type` = 33\n" +
> "GROUP BY\n" +
> "TUMBLE(`ts`, INTERVAL '5' SECOND)\n");
> tableEnv.sqlUpdate("INSERT INTO sink \n" +
> "SELECT\n" +
> "COUNT(1)\n" +
> "FROM\n" +
> "`source_view`\n" +
> "WHERE\n" +
> " `order_type` = 34\n" +
> "GROUP BY\n" +
> "TUMBLE(`ts`, INTERVAL '5' SECOND)\n");
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12479: [FLINK-17260] Fix StreamingKafkaITCase instabilities
flinkbot edited a comment on pull request #12479: URL: https://github.com/apache/flink/pull/12479#issuecomment-638721700 ## CI report: * 1bef9a6921ec7d86739b7d6821b6f23f8602bd1b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2704) * 2c7be46e975aaaef574cc3620169a3b3178b310e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2739) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17829) Add documentation for the new JDBC connector
[ https://issues.apache.org/jira/browse/FLINK-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-17829: --- Labels: pull-request-available (was: ) > Add documentation for the new JDBC connector > > > Key: FLINK-17829 > URL: https://issues.apache.org/jira/browse/FLINK-17829 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC, Documentation >Reporter: Jark Wu >Assignee: Leonard Xu >Priority: Blocker > Labels: pull-request-available > Fix For: 1.11.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] leonardBang opened a new pull request #12487: [FLINK-17829][docs][jdbc] Add documentation for the new JDBC connector.
leonardBang opened a new pull request #12487: URL: https://github.com/apache/flink/pull/12487 ## What is the purpose of the change *This pull request Add documentation for new JDBC connector.* ## Brief change log - Add new JDBC connector documentation which include include "Dependencies", "Example", "Options", "Data Type Mapping" sections. - Fix typo error in old documentation(temporary -> temporal). ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): ( no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12487: [FLINK-17829][docs][jdbc] Add documentation for the new JDBC connector.
flinkbot commented on pull request #12487: URL: https://github.com/apache/flink/pull/12487#issuecomment-638886784 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 91172715268381f9bdaa7b2284fd322a9608a4ca (Thu Jun 04 14:31:34 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] kl0u closed pull request #12455: [FLINK-17935] Move setting yarn.log-config-file to YarnClusterClientFactory
kl0u closed pull request #12455: URL: https://github.com/apache/flink/pull/12455 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18132) Add documentation for the new CSV format
Jark Wu created FLINK-18132: --- Summary: Add documentation for the new CSV format Key: FLINK-18132 URL: https://issues.apache.org/jira/browse/FLINK-18132 Project: Flink Issue Type: Sub-task Components: Documentation, Formats (JSON, Avro, Parquet, ORC, SequenceFile) Reporter: Jark Wu Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18131) Add documentation for the new JSON format
Jark Wu created FLINK-18131: --- Summary: Add documentation for the new JSON format Key: FLINK-18131 URL: https://issues.apache.org/jira/browse/FLINK-18131 Project: Flink Issue Type: Sub-task Components: Documentation, Formats (JSON, Avro, Parquet, ORC, SequenceFile) Reporter: Jark Wu Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18133) Add documentation for the new Avro format
Jark Wu created FLINK-18133: --- Summary: Add documentation for the new Avro format Key: FLINK-18133 URL: https://issues.apache.org/jira/browse/FLINK-18133 Project: Flink Issue Type: Sub-task Components: Documentation, Formats (JSON, Avro, Parquet, ORC, SequenceFile) Reporter: Jark Wu Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17935) Logs could not show up when deploying Flink on Yarn via "--executor"
[
https://issues.apache.org/jira/browse/FLINK-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas updated FLINK-17935:
---
Fix Version/s: 1.12.0
> Logs could not show up when deploying Flink on Yarn via "--executor"
>
>
> Key: FLINK-17935
> URL: https://issues.apache.org/jira/browse/FLINK-17935
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Yang Wang
>Assignee: Kostas Kloudas
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0, 1.12.0
>
>
> {code:java}
> ./bin/flink run -d -p 5 -e yarn-per-job
> examples/streaming/WindowJoin.jar{code}
> When we use the {{-e/--executor}} to specify the deploy target to Yarn
> per-job, the logs could not show up. The root cause is we do not set the
> logging files in {{ExecutorCLI}}. We only do it in the
> {{FlinkYarnSessionCli}}.
> If we use {{-m yarn-cluster}}, everything works well.
>
> Maybe we should move the {{setLogConfigFileInConfig}} to
> {{YarnClusterDescriptor}} to avoid this problem. cc [~kkl0u]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-17935) Logs could not show up when deploying Flink on Yarn via "--executor"
[
https://issues.apache.org/jira/browse/FLINK-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas closed FLINK-17935.
--
Resolution: Fixed
Fixed on master with a674b5e181a2af050aee4c4f23a86a6c61eda94f
and on release-1.11 with d22874ef6322bfb774d60b09c022c3dbbe295d70
> Logs could not show up when deploying Flink on Yarn via "--executor"
>
>
> Key: FLINK-17935
> URL: https://issues.apache.org/jira/browse/FLINK-17935
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Yang Wang
>Assignee: Kostas Kloudas
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> {code:java}
> ./bin/flink run -d -p 5 -e yarn-per-job
> examples/streaming/WindowJoin.jar{code}
> When we use the {{-e/--executor}} to specify the deploy target to Yarn
> per-job, the logs could not show up. The root cause is we do not set the
> logging files in {{ExecutorCLI}}. We only do it in the
> {{FlinkYarnSessionCli}}.
> If we use {{-m yarn-cluster}}, everything works well.
>
> Maybe we should move the {{setLogConfigFileInConfig}} to
> {{YarnClusterDescriptor}} to avoid this problem. cc [~kkl0u]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18134) Add documentation for the Debezium format
Jark Wu created FLINK-18134: --- Summary: Add documentation for the Debezium format Key: FLINK-18134 URL: https://issues.apache.org/jira/browse/FLINK-18134 Project: Flink Issue Type: Sub-task Components: Documentation, Formats (JSON, Avro, Parquet, ORC, SequenceFile) Reporter: Jark Wu Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18135) Add documentation for the Canal format
Jark Wu created FLINK-18135: --- Summary: Add documentation for the Canal format Key: FLINK-18135 URL: https://issues.apache.org/jira/browse/FLINK-18135 Project: Flink Issue Type: Sub-task Components: Documentation, Formats (JSON, Avro, Parquet, ORC, SequenceFile) Reporter: Jark Wu Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15331) [umberlla] Create a table from a changelog
[ https://issues.apache.org/jira/browse/FLINK-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-15331: Summary: [umberlla] Create a table from a changelog (was: Create a table from a changelog) > [umberlla] Create a table from a changelog > -- > > Key: FLINK-15331 > URL: https://issues.apache.org/jira/browse/FLINK-15331 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Timo Walther >Assignee: Jark Wu >Priority: Major > Fix For: 1.11.0 > > > This an umbrella issue for the feature of interpreting a changelog as a > table. It can be used to collect initial ideas and use cases until a FLIP is > proposed. > FLIP-105: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15331) [umbrella] Create a table from a changelog
[ https://issues.apache.org/jira/browse/FLINK-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-15331: Summary: [umbrella] Create a table from a changelog (was: [umberlla] Create a table from a changelog) > [umbrella] Create a table from a changelog > -- > > Key: FLINK-15331 > URL: https://issues.apache.org/jira/browse/FLINK-15331 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Timo Walther >Assignee: Jark Wu >Priority: Major > Fix For: 1.11.0 > > > This an umbrella issue for the feature of interpreting a changelog as a > table. It can be used to collect initial ideas and use cases until a FLIP is > proposed. > FLIP-105: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (FLINK-15331) Create a table from a changelog
[ https://issues.apache.org/jira/browse/FLINK-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reopened FLINK-15331: - > Create a table from a changelog > --- > > Key: FLINK-15331 > URL: https://issues.apache.org/jira/browse/FLINK-15331 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Timo Walther >Assignee: Jark Wu >Priority: Major > Fix For: 1.11.0 > > > This an umbrella issue for the feature of interpreting a changelog as a > table. It can be used to collect initial ideas and use cases until a FLIP is > proposed. > FLIP-105: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong commented on pull request #12436: [FLINK-17847][table sql / planner] ArrayIndexOutOfBoundsException happens in StreamExecCalc operator
wuchong commented on pull request #12436: URL: https://github.com/apache/flink/pull/12436#issuecomment-638894806 Build is passed in my repo: https://dev.azure.com/imjark/Flink/_build/results?buildId=136&view=results Merging... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong closed pull request #12436: [FLINK-17847][table sql / planner] ArrayIndexOutOfBoundsException happens in StreamExecCalc operator
wuchong closed pull request #12436: URL: https://github.com/apache/flink/pull/12436 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17847) ArrayIndexOutOfBoundsException happens when codegen StreamExec operator
[
https://issues.apache.org/jira/browse/FLINK-17847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-17847:
Fix Version/s: (was: 1.12.0)
> ArrayIndexOutOfBoundsException happens when codegen StreamExec operator
> ---
>
> Key: FLINK-17847
> URL: https://issues.apache.org/jira/browse/FLINK-17847
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> user case:
> {code:java}
> //source table
> create table json_table(
> w_es BIGINT,
> w_type STRING,
> w_isDdl BOOLEAN,
> w_data ARRAY account_pay_fee DOUBLE>>,
> w_ts TIMESTAMP(3),
> w_table STRING) WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = '0.10',
> 'connector.topic' = 'json-test2',
> 'connector.properties.zookeeper.connect' = 'localhost:2181',
> 'connector.properties.bootstrap.servers' = 'localhost:9092',
> 'connector.properties.group.id' = 'test-jdbc',
> 'connector.startup-mode' = 'earliest-offset',
> 'format.type' = 'json',
> 'format.derive-schema' = 'true'
> )
> // real data:
> {"w_es":1589870637000,"w_type":"INSERT","w_isDdl":false,"w_data":[{"pay_info":"channelId=82&onlineFee=89.0&outTradeNo=0&payId=0&payType=02&rechargeId=4&totalFee=89.0&tradeStatus=success&userId=32590183789575&sign=00","online_fee":"89.0","sign":"00","account_pay_fee":"0.0"}],"w_ts":"2020-05-20T13:58:37.131Z","w_table":"111"}
> //query
> select w_ts, 'test' as city1_id, w_data[0].pay_info AS cate3_id,
> w_data as pay_order_id from json_table
> {code}
> ~exception:~
> {code:java}
> //
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1427848Caused by:
> java.lang.ArrayIndexOutOfBoundsException: 1427848 at
> org.apache.flink.table.runtime.util.SegmentsUtil.getByteMultiSegments(SegmentsUtil.java:598)
> at
> org.apache.flink.table.runtime.util.SegmentsUtil.getByte(SegmentsUtil.java:590)
> at
> org.apache.flink.table.runtime.util.SegmentsUtil.bitGet(SegmentsUtil.java:534)
> at
> org.apache.flink.table.dataformat.BinaryArray.isNullAt(BinaryArray.java:117)
> at StreamExecCalc$10.processElement(Unknown Source)
> {code}
>
> Looks like in the codegen StreamExecCalc$10 operator some operation visit a
> '-1' index which should be wrong, this bug exits both in 1.10 and 1.11
>
> {code:java}
> public class StreamExecCalc$10 extends
> org.apache.flink.table.runtime.operators.AbstractProcessStreamOperator
> implements
> org.apache.flink.streaming.api.operators.OneInputStreamOperator {
> private final Object[] references;
> private final org.apache.flink.table.dataformat.BinaryString str$3 =
> org.apache.flink.table.dataformat.BinaryString.fromString("test");
> private transient
> org.apache.flink.table.runtime.typeutils.BaseArraySerializer typeSerializer$5;
> final org.apache.flink.table.dataformat.BoxedWrapperRow out = new
> org.apache.flink.table.dataformat.BoxedWrapperRow(4);
> private final
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord outElement = new
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord(null);
> public StreamExecCalc$10(
> Object[] references,
> org.apache.flink.streaming.runtime.tasks.StreamTask task,
> org.apache.flink.streaming.api.graph.StreamConfig config,
> org.apache.flink.streaming.api.operators.Output output) throws
> Exception {
> this.references = references;
> typeSerializer$5 =
> (((org.apache.flink.table.runtime.typeutils.BaseArraySerializer)
> references[0]));
> this.setup(task, config, output);
> }
> @Override
> public void open() throws Exception {
> super.open();
> }
> @Override
> public void
> processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> element) throws Exception {
> org.apache.flink.table.dataformat.BaseRow in1 =
> (org.apache.flink.table.dataformat.BaseRow) element.getValue();
> org.apache.flink.table.dataformat.SqlTimestamp field$2;
> boolean isNull$2;
> org.apache.flink.table.dataformat.BaseArray field$4;
> boolean isNull$4;
> org.apache.flink.table.dataformat.BaseArray field$6;
> org.apache.flink.table.dataformat.BinaryString field$8;
> boolean isNull$8;
> org.apache.flink.table.dataformat.BinaryString result$9;
> boolean isNull$9;
> isNull$2 = in1.isNullAt(4);
> field$2 = null;
> if (!isNull$2) {
> field$2 = in1.getTimestamp(4, 3);
> }
> isNull$4 = in1.isNullAt(3);
> field$4 = null;
> if (!isNull$4) {
> f
[GitHub] [flink] dianfu opened a new pull request #12488: [FLINK-18126][python] Correct the exception handling of the Python CompletableFuture
dianfu opened a new pull request #12488: URL: https://github.com/apache/flink/pull/12488 ## What is the purpose of the change *The implementation of method `exception` and `set_exception` are not correct. The purpose of the Python CompletableFuture is to hole the results from the Java CompletableFuture. We should expose the exception inside the Java CompletableFuture to users instead of allow Python users set the exception.* ## Brief change log - *Throw exception for the method set_result and set_exception* - *Return the exception reported in the Java CompletableFuture for the method exception* ## Verifying this change Verified mannully. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-17847) ArrayIndexOutOfBoundsException happens when codegen StreamExec operator
[
https://issues.apache.org/jira/browse/FLINK-17847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu closed FLINK-17847.
---
Resolution: Fixed
- master (1.12.0): 3ee4c1f0a094d7562f4188b9fc93a2313e872cde
- 1.11.0: 79b64fc42598eb799e7e4da28fc2b4ab6e446506
> ArrayIndexOutOfBoundsException happens when codegen StreamExec operator
> ---
>
> Key: FLINK-17847
> URL: https://issues.apache.org/jira/browse/FLINK-17847
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> user case:
> {code:java}
> //source table
> create table json_table(
> w_es BIGINT,
> w_type STRING,
> w_isDdl BOOLEAN,
> w_data ARRAY account_pay_fee DOUBLE>>,
> w_ts TIMESTAMP(3),
> w_table STRING) WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = '0.10',
> 'connector.topic' = 'json-test2',
> 'connector.properties.zookeeper.connect' = 'localhost:2181',
> 'connector.properties.bootstrap.servers' = 'localhost:9092',
> 'connector.properties.group.id' = 'test-jdbc',
> 'connector.startup-mode' = 'earliest-offset',
> 'format.type' = 'json',
> 'format.derive-schema' = 'true'
> )
> // real data:
> {"w_es":1589870637000,"w_type":"INSERT","w_isDdl":false,"w_data":[{"pay_info":"channelId=82&onlineFee=89.0&outTradeNo=0&payId=0&payType=02&rechargeId=4&totalFee=89.0&tradeStatus=success&userId=32590183789575&sign=00","online_fee":"89.0","sign":"00","account_pay_fee":"0.0"}],"w_ts":"2020-05-20T13:58:37.131Z","w_table":"111"}
> //query
> select w_ts, 'test' as city1_id, w_data[0].pay_info AS cate3_id,
> w_data as pay_order_id from json_table
> {code}
> ~exception:~
> {code:java}
> //
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1427848Caused by:
> java.lang.ArrayIndexOutOfBoundsException: 1427848 at
> org.apache.flink.table.runtime.util.SegmentsUtil.getByteMultiSegments(SegmentsUtil.java:598)
> at
> org.apache.flink.table.runtime.util.SegmentsUtil.getByte(SegmentsUtil.java:590)
> at
> org.apache.flink.table.runtime.util.SegmentsUtil.bitGet(SegmentsUtil.java:534)
> at
> org.apache.flink.table.dataformat.BinaryArray.isNullAt(BinaryArray.java:117)
> at StreamExecCalc$10.processElement(Unknown Source)
> {code}
>
> Looks like in the codegen StreamExecCalc$10 operator some operation visit a
> '-1' index which should be wrong, this bug exits both in 1.10 and 1.11
>
> {code:java}
> public class StreamExecCalc$10 extends
> org.apache.flink.table.runtime.operators.AbstractProcessStreamOperator
> implements
> org.apache.flink.streaming.api.operators.OneInputStreamOperator {
> private final Object[] references;
> private final org.apache.flink.table.dataformat.BinaryString str$3 =
> org.apache.flink.table.dataformat.BinaryString.fromString("test");
> private transient
> org.apache.flink.table.runtime.typeutils.BaseArraySerializer typeSerializer$5;
> final org.apache.flink.table.dataformat.BoxedWrapperRow out = new
> org.apache.flink.table.dataformat.BoxedWrapperRow(4);
> private final
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord outElement = new
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord(null);
> public StreamExecCalc$10(
> Object[] references,
> org.apache.flink.streaming.runtime.tasks.StreamTask task,
> org.apache.flink.streaming.api.graph.StreamConfig config,
> org.apache.flink.streaming.api.operators.Output output) throws
> Exception {
> this.references = references;
> typeSerializer$5 =
> (((org.apache.flink.table.runtime.typeutils.BaseArraySerializer)
> references[0]));
> this.setup(task, config, output);
> }
> @Override
> public void open() throws Exception {
> super.open();
> }
> @Override
> public void
> processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> element) throws Exception {
> org.apache.flink.table.dataformat.BaseRow in1 =
> (org.apache.flink.table.dataformat.BaseRow) element.getValue();
> org.apache.flink.table.dataformat.SqlTimestamp field$2;
> boolean isNull$2;
> org.apache.flink.table.dataformat.BaseArray field$4;
> boolean isNull$4;
> org.apache.flink.table.dataformat.BaseArray field$6;
> org.apache.flink.table.dataformat.BinaryString field$8;
> boolean isNull$8;
> org.apache.flink.table.dataformat.BinaryString result$9;
> boolean isNull$9;
> isNull$2 = in1.isNullAt(4);
> field$2 = null;
> if (!isNull$2) {
> field$2 = in1.getTimestamp(4, 3);
> }
> i
[jira] [Updated] (FLINK-18126) Correct the exception handling of the Python CompletableFuture
[ https://issues.apache.org/jira/browse/FLINK-18126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18126: --- Labels: pull-request-available (was: ) > Correct the exception handling of the Python CompletableFuture > -- > > Key: FLINK-18126 > URL: https://issues.apache.org/jira/browse/FLINK-18126 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.11.0 >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > The implementation of method `exception` and `set_exception` are not correct. > The purpose of the Python CompletableFuture is to hole the results from the > Java CompletableFuture. We should expose the exception inside the Java > CompletableFuture to users instead of allow Python users set the exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-10195) RabbitMQ Source With Checkpointing Doesn't Backpressure Correctly
[ https://issues.apache.org/jira/browse/FLINK-10195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125362#comment-17125362 ] Austin Cawley-Edwards edited comment on FLINK-10195 at 6/4/20, 2:50 PM: Yes, that is correct, though I think it is still an improvement on what's here and allows the users to tune their job according to their needs, but I don't think the performance issue that you cite can be fixed with what Rabbit provides. EDIT: we might be able to handle this by updating prefetch counts dynamically if the buffer has space and there are many unacked messages waiting to be acked on checkpoint, though I think that might be too much for an initial implementation. If the user has high volume queues and needs checkpointing for EXACTLY_ONCE/ AT_LEAST_ONCE guarantees, they'll need to tune their buffer length and checkpointing interval. This could also be an opt-in/ opt-out change if there are cases that need it disabled, and we should definitely update the docs[1]. Our company still actively uses Rabbit – I'm happy to build off your PR and test it out in our jobs. [1]: [https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/rabbitmq.html#rabbitmq-source] was (Author: austince): Yes, that is correct, though I think it is still an improvement on what's here and allows the users to tune their job according to their needs, but I don't think the performance issue that you cite can be fixed with what Rabbit provides. EDIT: we might be able to handle this by updating prefetch counts dynamically if the buffer has space and there are many unacked messages waiting to be acked on checkpoint, though I think that might be too much for an initial implementation. If the user has high volume queues and needs checkpointing for EXACTLY_ONCE/ AT_LEAST_ONCE guarantees, they'll need to tune their buffer length and checkpointing interval. This could also be an opt-in/ opt-out change if there are cases that need it disabled, and we should definitely update the docs. Our company still actively uses Rabbit – I'm happy to build off your PR and test it out in our jobs. [1]: [https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/rabbitmq.html#rabbitmq-source] > RabbitMQ Source With Checkpointing Doesn't Backpressure Correctly > - > > Key: FLINK-10195 > URL: https://issues.apache.org/jira/browse/FLINK-10195 > Project: Flink > Issue Type: Bug > Components: Connectors/ RabbitMQ >Affects Versions: 1.4.0, 1.5.0, 1.5.1, 1.6.0 >Reporter: Luka Jurukovski >Assignee: Luka Jurukovski >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The connection between the RabbitMQ server and the client does not > appropriately back pressure when auto acking is disabled. This becomes very > problematic when a downstream process throttles the data processing to slower > then RabbitMQ sends the data to the client. > The difference in records ends up being stored in the flink's heap space, > which grows indefinitely (or technically to "Integer Max" Deliveries). > Looking at RabbitMQ's metrics the number of unacked messages looks like > steadily rising saw tooth shape. > Upon further invesitgation it looks like this is due to how the > QueueingConsumer works, messages are added to the BlockingQueue faster then > they are being removed and processed, resulting in the previously described > behavior. > This may be intended behavior, however this isn't explicitly obvious in the > documentation or any of the examples I have seen. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12488: [FLINK-18126][python] Correct the exception handling of the Python CompletableFuture
flinkbot commented on pull request #12488: URL: https://github.com/apache/flink/pull/12488#issuecomment-638899477 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 4f60d1df216f704acb71f6546d295ed5dc640e1f (Thu Jun 04 14:50:36 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-14971) Make all the non-IO operations in CheckpointCoordinator single-threaded
[
https://issues.apache.org/jira/browse/FLINK-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125989#comment-17125989
]
Stephan Ewen commented on FLINK-14971:
--
In the second step I was referring to "committing asynchronously", because that
is also a blocking operation (write to ZooKeeper).
However, committing asynchronously is complex because there is a time when the
scheduler can ask for a checkpoint but it is not clear which the latest one is
(due do async committing).
Concerning the cleanup problem: This should happen asychronously (not block JM
and not block committing) but it needs to backpressure new checkpoint creation.
It sounds to me like the best way would be to take this into account when
triggering checkpoints, as an additional condition.
For example, under default settings, a new checkpoint can only be triggered if
no other periodic checkpoint is in progress, and there is no more than one
checkpoint pending under cleanup.
> Make all the non-IO operations in CheckpointCoordinator single-threaded
> ---
>
> Key: FLINK-14971
> URL: https://issues.apache.org/jira/browse/FLINK-14971
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently the ACK and declined message handling are executed in IO thread.
> This is the only rest part that non-IO operations are executed in IO thread.
> It blocks introducing main thread executor for {{CheckpointCoordinator}}. It
> would be resolved in this task.
> After resolving the ACK and declined message issue, the main thread executor
> would be introduced into {{CheckpointCoordinator}} to instead of timer
> thread. However the timer thread would be kept (maybe for a while
> temporarily) to schedule periodic triggering, since FLINK-13848 is not
> accepted yet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14971) Make all the non-IO operations in CheckpointCoordinator single-threaded
[
https://issues.apache.org/jira/browse/FLINK-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125989#comment-17125989
]
Stephan Ewen edited comment on FLINK-14971 at 6/4/20, 3:02 PM:
---
In the second step I was referring to "committing asynchronously", because that
is also a blocking operation (write to ZooKeeper).
However, committing asynchronously is complex because there is a time when the
scheduler can ask for a checkpoint but it is not clear which the latest one is
(due do async committing). I think we can only approach this once we have
support for async restore in the scheduler.
Concerning the cleanup problem: This should happen asychronously (not block JM
and not block committing) but it needs to backpressure new checkpoint creation.
It sounds to me like the best way would be to take this into account when
triggering checkpoints, as an additional condition.
For example, under default settings, a new checkpoint can only be triggered if
no other periodic checkpoint is in progress, and there is no more than one
checkpoint pending under cleanup.
was (Author: stephanewen):
In the second step I was referring to "committing asynchronously", because that
is also a blocking operation (write to ZooKeeper).
However, committing asynchronously is complex because there is a time when the
scheduler can ask for a checkpoint but it is not clear which the latest one is
(due do async committing).
Concerning the cleanup problem: This should happen asychronously (not block JM
and not block committing) but it needs to backpressure new checkpoint creation.
It sounds to me like the best way would be to take this into account when
triggering checkpoints, as an additional condition.
For example, under default settings, a new checkpoint can only be triggered if
no other periodic checkpoint is in progress, and there is no more than one
checkpoint pending under cleanup.
> Make all the non-IO operations in CheckpointCoordinator single-threaded
> ---
>
> Key: FLINK-14971
> URL: https://issues.apache.org/jira/browse/FLINK-14971
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently the ACK and declined message handling are executed in IO thread.
> This is the only rest part that non-IO operations are executed in IO thread.
> It blocks introducing main thread executor for {{CheckpointCoordinator}}. It
> would be resolved in this task.
> After resolving the ACK and declined message issue, the main thread executor
> would be introduced into {{CheckpointCoordinator}} to instead of timer
> thread. However the timer thread would be kept (maybe for a while
> temporarily) to schedule periodic triggering, since FLINK-13848 is not
> accepted yet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12435: [FLINK-18059] [sql-client] Fix create/drop catalog statement can not be executed in sql client
flinkbot edited a comment on pull request #12435: URL: https://github.com/apache/flink/pull/12435#issuecomment-637372360 ## CI report: * 84524977ee6597667093943fce7c1730ab934c35 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2732) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12475: [FLINK-18076][table sql / client] Sql client uses wrong class loader when parsing queries
flinkbot edited a comment on pull request #12475: URL: https://github.com/apache/flink/pull/12475#issuecomment-638593958 ## CI report: * 41240cd524255ffc7cf63d2bc2ee9ff38f3abcbd UNKNOWN * 49c19b93edb74ef702c97a4742f4936145913f35 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2717) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12487: [FLINK-17829][docs][jdbc] Add documentation for the new JDBC connector.
flinkbot commented on pull request #12487: URL: https://github.com/apache/flink/pull/12487#issuecomment-638911231 ## CI report: * 91172715268381f9bdaa7b2284fd322a9608a4ca UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12456: [FLINK-17113][sql-cli] Refactor view support in SQL Client
flinkbot edited a comment on pull request #12456: URL: https://github.com/apache/flink/pull/12456#issuecomment-638032908 ## CI report: * 028c7804132598c982cdc82506781df5a4fb753a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2736) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12485: [FLINK-18130][hive][fs-connector] File name conflict for different jobs in filesystem/hive sink
flinkbot edited a comment on pull request #12485: URL: https://github.com/apache/flink/pull/12485#issuecomment-638848123 ## CI report: * 7fe7332d4767bbfb996e86f1b59d99c8c4713192 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2737) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12488: [FLINK-18126][python] Correct the exception handling of the Python CompletableFuture
flinkbot commented on pull request #12488: URL: https://github.com/apache/flink/pull/12488#issuecomment-638911342 ## CI report: * 4f60d1df216f704acb71f6546d295ed5dc640e1f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lsyldliu commented on a change in pull request #12303: [FLINK-17625] [table] Fix ArrayIndexOutOfBoundsException in AppendOnlyTopNFunction
lsyldliu commented on a change in pull request #12303:
URL: https://github.com/apache/flink/pull/12303#discussion_r435333138
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/rank/AppendOnlyTopNFunction.java
##
@@ -210,14 +210,19 @@ private void processElementWithoutRowNumber(RowData
input, Collector ou
RowData lastKey = lastEntry.getKey();
List lastList = (List)
lastEntry.getValue();
// remove last one
- RowData lastElement = lastList.remove(lastList.size() -
1);
- if (lastList.isEmpty()) {
+ int size = lastList.size();
+ RowData lastElement = null;
+ if (size > 0) {
+ lastElement = lastList.get(size - 1);
Review comment:
@wuchong I agree with you, but I think we also should subtract
buffer.currentTopNum = buffer.currentTopNum - 1 after remove lastElement, So
is it appropriate to add a method in TopNBuffer such as `public void
minusTopNum(int delta) {
currentTopNum -= delta;
}`,
we should call this method behind `dataState.put(lastKey, new
ArrayList<>(lastList))`, what do you think it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-17732) RescalingITCase.testSavepointRescalingOutPartitionedOperatorStateList
[
https://issues.apache.org/jira/browse/FLINK-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125998#comment-17125998
]
Robert Metzger commented on FLINK-17732:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2709&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=f508e270-48d6-5f1e-3138-42a17e0714f0
{code}
2020-06-04T11:24:09.0403191Z [INFO] Running
org.apache.flink.test.checkpointing.UnalignedCheckpointCompatibilityITCase
2020-06-04T11:24:14.5454441Z [ERROR] Tests run: 24, Failures: 0, Errors: 1,
Skipped: 0, Time elapsed: 289.948 s <<< FAILURE! - in
org.apache.flink.test.checkpointing.RescalingITCase
2020-06-04T11:24:14.5456157Z [ERROR]
testSavepointRescalingInBroadcastOperatorState[backend =
rocksdb](org.apache.flink.test.checkpointing.RescalingITCase) Time elapsed:
172.589 s <<< ERROR!
2020-06-04T11:24:14.5457382Z java.util.concurrent.ExecutionException:
org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not
complete the operation. Number of retries has been exhausted.
2020-06-04T11:24:14.5458881Zat
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
2020-06-04T11:24:14.5459746Zat
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
2020-06-04T11:24:14.5460639Zat
org.apache.flink.test.checkpointing.RescalingITCase.testSavepointRescalingPartitionedOperatorState(RescalingITCase.java:470)
2020-06-04T11:24:14.5461672Zat
org.apache.flink.test.checkpointing.RescalingITCase.testSavepointRescalingInBroadcastOperatorState(RescalingITCase.java:409)
2020-06-04T11:24:14.5462426Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-06-04T11:24:14.5463058Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-06-04T11:24:14.5463839Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-06-04T11:24:14.5464724Zat
java.lang.reflect.Method.invoke(Method.java:498)
2020-06-04T11:24:14.5465464Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-06-04T11:24:14.5466319Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-06-04T11:24:14.5467115Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-06-04T11:24:14.5467884Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-06-04T11:24:14.5468854Zat
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2020-06-04T11:24:14.5469522Zat
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2020-06-04T11:24:14.5470302Zat
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-06-04T11:24:14.5470901Zat
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2020-06-04T11:24:14.5471604Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2020-06-04T11:24:14.5472335Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2020-06-04T11:24:14.5473034Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-06-04T11:24:14.5473677Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-06-04T11:24:14.5474377Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-06-04T11:24:14.5475179Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-06-04T11:24:14.5475815Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-06-04T11:24:14.5476440Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-06-04T11:24:14.5476991Zat
org.junit.runners.Suite.runChild(Suite.java:128)
2020-06-04T11:24:14.5477669Zat
org.junit.runners.Suite.runChild(Suite.java:27)
2020-06-04T11:24:14.5478277Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-06-04T11:24:14.5478924Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-06-04T11:24:14.5479730Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-06-04T11:24:14.5480375Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-06-04T11:24:14.5481030Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-06-04T11:24:14.5481769Zat
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2020-06-04T11:24:14.5482511Zat
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-06-04T11:24:14.5483177Zat
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-06-04T11:24:14.5483750Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-06-04T11:24:14.5484424Zat
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
2020-06-04T
[jira] [Updated] (FLINK-17732) RescalingITCase.testSavepointRescalingOutPartitionedOperatorStateList
[
https://issues.apache.org/jira/browse/FLINK-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-17732:
---
Affects Version/s: 1.12.0
> RescalingITCase.testSavepointRescalingOutPartitionedOperatorStateList
> -
>
> Key: FLINK-17732
> URL: https://issues.apache.org/jira/browse/FLINK-17732
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
>
> CI:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1413&view=logs&j=16ccbdb7-2a3e-53da-36eb-fb718edc424a&t=cf61ce33-6fba-5fbe-2c0c-e41c4013e891
> {code}
> 2020-05-15T12:09:16.9432669Z [ERROR]
> testSavepointRescalingOutPartitionedOperatorStateList[backend =
> filesystem](org.apache.flink.test.checkpointing.RescalingITCase) Time
> elapsed: 180.189 s <<< ERROR!
> 2020-05-15T12:09:16.9433577Z java.util.concurrent.TimeoutException
> 2020-05-15T12:09:16.9434238Z at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
> 2020-05-15T12:09:16.9435119Z at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> 2020-05-15T12:09:16.9436062Z at
> org.apache.flink.test.checkpointing.RescalingITCase.testSavepointRescalingPartitionedOperatorState(RescalingITCase.java:473)
> 2020-05-15T12:09:16.9437313Z at
> org.apache.flink.test.checkpointing.RescalingITCase.testSavepointRescalingOutPartitionedOperatorStateList(RescalingITCase.java:427)
> 2020-05-15T12:09:16.9438112Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-05-15T12:09:16.9438858Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-05-15T12:09:16.9439611Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-05-15T12:09:16.9440367Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-05-15T12:09:16.9441502Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-05-15T12:09:16.9442020Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-05-15T12:09:16.9442535Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-05-15T12:09:16.9442984Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-05-15T12:09:16.9443557Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2020-05-15T12:09:16.9444014Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-05-15T12:09:16.9444379Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-05-15T12:09:16.9444901Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-05-15T12:09:16.9445314Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-05-15T12:09:16.9445812Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-05-15T12:09:16.9446289Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-05-15T12:09:16.9446660Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-05-15T12:09:16.9447097Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-05-15T12:09:16.9447478Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-05-15T12:09:16.9447915Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-05-15T12:09:16.9448284Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-05-15T12:09:16.9448689Z at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2020-05-15T12:09:16.9449062Z at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2020-05-15T12:09:16.9449405Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-05-15T12:09:16.9450063Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-05-15T12:09:16.9450620Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-05-15T12:09:16.9451313Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-05-15T12:09:16.9451741Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-05-15T12:09:16.9452218Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-05-15T12:09:16.9452694Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-05-15T12:09:16.9453065Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-05-15T12:09:16.9453468Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-05-15T12:09:16.9454027Z at
> org.juni
[jira] [Updated] (FLINK-17732) RescalingITCase.testSavepointRescalingOutPartitionedOperatorStateList
[
https://issues.apache.org/jira/browse/FLINK-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-17732:
---
Labels: test-stability (was: )
> RescalingITCase.testSavepointRescalingOutPartitionedOperatorStateList
> -
>
> Key: FLINK-17732
> URL: https://issues.apache.org/jira/browse/FLINK-17732
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
>
> CI:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1413&view=logs&j=16ccbdb7-2a3e-53da-36eb-fb718edc424a&t=cf61ce33-6fba-5fbe-2c0c-e41c4013e891
> {code}
> 2020-05-15T12:09:16.9432669Z [ERROR]
> testSavepointRescalingOutPartitionedOperatorStateList[backend =
> filesystem](org.apache.flink.test.checkpointing.RescalingITCase) Time
> elapsed: 180.189 s <<< ERROR!
> 2020-05-15T12:09:16.9433577Z java.util.concurrent.TimeoutException
> 2020-05-15T12:09:16.9434238Z at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
> 2020-05-15T12:09:16.9435119Z at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> 2020-05-15T12:09:16.9436062Z at
> org.apache.flink.test.checkpointing.RescalingITCase.testSavepointRescalingPartitionedOperatorState(RescalingITCase.java:473)
> 2020-05-15T12:09:16.9437313Z at
> org.apache.flink.test.checkpointing.RescalingITCase.testSavepointRescalingOutPartitionedOperatorStateList(RescalingITCase.java:427)
> 2020-05-15T12:09:16.9438112Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-05-15T12:09:16.9438858Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-05-15T12:09:16.9439611Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-05-15T12:09:16.9440367Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-05-15T12:09:16.9441502Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-05-15T12:09:16.9442020Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-05-15T12:09:16.9442535Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-05-15T12:09:16.9442984Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-05-15T12:09:16.9443557Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2020-05-15T12:09:16.9444014Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-05-15T12:09:16.9444379Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-05-15T12:09:16.9444901Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-05-15T12:09:16.9445314Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-05-15T12:09:16.9445812Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-05-15T12:09:16.9446289Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-05-15T12:09:16.9446660Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-05-15T12:09:16.9447097Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-05-15T12:09:16.9447478Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-05-15T12:09:16.9447915Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-05-15T12:09:16.9448284Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-05-15T12:09:16.9448689Z at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2020-05-15T12:09:16.9449062Z at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2020-05-15T12:09:16.9449405Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-05-15T12:09:16.9450063Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-05-15T12:09:16.9450620Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-05-15T12:09:16.9451313Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-05-15T12:09:16.9451741Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-05-15T12:09:16.9452218Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-05-15T12:09:16.9452694Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-05-15T12:09:16.9453065Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-05-15T12:09:16.9453468Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>
[jira] [Updated] (FLINK-17732) RescalingITCase.testSavepointRescalingOutPartitionedOperatorStateList
[
https://issues.apache.org/jira/browse/FLINK-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-17732:
---
Component/s: Tests
Runtime / Checkpointing
> RescalingITCase.testSavepointRescalingOutPartitionedOperatorStateList
> -
>
> Key: FLINK-17732
> URL: https://issues.apache.org/jira/browse/FLINK-17732
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Task, Tests
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
>
> CI:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1413&view=logs&j=16ccbdb7-2a3e-53da-36eb-fb718edc424a&t=cf61ce33-6fba-5fbe-2c0c-e41c4013e891
> {code}
> 2020-05-15T12:09:16.9432669Z [ERROR]
> testSavepointRescalingOutPartitionedOperatorStateList[backend =
> filesystem](org.apache.flink.test.checkpointing.RescalingITCase) Time
> elapsed: 180.189 s <<< ERROR!
> 2020-05-15T12:09:16.9433577Z java.util.concurrent.TimeoutException
> 2020-05-15T12:09:16.9434238Z at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
> 2020-05-15T12:09:16.9435119Z at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> 2020-05-15T12:09:16.9436062Z at
> org.apache.flink.test.checkpointing.RescalingITCase.testSavepointRescalingPartitionedOperatorState(RescalingITCase.java:473)
> 2020-05-15T12:09:16.9437313Z at
> org.apache.flink.test.checkpointing.RescalingITCase.testSavepointRescalingOutPartitionedOperatorStateList(RescalingITCase.java:427)
> 2020-05-15T12:09:16.9438112Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-05-15T12:09:16.9438858Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-05-15T12:09:16.9439611Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-05-15T12:09:16.9440367Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-05-15T12:09:16.9441502Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-05-15T12:09:16.9442020Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-05-15T12:09:16.9442535Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-05-15T12:09:16.9442984Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-05-15T12:09:16.9443557Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2020-05-15T12:09:16.9444014Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-05-15T12:09:16.9444379Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-05-15T12:09:16.9444901Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-05-15T12:09:16.9445314Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-05-15T12:09:16.9445812Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-05-15T12:09:16.9446289Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-05-15T12:09:16.9446660Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-05-15T12:09:16.9447097Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-05-15T12:09:16.9447478Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-05-15T12:09:16.9447915Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-05-15T12:09:16.9448284Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-05-15T12:09:16.9448689Z at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2020-05-15T12:09:16.9449062Z at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2020-05-15T12:09:16.9449405Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-05-15T12:09:16.9450063Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-05-15T12:09:16.9450620Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-05-15T12:09:16.9451313Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-05-15T12:09:16.9451741Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-05-15T12:09:16.9452218Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-05-15T12:09:16.9452694Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-05-15T12:09:16.9453065Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-05-15T12:09:16.9453468Z at
>
[GitHub] [flink] rmetzger merged pull request #12458: [FLINK-17404] Make sure netty 3.10.6 is used in flink-runtime
rmetzger merged pull request #12458: URL: https://github.com/apache/flink/pull/12458 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-17404) Running HA per-job cluster (rocks, non-incremental) gets stuck killing a non-existing pid in Hadoop 3 build profile
[
https://issues.apache.org/jira/browse/FLINK-17404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger resolved FLINK-17404.
Resolution: Fixed
Merged to master (1.12.0) in
https://github.com/apache/flink/commit/9211cb5eaa98f1c51e1569f92c46f65240b0ecc2
Merged to release-1.11 in
https://github.com/apache/flink/commit/83644269acdc577d26ed376acbe8068af56edbee
> Running HA per-job cluster (rocks, non-incremental) gets stuck killing a
> non-existing pid in Hadoop 3 build profile
> ---
>
> Key: FLINK-17404
> URL: https://issues.apache.org/jira/browse/FLINK-17404
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Test Infrastructure, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.11.0
>
> Attachments: 255
>
>
> CI log: https://api.travis-ci.org/v3/job/678609505/log.txt
> {code}
> Waiting for text Completed checkpoint [1-9]* for job
> to appear 2 of times in logs...
> grep:
> /home/travis/build/apache/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/log/*standalonejob-2*.log:
> No such file or directory
> grep:
> /home/travis/build/apache/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/log/*standalonejob-2*.log:
> No such file or directory
> Starting standalonejob daemon on host
> travis-job-e606668f-b674-49c0-8590-e3508e22b99d.
> grep:
> /home/travis/build/apache/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/log/*standalonejob-2*.log:
> No such file or directory
> grep:
> /home/travis/build/apache/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/log/*standalonejob-2*.log:
> No such file or directory
> Killed TM @ 18864
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or
> kill -l [sigspec]
> Killed TM @
> No output has been received in the last 10m0s, this potentially indicates a
> stalled build or something wrong with the build itself.
> Check the details on how to adjust your build configuration on:
> https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received
> The build has been terminated
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rmetzger commented on pull request #12458: [FLINK-17404] Make sure netty 3.10.6 is used in flink-runtime
rmetzger commented on pull request #12458: URL: https://github.com/apache/flink/pull/12458#issuecomment-638918828 Thanks a lot for the review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-17404) Running HA per-job cluster (rocks, non-incremental) gets stuck killing a non-existing pid in Hadoop 3 build profile
[
https://issues.apache.org/jira/browse/FLINK-17404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger closed FLINK-17404.
--
> Running HA per-job cluster (rocks, non-incremental) gets stuck killing a
> non-existing pid in Hadoop 3 build profile
> ---
>
> Key: FLINK-17404
> URL: https://issues.apache.org/jira/browse/FLINK-17404
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Test Infrastructure, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.11.0
>
> Attachments: 255
>
>
> CI log: https://api.travis-ci.org/v3/job/678609505/log.txt
> {code}
> Waiting for text Completed checkpoint [1-9]* for job
> to appear 2 of times in logs...
> grep:
> /home/travis/build/apache/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/log/*standalonejob-2*.log:
> No such file or directory
> grep:
> /home/travis/build/apache/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/log/*standalonejob-2*.log:
> No such file or directory
> Starting standalonejob daemon on host
> travis-job-e606668f-b674-49c0-8590-e3508e22b99d.
> grep:
> /home/travis/build/apache/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/log/*standalonejob-2*.log:
> No such file or directory
> grep:
> /home/travis/build/apache/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/log/*standalonejob-2*.log:
> No such file or directory
> Killed TM @ 18864
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or
> kill -l [sigspec]
> Killed TM @
> No output has been received in the last 10m0s, this potentially indicates a
> stalled build or something wrong with the build itself.
> Check the details on how to adjust your build configuration on:
> https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received
> The build has been terminated
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong commented on a change in pull request #12303: [FLINK-17625] [table] Fix ArrayIndexOutOfBoundsException in AppendOnlyTopNFunction
wuchong commented on a change in pull request #12303:
URL: https://github.com/apache/flink/pull/12303#discussion_r435340328
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/rank/AppendOnlyTopNFunction.java
##
@@ -210,14 +210,19 @@ private void processElementWithoutRowNumber(RowData
input, Collector ou
RowData lastKey = lastEntry.getKey();
List lastList = (List)
lastEntry.getValue();
// remove last one
- RowData lastElement = lastList.remove(lastList.size() -
1);
- if (lastList.isEmpty()) {
+ int size = lastList.size();
+ RowData lastElement = null;
+ if (size > 0) {
+ lastElement = lastList.get(size - 1);
Review comment:
Good point. I just find that maybe we can use `TopNBuffer#removeLast`
directly here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rmetzger merged pull request #12467: [FLINK-16572] Clean up PubSub connector e2e test
rmetzger merged pull request #12467: URL: https://github.com/apache/flink/pull/12467 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #12467: [FLINK-16572] Clean up PubSub connector e2e test
rmetzger commented on pull request #12467: URL: https://github.com/apache/flink/pull/12467#issuecomment-638919331 Thank for your the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16572) CheckPubSubEmulatorTest is flaky on Azure
[ https://issues.apache.org/jira/browse/FLINK-16572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126003#comment-17126003 ] Robert Metzger commented on FLINK-16572: Added more debugging in https://github.com/apache/flink/commit/52861e30b65d6b50f24028e31a972d68a9a9f9f8 > CheckPubSubEmulatorTest is flaky on Azure > - > > Key: FLINK-16572 > URL: https://issues.apache.org/jira/browse/FLINK-16572 > Project: Flink > Issue Type: Bug > Components: Connectors / Google Cloud PubSub, Tests >Affects Versions: 1.11.0 >Reporter: Aljoscha Krettek >Assignee: Richard Deurwaarder >Priority: Critical > Labels: pull-request-available, test-stability > Fix For: 1.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Log: > https://dev.azure.com/aljoschakrettek/Flink/_build/results?buildId=56&view=logs&j=1f3ed471-1849-5d3c-a34c-19792af4ad16&t=ce095137-3e3b-5f73-4b79-c42d3d5f8283&l=7842 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] walterddr commented on pull request #12462: [FLINK-18045] Fix Kerberos credentials checking
walterddr commented on pull request #12462: URL: https://github.com/apache/flink/pull/12462#issuecomment-638923431 thanks for the quick update. in general the patch looks good to me. I still have only one concern regarding the alternation of 2 public APIs. but IMO `HadoopUtils` are more of an internal runtime util that was shared between different deployment environments. thus I am not sure whether the standard public API alternation rules applies here. I would wait for @tillrohrmann for a final review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12487: [FLINK-17829][docs][jdbc] Add documentation for the new JDBC connector.
flinkbot edited a comment on pull request #12487: URL: https://github.com/apache/flink/pull/12487#issuecomment-638911231 ## CI report: * 91172715268381f9bdaa7b2284fd322a9608a4ca Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2744) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12488: [FLINK-18126][python] Correct the exception handling of the Python CompletableFuture
flinkbot edited a comment on pull request #12488: URL: https://github.com/apache/flink/pull/12488#issuecomment-638911342 ## CI report: * 4f60d1df216f704acb71f6546d295ed5dc640e1f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2745) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12278: [FLINK-17019][runtime] Fulfill slot requests in request order
azagrebin commented on a change in pull request #12278: URL: https://github.com/apache/flink/pull/12278#discussion_r435362084 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolImpl.java ## @@ -648,26 +648,8 @@ boolean offerSlot( slotOffer.getResourceProfile(), taskManagerGateway); - // check whether we have request waiting for this slot - PendingRequest pendingRequest = pendingRequests.removeKeyB(allocationID); Review comment: There is this `UnfulfillableSlotRequestException` which is still a fail fast route if RM finds that a certain request profile cannot be fulfilled at all with any existing slot and cannot be allocated. It is relevant for both batch and streaming and bulk as I see. I do not know the whole background of this. At first glance, it seems to be a complication but it might be necessary. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12278: [FLINK-17019][runtime] Fulfill slot requests in request order
azagrebin commented on a change in pull request #12278: URL: https://github.com/apache/flink/pull/12278#discussion_r435362084 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolImpl.java ## @@ -648,26 +648,8 @@ boolean offerSlot( slotOffer.getResourceProfile(), taskManagerGateway); - // check whether we have request waiting for this slot - PendingRequest pendingRequest = pendingRequests.removeKeyB(allocationID); Review comment: There is this `UnfulfillableSlotRequestException` which is still a fail fast route if RM finds that a certain request profile cannot be fulfilled at all with any existing slot and cannot be allocated. It is relevant for both batch and streaming and bulk as I see. I do not know the whole background of this. At first glance, it seems to be a complication but it is probably necessary to avoid timeout waiting to cancel everything if it is already clear that allocation cannot succeed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12326: [FLINK-16577] [table-planner-blink] Fix numeric type mismatch error in column interval relmetadata
flinkbot edited a comment on pull request #12326: URL: https://github.com/apache/flink/pull/12326#issuecomment-633786013 ## CI report: * 0b14ce7cf01c6664a3a98c35dd1c494a7ad4fc32 UNKNOWN * 8c198bc1bb0b6be3321a01898e09d7492169b47e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2725) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12278: [FLINK-17019][runtime] Fulfill slot requests in request order
azagrebin commented on a change in pull request #12278: URL: https://github.com/apache/flink/pull/12278#discussion_r435362084 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolImpl.java ## @@ -648,26 +648,8 @@ boolean offerSlot( slotOffer.getResourceProfile(), taskManagerGateway); - // check whether we have request waiting for this slot - PendingRequest pendingRequest = pendingRequests.removeKeyB(allocationID); Review comment: There is this `UnfulfillableSlotRequestException` which is still a fail fast route if RM finds that a certain request profile cannot be fulfilled at all with any existing slot and cannot be allocated. It is relevant for both batch and streaming and bulk as I see. I do not know the whole background of this. At first glance, it seems to be a complication but it is probably necessary to avoid timeout waiting to cancel everything if it is already clear that allocation can never succeed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12488: [FLINK-18126][python] Correct the exception handling of the Python CompletableFuture
flinkbot edited a comment on pull request #12488: URL: https://github.com/apache/flink/pull/12488#issuecomment-638911342 ## CI report: * 4f60d1df216f704acb71f6546d295ed5dc640e1f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2745) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12375: [FLINK-17017][runtime] Implements bulk allocation for physical slots
azagrebin commented on a change in pull request #12375:
URL: https://github.com/apache/flink/pull/12375#discussion_r435366834
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/BulkSlotProviderImpl.java
##
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmaster.SlotInfo;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.util.clock.Clock;
+import org.apache.flink.util.clock.SystemClock;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Default implementation of {@link BulkSlotProvider}.
+ */
+class BulkSlotProviderImpl implements BulkSlotProvider {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(BulkSlotProviderImpl.class);
+
+ private ComponentMainThreadExecutor componentMainThreadExecutor;
+
+ private final SlotSelectionStrategy slotSelectionStrategy;
+
+ private final SlotPool slotPool;
+
+ private final Clock clock;
+
+ private final PhysicalSlotRequestBulkTracker slotRequestBulkTracker;
+
+ BulkSlotProviderImpl(final SlotSelectionStrategy slotSelectionStrategy,
final SlotPool slotPool) {
+ this(slotSelectionStrategy, slotPool,
SystemClock.getInstance());
+ }
+
+ @VisibleForTesting
+ BulkSlotProviderImpl(
+ final SlotSelectionStrategy slotSelectionStrategy,
+ final SlotPool slotPool,
+ final Clock clock) {
+
+ this.slotSelectionStrategy =
checkNotNull(slotSelectionStrategy);
+ this.slotPool = checkNotNull(slotPool);
+ this.clock = checkNotNull(clock);
+
+ this.slotRequestBulkTracker = new
PhysicalSlotRequestBulkTracker(clock);
+
+ this.componentMainThreadExecutor = new
ComponentMainThreadExecutor.DummyComponentMainThreadExecutor(
+ "Scheduler is not initialized with proper main thread
executor. " +
+ "Call to BulkSlotProvider.start(...)
required.");
+ }
+
+ @Override
+ public void start(final ComponentMainThreadExecutor mainThreadExecutor)
{
+ this.componentMainThreadExecutor = mainThreadExecutor;
+ }
+
+ @Override
+ public CompletableFuture>
allocatePhysicalSlots(
+ final Collection
physicalSlotRequests,
+ final Time timeout) {
+
+ componentMainThreadExecutor.assertRunningInMainThread();
+
+ LOG.debug("Received {} slot requests.",
physicalSlotRequests.size());
+
+ final PhysicalSlotRequestBulk slotRequestBulk = new
PhysicalSlotRequestBulk(physicalSlotRequests);
+
+ final List>
resultFutures = new ArrayList<>(physicalSlotRequests.size());
+ for (PhysicalSlotRequest request : physicalSlotRequests) {
+ final CompletableFuture
resultFuture =
+ allocatePhysicalSlot(request,
timeout).thenApply(result -> {
+ slotRequestBulk.markRequestFulfilled(
+ result.getSlotRequestId(),
+
resul
[GitHub] [flink] azagrebin commented on a change in pull request #12278: [FLINK-17019][runtime] Fulfill slot requests in request order
azagrebin commented on a change in pull request #12278: URL: https://github.com/apache/flink/pull/12278#discussion_r435362084 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolImpl.java ## @@ -648,26 +648,8 @@ boolean offerSlot( slotOffer.getResourceProfile(), taskManagerGateway); - // check whether we have request waiting for this slot - PendingRequest pendingRequest = pendingRequests.removeKeyB(allocationID); Review comment: There is this `UnfulfillableSlotRequestException` which is still a fail fast route if RM finds that a certain request profile cannot be fulfilled at all with any existing slot and cannot be allocated. It is relevant for both batch and streaming and bulk as I see. I do not know the whole background of this. At first glance, this looks to me as an optimisation that complicates things a bit at the moment. It is probably necessary to avoid timeout waiting to cancel everything if it is already clear that allocation can never succeed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] leonardBang commented on pull request #12475: [FLINK-18076][table sql / client] Sql client uses wrong class loader when parsing queries
leonardBang commented on pull request #12475: URL: https://github.com/apache/flink/pull/12475#issuecomment-638946255 The PR updated @dawidwys , please help take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-17954) Do not multiplex remote function state into single PersistedTable
[
https://issues.apache.org/jira/browse/FLINK-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai reassigned FLINK-17954:
---
Assignee: Tzu-Li (Gordon) Tai
> Do not multiplex remote function state into single PersistedTable
> -
>
> Key: FLINK-17954
> URL: https://issues.apache.org/jira/browse/FLINK-17954
> Project: Flink
> Issue Type: Task
> Components: Stateful Functions
>Affects Versions: statefun-2.0.1, statefun-2.1.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
>
> We are currently multiplexing multiple remote function's user value states
> into a single {{PersistedTable}}, using the state name as the table key.
> This is not nice since:
> - It does not allow individual states to have different properties, such as
> TTL expiration.
> - We are restricted to only value states for remote functions
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12471: [FLINK-18073][FLINK-18029][avro] Fix AvroRowDataSerializationSchema is not serializable and add IT cases
flinkbot edited a comment on pull request #12471: URL: https://github.com/apache/flink/pull/12471#issuecomment-638362836 ## CI report: * ec7b1367e121a52731e84f0f5b8610b362c4c169 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2730) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12468: [FLINK-18075] Wrap the SerializationSchema in KafkaSerializationSchema in Kafka connector
flinkbot edited a comment on pull request #12468: URL: https://github.com/apache/flink/pull/12468#issuecomment-638243685 ## CI report: * 7bf026a4e23a89df952f6b361ab8fe435c0a612d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2727) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12475: [FLINK-18076][table sql / client] Sql client uses wrong class loader when parsing queries
flinkbot edited a comment on pull request #12475: URL: https://github.com/apache/flink/pull/12475#issuecomment-638593958 ## CI report: * 41240cd524255ffc7cf63d2bc2ee9ff38f3abcbd UNKNOWN * 49c19b93edb74ef702c97a4742f4936145913f35 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2717) * e1ee41cc7063f83dadf380786b60733dec1639ed UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #11541: [FLINK-15416][network] add task manager netty client retry mechenism
HuangZhenQiu commented on pull request #11541: URL: https://github.com/apache/flink/pull/11541#issuecomment-638957276 @pnowojski For your comments. 1) There is no test that is marked to ignore by this PR. Would you please double-check? 2) Rebased master again. 3) Add join logic to wait for all of the threads successfully executed. I will try my best to make the PR needs the requirement so that it will use less time to merge it later. As it is needed for our internal release, hopefully, it can be merged soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18136) Don't start channel state writing for savepoints (RPC)
Roman Khachatryan created FLINK-18136: - Summary: Don't start channel state writing for savepoints (RPC) Key: FLINK-18136 URL: https://issues.apache.org/jira/browse/FLINK-18136 Project: Flink Issue Type: Bug Components: Runtime / Checkpointing, Runtime / Task Affects Versions: 1.11.0 Reporter: Roman Khachatryan Assignee: Roman Khachatryan Fix For: 1.11.0, 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on pull request #12460: [FLINK-18063][checkpointing] Fix the race condition for aborting current checkpoint in CheckpointBarrierUnaligner
zhijiangW commented on pull request #12460: URL: https://github.com/apache/flink/pull/12460#issuecomment-638963043 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan opened a new pull request #12489: [FLINK-18136] Don't start channel state writing for savepoints (RPC)
rkhachatryan opened a new pull request #12489: URL: https://github.com/apache/flink/pull/12489 ## What is the purpose of the change `StreamTask.triggerCheckpoint` calls `channelStateWriter.start` unconditionally. For savepoints and when unaligned mode is disabled this is incorrect. ## Brief change log - add `subtaskCheckpointCoordinator.initCheckpoint` and use it in `StreamTask` - replace `ChannelStateWriter` with `SubtaskCheckpointCoordinator` in `Barrierhandler` and call `initCheckpoint` there too ## Verifying this change - Added `SubtaskCheckpointCoordinatorTest.testInitCheckpoint` (unit test) ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18136) Don't start channel state writing for savepoints (RPC)
[ https://issues.apache.org/jira/browse/FLINK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18136: --- Labels: pull-request-available (was: ) > Don't start channel state writing for savepoints (RPC) > -- > > Key: FLINK-18136 > URL: https://issues.apache.org/jira/browse/FLINK-18136 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing, Runtime / Task >Affects Versions: 1.11.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0, 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12489: [FLINK-18136] Don't start channel state writing for savepoints (RPC)
flinkbot commented on pull request #12489: URL: https://github.com/apache/flink/pull/12489#issuecomment-638974327 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 5136fc6b80380c9ff9be14c67d6a286dbac118a5 (Thu Jun 04 16:44:44 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12375: [FLINK-17017][runtime] Implements bulk allocation for physical slots
flinkbot edited a comment on pull request #12375: URL: https://github.com/apache/flink/pull/12375#issuecomment-635184560 ## CI report: * 35ef82ee0bf1d008719a52e8182f9b63f3eddb11 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2729) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12475: [FLINK-18076][table sql / client] Sql client uses wrong class loader when parsing queries
flinkbot edited a comment on pull request #12475: URL: https://github.com/apache/flink/pull/12475#issuecomment-638593958 ## CI report: * 41240cd524255ffc7cf63d2bc2ee9ff38f3abcbd UNKNOWN * 49c19b93edb74ef702c97a4742f4936145913f35 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2717) * e1ee41cc7063f83dadf380786b60733dec1639ed Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2749) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12482: [FLINK-17831][doc] Add documentation for the new Kafka connector
flinkbot edited a comment on pull request #12482: URL: https://github.com/apache/flink/pull/12482#issuecomment-638825868 ## CI report: * c8c39d8552852d4b7fdbd84501052d08d9b0963d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2731) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #12470: [FLINK-18074][checkpoint] Ensure task could fail when exception thrown out on notified of checkpoint completed/aborted
pnowojski commented on a change in pull request #12470:
URL: https://github.com/apache/flink/pull/12470#discussion_r435404303
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
##
@@ -919,9 +920,33 @@ public final ExecutorService
getAsyncOperationsThreadPool() {
@Override
public Future notifyCheckpointCompleteAsync(long checkpointId) {
- return
mailboxProcessor.getMailboxExecutor(TaskMailbox.MAX_PRIORITY).submit(
- () -> notifyCheckpointComplete(checkpointId),
- "checkpoint %d complete", checkpointId);
+ return notifyCheckpointOperation(
+ () -> notifyCheckpointComplete(checkpointId),
+ String.format("checkpoint %d complete", checkpointId));
+ }
+
+ @Override
+ public Future notifyCheckpointAbortAsync(long checkpointId) {
+ return notifyCheckpointOperation(
+ () ->
subtaskCheckpointCoordinator.notifyCheckpointAborted(checkpointId,
operatorChain, this::isRunning),
+ String.format("checkpoint %d aborted", checkpointId));
+ }
+
+ private Future notifyCheckpointOperation(RunnableWithException
runnable, String description) {
Review comment:
Could you re-use this method in
`org.apache.flink.streaming.runtime.tasks.StreamTask#triggerCheckpointAsync`?
However I see there is a difference in the used mailbox executor (different
priority), so if this would cause too much extra complexity I'm fine with
leaving it as it is.
##
File path:
flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskTest.java
##
@@ -969,6 +970,34 @@ public void testNotifyCheckpointOnClosedOperator() throws
Throwable {
assertEquals(true, operator.closed.get());
}
+ @Test
+ public void testFailToConfirmCheckpointCompleted() throws Exception {
+ testFailToConfirmCheckpointMessage(streamTask ->
streamTask.notifyCheckpointCompleteAsync(1L));
+ }
+
+ @Test
+ public void testFailToConfirmCheckpointAborted() throws Exception {
+ testFailToConfirmCheckpointMessage(streamTask ->
streamTask.notifyCheckpointAbortAsync(1L));
+ }
+
+ private void testFailToConfirmCheckpointMessage(Consumer> consumer) throws Exception {
+ FailOnNotifyCheckpointOperator operator = new
FailOnNotifyCheckpointOperator<>();
+ MultipleInputStreamTaskTestHarnessBuilder builder =
+ new
MultipleInputStreamTaskTestHarnessBuilder<>(OneInputStreamTask::new,
BasicTypeInfo.INT_TYPE_INFO)
+ .addInput(BasicTypeInfo.INT_TYPE_INFO);
+ StreamTaskMailboxTestHarness harness = builder
+ .setupOutputForSingletonOperatorChain(operator)
+ .build();
+
+ try {
+ consumer.accept(harness.streamTask);
+ harness.streamTask.runMailboxStep();
+ fail();
+ } catch (ExpectedTestException expected) {
+ // expected exception
Review comment:
I presume that this test was failing on master branch (without your fix)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12352: [FLINK-17717][sql-parser] Throws for DDL create temporary system func…
flinkbot edited a comment on pull request #12352: URL: https://github.com/apache/flink/pull/12352#issuecomment-634517986 ## CI report: * 6d12fc558cc4c24403fc39884d4d7fea41dfffcf Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2735) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12452: [FLINK-18056][fs-connector] Removing the remaining in-progress file on initialization for hadoop path-based writer
flinkbot edited a comment on pull request #12452: URL: https://github.com/apache/flink/pull/12452#issuecomment-637922636 ## CI report: * 7cb10bd1269179bbe66f5b840c0f0d2494c66a55 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2598) * fef42fb8497d9822a12ab18114ecb0ea4f36509f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12489: [FLINK-18136] Don't start channel state writing for savepoints (RPC)
flinkbot commented on pull request #12489: URL: https://github.com/apache/flink/pull/12489#issuecomment-638985663 ## CI report: * 5136fc6b80380c9ff9be14c67d6a286dbac118a5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12483: [FLINK-15339][table][docs] Correct the terminology of "Time-windowed Join" to "Interval Join" in Table API & SQL
flinkbot edited a comment on pull request #12483: URL: https://github.com/apache/flink/pull/12483#issuecomment-638835966 ## CI report: * d8400d687e17309f9c69c296fdbf16a1323a07f4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2733) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18038) StateBackendLoader logs application-defined state before it is fully configured
[
https://issues.apache.org/jira/browse/FLINK-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126082#comment-17126082
]
Steve Bairos commented on FLINK-18038:
--
That's fair. What if we modify the "Configuring application-defined state
backend with job/cluster config" log line to be something like
{code:java}
log.info("Using job/cluster config to configure application-defined state
backend: {}", fromApplication); {code}
So that it's clear that this is from before the state backend was configured.
Then we could still move the "Using application-defined state backend: {}" log
line to the end of the if block that it's in so that it shows the configured
version as the version that's actually being used.
> StateBackendLoader logs application-defined state before it is fully
> configured
> ---
>
> Key: FLINK-18038
> URL: https://issues.apache.org/jira/browse/FLINK-18038
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.1
>Reporter: Steve Bairos
>Priority: Trivial
>
> In the
> [StateBackendLoader|[https://github.com/apache/flink/blob/bb46756b84940a6134910e74406bfaff4f2f37e9/flink-runtime/src/main/java/org/apache/flink/runtime/state/StateBackendLoader.java#L201]],
> there's this log line:
> {code:java}
> logger.info("Using application-defined state backend: {}", fromApplication);
> {code}
> It seems like this is inaccurate though because immediately after logging
> this, if fromApplication is a ConfigurableStateBackend, we call the
> .configure() function and it is replaced by a newly configured StateBackend.
> To me, it seems like it would be better if we logged the state backend after
> it was fully configured. In the current setup, we get confusing logs like
> this:
> {code:java}
> 2020-05-29 21:39:44,387 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask - Using
> application-defined state backend:
> RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints:
> 's3://pinterest-montreal/checkpoints/xenon-dev-001-20191210/Xenon/BasicJavaStream',
> savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1),
> localRocksDbDirectories=null, enableIncrementalCheckpointing=UNDEFINED,
> numberOfTransferingThreads=-1}2020-05-29 21:39:44,387 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask - Configuring
> application-defined state backend with job/cluster config{code}
> Which makes it ambiguous whether or not settings in our flink-conf.yaml like
> "state.backend.incremental: true" are being applied properly or not.
>
> I can make a diff for the change if there aren't any objections
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tillrohrmann commented on a change in pull request #12446: [FLINK-16225] Implement user class loading exception handler
tillrohrmann commented on a change in pull request #12446:
URL: https://github.com/apache/flink/pull/12446#discussion_r435420178
##
File path:
flink-core/src/main/java/org/apache/flink/util/ClassLoaderWithErrorHandler.java
##
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.util;
+
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.function.Consumer;
+
+/**
+ * This class loader accepts a custom handler if an exception occurs in {@link
#loadClass(String, boolean)}.
+ */
+public abstract class ClassLoaderWithErrorHandler extends URLClassLoader {
+ public static final Consumer EMPTY_EXCEPTION_HANDLER =
classLoadingException -> {};
+
+ private final Consumer classLoadingExceptionHandler;
+
+ protected ClassLoaderWithErrorHandler(URL[] urls, ClassLoader parent) {
+ this(urls, parent, EMPTY_EXCEPTION_HANDLER);
+ }
+
+ protected ClassLoaderWithErrorHandler(
+ URL[] urls,
+ ClassLoader parent,
+ Consumer classLoadingExceptionHandler) {
+ super(urls, parent);
+ this.classLoadingExceptionHandler =
classLoadingExceptionHandler;
+ }
+
+ @SuppressWarnings("FinalMethod")
Review comment:
Why do we need this suppression here? Is it because we make the
protected method `final`?
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobManagerSharedServices.java
##
@@ -140,7 +141,8 @@ public static JobManagerSharedServices fromConfiguration(
blobServer,
BlobLibraryCacheManager.defaultClassLoaderFactory(
FlinkUserCodeClassLoaders.ResolveOrder.fromString(classLoaderResolveOrder),
- alwaysParentFirstLoaderPatterns));
+ alwaysParentFirstLoaderPatterns,
+
ClassLoaderWithErrorHandler.EMPTY_EXCEPTION_HANDLER));
Review comment:
Shouldn't we also fail if we encounter a metaspace OOM on the
`JobManager` side?
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobManagerSharedServices.java
##
@@ -140,7 +141,8 @@ public static JobManagerSharedServices fromConfiguration(
blobServer,
BlobLibraryCacheManager.defaultClassLoaderFactory(
FlinkUserCodeClassLoaders.ResolveOrder.fromString(classLoaderResolveOrder),
- alwaysParentFirstLoaderPatterns));
+ alwaysParentFirstLoaderPatterns,
+
ClassLoaderWithErrorHandler.EMPTY_EXCEPTION_HANDLER));
Review comment:
Why are we using an `EMPTY_EXCEPTION_HANDLER` here?
##
File path:
flink-core/src/test/java/org/apache/flink/util/ClassLoaderWithErrorHandlerTest.java
##
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.util;
+
+import org.junit.Test;
+
+import java.io.IOException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.hamcrest.CoreMatchers.is;
+import static o
[jira] [Updated] (FLINK-18137) JobMasterTriggerSavepointITCase.testStopJobAfterSavepoint fails with AskTimeoutException
[
https://issues.apache.org/jira/browse/FLINK-18137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-18137:
---
Labels: test-stability (was: )
> JobMasterTriggerSavepointITCase.testStopJobAfterSavepoint fails with
> AskTimeoutException
>
>
> Key: FLINK-18137
> URL: https://issues.apache.org/jira/browse/FLINK-18137
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Runtime / Task
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.11.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2747&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=45cc9205-bdb7-5b54-63cd-89fdc0983323
> {code}
> 2020-06-04T16:17:20.4404189Z [ERROR] Tests run: 4, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 14.352 s <<< FAILURE! - in
> org.apache.flink.runtime.jobmaster.JobMasterTriggerSavepointITCase
> 2020-06-04T16:17:20.4405548Z [ERROR]
> testStopJobAfterSavepoint(org.apache.flink.runtime.jobmaster.JobMasterTriggerSavepointITCase)
> Time elapsed: 10.058 s <<< ERROR!
> 2020-06-04T16:17:20.4407342Z java.util.concurrent.ExecutionException:
> java.util.concurrent.TimeoutException: Invocation of public default
> java.util.concurrent.CompletableFuture
> org.apache.flink.runtime.webmonitor.RestfulGateway.triggerSavepoint(org.apache.flink.api.common.JobID,java.lang.String,boolean,org.apache.flink.api.common.time.Time)
> timed out.
> 2020-06-04T16:17:20.4409562Z at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2020-06-04T16:17:20.4410333Z at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2020-06-04T16:17:20.4411259Z at
> org.apache.flink.runtime.jobmaster.JobMasterTriggerSavepointITCase.cancelWithSavepoint(JobMasterTriggerSavepointITCase.java:264)
> 2020-06-04T16:17:20.4412292Z at
> org.apache.flink.runtime.jobmaster.JobMasterTriggerSavepointITCase.testStopJobAfterSavepoint(JobMasterTriggerSavepointITCase.java:127)
> 2020-06-04T16:17:20.4413163Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-06-04T16:17:20.4413990Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-06-04T16:17:20.4414783Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-06-04T16:17:20.4415936Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-06-04T16:17:20.4416693Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-06-04T16:17:20.4417632Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-06-04T16:17:20.4418637Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-06-04T16:17:20.4419367Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-06-04T16:17:20.4420118Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-06-04T16:17:20.4420742Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-06-04T16:17:20.4421909Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-06-04T16:17:20.4422493Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-06-04T16:17:20.4423247Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-06-04T16:17:20.4424263Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-06-04T16:17:20.4424876Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-06-04T16:17:20.4426346Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-06-04T16:17:20.4427052Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-06-04T16:17:20.4427772Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-06-04T16:17:20.4428562Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-06-04T16:17:20.4429158Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-06-04T16:17:20.4429861Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-06-04T16:17:20.4430448Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-06-04T16:17:20.4431060Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-06-04T16:17:20.4431678Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-06-04T16:17:20.4432513Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Pro
[jira] [Created] (FLINK-18137) JobMasterTriggerSavepointITCase.testStopJobAfterSavepoint fails with AskTimeoutException
Robert Metzger created FLINK-18137:
--
Summary: JobMasterTriggerSavepointITCase.testStopJobAfterSavepoint
fails with AskTimeoutException
Key: FLINK-18137
URL: https://issues.apache.org/jira/browse/FLINK-18137
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination, Runtime / Task
Affects Versions: 1.11.0
Reporter: Robert Metzger
Fix For: 1.11.0
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2747&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=45cc9205-bdb7-5b54-63cd-89fdc0983323
{code}
2020-06-04T16:17:20.4404189Z [ERROR] Tests run: 4, Failures: 0, Errors: 1,
Skipped: 0, Time elapsed: 14.352 s <<< FAILURE! - in
org.apache.flink.runtime.jobmaster.JobMasterTriggerSavepointITCase
2020-06-04T16:17:20.4405548Z [ERROR]
testStopJobAfterSavepoint(org.apache.flink.runtime.jobmaster.JobMasterTriggerSavepointITCase)
Time elapsed: 10.058 s <<< ERROR!
2020-06-04T16:17:20.4407342Z java.util.concurrent.ExecutionException:
java.util.concurrent.TimeoutException: Invocation of public default
java.util.concurrent.CompletableFuture
org.apache.flink.runtime.webmonitor.RestfulGateway.triggerSavepoint(org.apache.flink.api.common.JobID,java.lang.String,boolean,org.apache.flink.api.common.time.Time)
timed out.
2020-06-04T16:17:20.4409562Zat
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
2020-06-04T16:17:20.4410333Zat
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
2020-06-04T16:17:20.4411259Zat
org.apache.flink.runtime.jobmaster.JobMasterTriggerSavepointITCase.cancelWithSavepoint(JobMasterTriggerSavepointITCase.java:264)
2020-06-04T16:17:20.4412292Zat
org.apache.flink.runtime.jobmaster.JobMasterTriggerSavepointITCase.testStopJobAfterSavepoint(JobMasterTriggerSavepointITCase.java:127)
2020-06-04T16:17:20.4413163Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-06-04T16:17:20.4413990Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-06-04T16:17:20.4414783Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-06-04T16:17:20.4415936Zat
java.lang.reflect.Method.invoke(Method.java:498)
2020-06-04T16:17:20.4416693Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-06-04T16:17:20.4417632Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-06-04T16:17:20.4418637Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-06-04T16:17:20.4419367Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-06-04T16:17:20.4420118Zat
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-06-04T16:17:20.4420742Zat
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2020-06-04T16:17:20.4421909Zat
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-06-04T16:17:20.4422493Zat
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2020-06-04T16:17:20.4423247Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2020-06-04T16:17:20.4424263Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2020-06-04T16:17:20.4424876Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-06-04T16:17:20.4426346Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-06-04T16:17:20.4427052Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-06-04T16:17:20.4427772Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-06-04T16:17:20.4428562Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-06-04T16:17:20.4429158Zat
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-06-04T16:17:20.4429861Zat
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-06-04T16:17:20.4430448Zat
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-06-04T16:17:20.4431060Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-06-04T16:17:20.4431678Zat
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
2020-06-04T16:17:20.4432513Zat
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
2020-06-04T16:17:20.4433396Zat
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
2020-06-04T16:17:20.4434298Zat
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
2020-06-04T16:17:20.4440904Zat
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2020-06-
[jira] [Updated] (FLINK-17384) Support reading hbase conf dir from flink-conf.yaml
[ https://issues.apache.org/jira/browse/FLINK-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-17384: -- Summary: Support reading hbase conf dir from flink-conf.yaml (was: support read hbase conf dir from flink.conf just like hadoop_conf) > Support reading hbase conf dir from flink-conf.yaml > --- > > Key: FLINK-17384 > URL: https://issues.apache.org/jira/browse/FLINK-17384 > Project: Flink > Issue Type: Bug > Components: Connectors / HBase, Deployment / Scripts >Affects Versions: 1.10.0 >Reporter: jackylau >Assignee: jackylau >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > hi all: > when user interacts with hbase should do 2 things when using sql > # export HBASE_CONF_DIR > # add hbase libs to flink_lib(because the hbase connnector doesn't have > client's( and others) jar) > i think it needs to optimise it. > for 1) we should support read hbase conf dir from flink.conf just like > hadoop_conf in config.sh > for 2) we should support HBASE_CLASSPATH in config.sh. In case of jar > conflicts such as guava , we also should support flink-hbase-shaded just like > hadoop does -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] carp84 closed pull request #12144: [FLINK-17384][flink-dist] support read hbase conf dir from flink.conf and change HBaseConfiguration construction.
carp84 closed pull request #12144: URL: https://github.com/apache/flink/pull/12144 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18138) KeyedComplexChainTest.testMigrationAndRestore fails with AskTimeoutException on CI
Robert Metzger created FLINK-18138:
--
Summary: KeyedComplexChainTest.testMigrationAndRestore fails with
AskTimeoutException on CI
Key: FLINK-18138
URL: https://issues.apache.org/jira/browse/FLINK-18138
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination, Tests
Affects Versions: 1.12.0
Reporter: Robert Metzger
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2723&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=f508e270-48d6-5f1e-3138-42a17e0714f0
{code}
2020-06-04T12:46:11.4458612Z [ERROR] Tests run: 7, Failures: 0, Errors: 1,
Skipped: 0, Time elapsed: 18.942 s <<< FAILURE! - in
org.apache.flink.test.state.operator.restore.keyed.KeyedComplexChainTest
2020-06-04T12:46:11.4461142Z [ERROR] testMigrationAndRestore[Migrate Savepoint:
1.7](org.apache.flink.test.state.operator.restore.keyed.KeyedComplexChainTest)
Time elapsed: 10.486 s <<< ERROR!
2020-06-04T12:46:11.4464014Z java.util.concurrent.ExecutionException:
java.util.concurrent.TimeoutException: Invocation of public default
java.util.concurrent.CompletableFuture
org.apache.flink.runtime.webmonitor.RestfulGateway.triggerSavepoint(org.apache.flink.api.common.JobID,java.lang.String,boolean,org.apache.flink.api.common.time.Time)
timed out.
2020-06-04T12:46:11.4466561Zat
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
2020-06-04T12:46:11.4468016Zat
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
2020-06-04T12:46:11.4469645Zat
org.apache.flink.test.state.operator.restore.AbstractOperatorRestoreTestBase.migrateJob(AbstractOperatorRestoreTestBase.java:145)
2020-06-04T12:46:11.4475048Zat
org.apache.flink.test.state.operator.restore.AbstractOperatorRestoreTestBase.testMigrationAndRestore(AbstractOperatorRestoreTestBase.java:107)
2020-06-04T12:46:11.4476558Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-06-04T12:46:11.4477687Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-06-04T12:46:11.4479033Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-06-04T12:46:11.4480310Zat
java.lang.reflect.Method.invoke(Method.java:498)
2020-06-04T12:46:11.4481369Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-06-04T12:46:11.4482508Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-06-04T12:46:11.4483663Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-06-04T12:46:11.4485020Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-06-04T12:46:11.4486066Zat
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-06-04T12:46:11.4487061Zat
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-06-04T12:46:11.4488097Zat
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2020-06-04T12:46:11.4488951Zat
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-06-04T12:46:11.4489934Zat
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2020-06-04T12:46:11.4493504Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2020-06-04T12:46:11.4495100Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2020-06-04T12:46:11.4496193Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-06-04T12:46:11.4497136Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-06-04T12:46:11.4498138Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-06-04T12:46:11.4499404Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-06-04T12:46:11.4500352Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-06-04T12:46:11.4501250Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-06-04T12:46:11.4502194Zat
org.junit.runners.Suite.runChild(Suite.java:128)
2020-06-04T12:46:11.4502980Zat
org.junit.runners.Suite.runChild(Suite.java:27)
2020-06-04T12:46:11.4503827Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-06-04T12:46:11.4504861Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-06-04T12:46:11.4505812Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-06-04T12:46:11.4506761Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-06-04T12:46:11.4507953Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-06-04T12:46:11.4508872Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-06-04T12:46:11.4509847Zat
org.apache.maven.surefire
[jira] [Updated] (FLINK-17384) Support reading hbase conf dir from flink-conf.yaml
[ https://issues.apache.org/jira/browse/FLINK-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-17384: -- Fix Version/s: (was: 1.11.0) 1.12.0 Issue Type: Improvement (was: Bug) Changing JIRA type to Improvement since it's not a bug, and change fix version to 1.12.0 since we've long passed feature freeze of 1.11.0 > Support reading hbase conf dir from flink-conf.yaml > --- > > Key: FLINK-17384 > URL: https://issues.apache.org/jira/browse/FLINK-17384 > Project: Flink > Issue Type: Improvement > Components: Connectors / HBase, Deployment / Scripts >Affects Versions: 1.10.0 >Reporter: jackylau >Assignee: jackylau >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > hi all: > when user interacts with hbase should do 2 things when using sql > # export HBASE_CONF_DIR > # add hbase libs to flink_lib(because the hbase connnector doesn't have > client's( and others) jar) > i think it needs to optimise it. > for 1) we should support read hbase conf dir from flink.conf just like > hadoop_conf in config.sh > for 2) we should support HBASE_CLASSPATH in config.sh. In case of jar > conflicts such as guava , we also should support flink-hbase-shaded just like > hadoop does -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12489: [FLINK-18136] Don't start channel state writing for savepoints (RPC)
flinkbot edited a comment on pull request #12489: URL: https://github.com/apache/flink/pull/12489#issuecomment-638985663 ## CI report: * 5136fc6b80380c9ff9be14c67d6a286dbac118a5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2751) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12452: [FLINK-18056][fs-connector] Removing the remaining in-progress file on initialization for hadoop path-based writer
flinkbot edited a comment on pull request #12452: URL: https://github.com/apache/flink/pull/12452#issuecomment-637922636 ## CI report: * 7cb10bd1269179bbe66f5b840c0f0d2494c66a55 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2598) * fef42fb8497d9822a12ab18114ecb0ea4f36509f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2750) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17384) Support reading hbase conf dir from flink-conf.yaml
[ https://issues.apache.org/jira/browse/FLINK-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-17384: -- Description: Currently when using hbase connector with Flink SQL, below manual steps are required: # export HBASE_CONF_DIR # add hbase libs to flink_lib(because the hbase connnector doesn't have client's( and others) jar) And we should improve this. For 1) we could support read hbase conf dir from flink-conf.yaml just like hadoop/yarn does. For 2) we should support HBASE_CLASSPATH in config.sh. In case of jar conflicts such as guava, we also should support flink-hbase-shaded just like hadoop does In this JIRA we focus on implementing the 1st proposal. was: hi all: when user interacts with hbase should do 2 things when using sql # export HBASE_CONF_DIR # add hbase libs to flink_lib(because the hbase connnector doesn't have client's( and others) jar) i think it needs to optimise it. for 1) we should support read hbase conf dir from flink.conf just like hadoop_conf in config.sh for 2) we should support HBASE_CLASSPATH in config.sh. In case of jar conflicts such as guava , we also should support flink-hbase-shaded just like hadoop does Editing the JIRA description to better reflect what the related PR resolves. Please feel free to open another one for the 2nd proposal (while I think the shaded part would need some discussions and reach consensus before working on it) [~jackylau]. > Support reading hbase conf dir from flink-conf.yaml > --- > > Key: FLINK-17384 > URL: https://issues.apache.org/jira/browse/FLINK-17384 > Project: Flink > Issue Type: Improvement > Components: Connectors / HBase, Deployment / Scripts >Affects Versions: 1.10.0 >Reporter: jackylau >Assignee: jackylau >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > Currently when using hbase connector with Flink SQL, below manual steps are > required: > # export HBASE_CONF_DIR > # add hbase libs to flink_lib(because the hbase connnector doesn't have > client's( and others) jar) > And we should improve this. > For 1) we could support read hbase conf dir from flink-conf.yaml just like > hadoop/yarn does. > For 2) we should support HBASE_CLASSPATH in config.sh. In case of jar > conflicts such as guava, we also should support flink-hbase-shaded just like > hadoop does > In this JIRA we focus on implementing the 1st proposal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-17384) Support reading hbase conf dir from flink-conf.yaml
[ https://issues.apache.org/jira/browse/FLINK-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li closed FLINK-17384. - Release Note: After FLINK-17384 hbase connector supports reading hbase configuration directory from flink-conf.yaml through the `env.hbase.conf.dir` property when the `HBASE_CONF_DIR` environment variable is not set. Resolution: Implemented Merged into master via 0799b5c20a127110e47439668cf8f8db2e4ecbf3 > Support reading hbase conf dir from flink-conf.yaml > --- > > Key: FLINK-17384 > URL: https://issues.apache.org/jira/browse/FLINK-17384 > Project: Flink > Issue Type: Improvement > Components: Connectors / HBase, Deployment / Scripts >Affects Versions: 1.10.0 >Reporter: jackylau >Assignee: jackylau >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > Currently when using hbase connector with Flink SQL, below manual steps are > required: > # export HBASE_CONF_DIR > # add hbase libs to flink_lib(because the hbase connnector doesn't have > client's( and others) jar) > And we should improve this. > For 1) we could support read hbase conf dir from flink-conf.yaml just like > hadoop/yarn does. > For 2) we should support HBASE_CLASSPATH in config.sh. In case of jar > conflicts such as guava, we also should support flink-hbase-shaded just like > hadoop does > In this JIRA we focus on implementing the 1st proposal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17860) Recursively remove channel state directories
[ https://issues.apache.org/jira/browse/FLINK-17860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126121#comment-17126121 ] Roman Khachatryan commented on FLINK-17860: --- After fixing FLINK-17988 there is no immediate need to Recursively remove channel state directories However, I think it should be considered for the next release of Unaligned Checkpoints. > Recursively remove channel state directories > > > Key: FLINK-17860 > URL: https://issues.apache.org/jira/browse/FLINK-17860 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.11.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Critical > Fix For: 1.12.0 > > > With a high degree of parallelism, we end up with n*s number of files in each > checkpoint (n = parallelism, s = stages). Writing them if fast (from many > subtasks), removing them is slow (from JM). > This can't be mitigated by state.backend.fs.memory-threshold because most > states are ten to hundreds Mb. > > Instead of going through them 1 by 1, we could remove the directory > recursively. > > The easiest way is to remove channelStateHandle.discard() calls and use > isRecursive=true in > FsCompletedCheckpointStorageLocation.disposeStorageLocation. > Note: with the current isRecursive=false there will be an exception if there > are any files left under that folder. > > This can be extended to other state handles in future as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
