[jira] [Comment Edited] (FLINK-15571) Create a Redis Streams Connector for Flink
[ https://issues.apache.org/jira/browse/FLINK-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014238#comment-17014238 ] Yun Tang edited comment on FLINK-15571 at 1/13/20 11:39 AM: [~tgrall] , Flink once had a Redis sink connector and moved to [Apache Bahir|https://bahir.apache.org/] three years ago, you can refer to the discussion mail [here|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Move-Redis-and-Flume-connectors-to-Apache-Bahir-and-redirect-contributions-there-td13102.html] . Since Redis stream is introduced since Redis-5.0 and different from previous Redis sink. I think you could start a discussion in dev mail list to talk about whether to introduce a new Redis connector. was (Author: yunta): Flink once had a Redis sink connector and moved to [Apache Bahir|https://bahir.apache.org/] three years ago, you can refer to the discussion mail [here|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Move-Redis-and-Flume-connectors-to-Apache-Bahir-and-redirect-contributions-there-td13102.html] . Since Redis stream is introduced since Redis-5.0 and different from previous Redis sink. I think you could start a discussion in dev mail list to talk about whether to introduce a new Redis connector. > Create a Redis Streams Connector for Flink > -- > > Key: FLINK-15571 > URL: https://issues.apache.org/jira/browse/FLINK-15571 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Tugdual Grall >Priority: Minor > > Redis has a "log data structure" called Redis Streams, it would be nice to > integrate Redis Streams and Apache Flink as: > * Source > * Sink > See Redis Streams introduction: [https://redis.io/topics/streams-intro] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
flinkbot edited a comment on issue #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests URL: https://github.com/apache/flink/pull/10665#issuecomment-568440738 ## CI report: * 5a0d5d3d499347ca216e19175ff5f066a6d9b458 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142099952) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3861) * 255d89be8069b36be2b980ea6dba4798568160bb Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143507134) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4176) * 9168102e928bacaa8026407f77a33b80a8ddeae4 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143514708) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4179) * 1465ba2465b0fba3c91377735481db17abb1c21f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143667456) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4203) * 76f3dadc5062279dda9df063901a548315a8b8da Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143714109) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4219) * 677c4fcd1b82c766f79e296883f8464daafb6a44 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144091754) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4278) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15571) Create a Redis Streams Connector for Flink
[ https://issues.apache.org/jira/browse/FLINK-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014238#comment-17014238 ] Yun Tang commented on FLINK-15571: -- Flink once had a Redis sink connector and moved to [Apache Bahir|https://bahir.apache.org/] three years ago, you can refer to the discussion mail [here|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Move-Redis-and-Flume-connectors-to-Apache-Bahir-and-redirect-contributions-there-td13102.html] . Since Redis stream is introduced since Redis-5.0 and different from previous Redis sink. I think you could start a discussion in dev mail list to talk about whether to introduce a new Redis connector. > Create a Redis Streams Connector for Flink > -- > > Key: FLINK-15571 > URL: https://issues.apache.org/jira/browse/FLINK-15571 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Tugdual Grall >Priority: Minor > > Redis has a "log data structure" called Redis Streams, it would be nice to > integrate Redis Streams and Apache Flink as: > * Source > * Sink > See Redis Streams introduction: [https://redis.io/topics/streams-intro] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10682: [FLINK-15247][Runtime] Wait for all slots to be free before task executor services shutdown upon stopping
flinkbot edited a comment on issue #10682: [FLINK-15247][Runtime] Wait for all slots to be free before task executor services shutdown upon stopping URL: https://github.com/apache/flink/pull/10682#issuecomment-568850864 ## CI report: * a7411a5d1a86381078670898c1f20be4fe3a88f4 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142305023) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3903) * 178357b40789d132e1db954a199bf1c62efff474 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143770612) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4240) * 8d2db0ad7b2ce3c268cd8c52a6332a2bc86aca61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144064773) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4271) * 5625904d8f9e74bb7a7f1ff392094a8c1ea9b9c2 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144068497) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4272) * e3be286cd26d25b91e5805eed149bdf5a39aa292 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144075835) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4275) * d9b76b1b111564fa5aed4a37888f0ec1001f895d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10674: [FLINK-15220][Connector/Kafka][Table] Add startFromTimestamp in KafkaTableSource
flinkbot edited a comment on issue #10674: [FLINK-15220][Connector/Kafka][Table] Add startFromTimestamp in KafkaTableSource URL: https://github.com/apache/flink/pull/10674#issuecomment-568687684 ## CI report: * f59106d121f9dd0fc40b23640ec2ad8a663d6020 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142201948) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3880) * a69cf4921d9a947d437a77c173ae979998ad14a1 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142214808) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3889) * a8681f8b91fd819cbee65212479e1263fb88bf64 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144107176) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4283) * 7a75e1c85bc511af3be0eaf977288af816a12b78 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144117224) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4288) * 632cf6bd9e8f24e2af1da1688227ca3e73fe6858 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] Li-Aihua closed pull request #10409: [FLINK-14918] [flink-end-to-end-perf-tests] Add performance e2e test module and scripts
Li-Aihua closed pull request #10409: [FLINK-14918] [flink-end-to-end-perf-tests] Add performance e2e test module and scripts URL: https://github.com/apache/flink/pull/10409 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15489) Cannot update jobmanager/taskmanager logs
[ https://issues.apache.org/jira/browse/FLINK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yadong Xie updated FLINK-15489: --- Attachment: repro.gif > Cannot update jobmanager/taskmanager logs > - > > Key: FLINK-15489 > URL: https://issues.apache.org/jira/browse/FLINK-15489 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.9.1, 1.10.0 >Reporter: Dawid Wysakowicz >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0 > > Attachments: 6E9A4D72-6664-4B00-B02F-57EB2695D967.png, > 7A1CF604-032D-46D8-AC70-FBB884E868B9.png, repro.gif, screenshot-1.png, > screenshot-2.png > > Time Spent: 10m > Remaining Estimate: 0h > > There is no way to query the latest state of logs of jobmanager/taskmanager. > The Web UI show only the first version that was ever displayed. > How to reproduce: > * (not sure if necessary) configure logback as described here: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html#use-logback-when-running-flink-on-a-cluster > * start a cluster > * show jobmanager logs in the Web UI > * run example job > * check again the jobmanager logs, there is no trace of the job. Clicking the > refresh button does not help -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15573) Let Flink SQL PlannerExpressionParserImpl#FieldRefrence use Unicode as its default charset
Lsw_aka_laplace created FLINK-15573:
---
Summary: Let Flink SQL PlannerExpressionParserImpl#FieldRefrence
use Unicode as its default charset
Key: FLINK-15573
URL: https://issues.apache.org/jira/browse/FLINK-15573
Project: Flink
Issue Type: Improvement
Components: Table SQL / Planner
Reporter: Lsw_aka_laplace
Now I am talking about the `PlannerExpressionParserImpl`
For now the fieldRefrence‘s charset is JavaIdentifier，why not change it
to UnicodeIdentifier?
Currently in my team, we do actually have this problem. For instance, data
from Es always contains `@timestamp` field , which can not meet JavaIdentifier.
So what we did is just let the fieldRefrence Charset use Unicode
{code:scala}
lazy val extensionIdent: Parser[String] = ( "" ~> // handle whitespace
rep1(acceptIf(Character.isUnicodeIdentifierStart)("identifier expected but '" +
_ + "' found"), elem("identifier part", Character.isUnicodeIdentifierPart(:
Char))) ^^ (.mkString) )
lazy val fieldReference: PackratParser[UnresolvedReferenceExpression] = (STAR
| ident | extensionIdent) ^^ { sym => unresolvedRef(sym) }{code}
It is simple but really make sense~
Looking forward for any opinion
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10838: [hotfix][table-common] Simplify formatted exceptions
flinkbot edited a comment on issue #10838: [hotfix][table-common] Simplify formatted exceptions URL: https://github.com/apache/flink/pull/10838#issuecomment-573544312 ## CI report: * 6d7cd665ed96239643ccbc81bafacc8fd0e12e80 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144107161) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4282) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10837: [FLINK-15541] Fix unstable case FlinkKinesisConsumerTest
flinkbot edited a comment on issue #10837: [FLINK-15541] Fix unstable case FlinkKinesisConsumerTest URL: https://github.com/apache/flink/pull/10837#issuecomment-573544281 ## CI report: * 4d045d632eb72a162e5d56aa52a6819042d2f1aa Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144107145) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4281) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL
flinkbot edited a comment on issue #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL URL: https://github.com/apache/flink/pull/10839#issuecomment-573563076 ## CI report: * c5595ea0c994dcf664cb68ec45a6f500e705a3b7 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144113663) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4285) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10841: [FLINK-15565][table-planner-blink] Fix error conversion of TinyInt, SmallInt literals between Flink and Calcite
flinkbot edited a comment on issue #10841: [FLINK-15565][table-planner-blink] Fix error conversion of TinyInt,SmallInt literals between Flink and Calcite URL: https://github.com/apache/flink/pull/10841#issuecomment-573572659 ## CI report: * d3abca3b9c9ca0ad030f7e208843502d14ff13a5 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144117186) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4287) * b5b112c7954d4961cfbc71f9d268a7a2561e6051 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10842: [FLINK-15489][web]: add cache control no-cache to log api
flinkbot commented on issue #10842: [FLINK-15489][web]: add cache control no-cache to log api URL: https://github.com/apache/flink/pull/10842#issuecomment-573618229 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit fbc3d61dc5657907e9e55497cf70431ceff2d2d9 (Mon Jan 13 11:25:21 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-15489).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10815: [FLINK-15537][table-planner-blink] Type of keys should be `BinaryRow`…
flinkbot edited a comment on issue #10815: [FLINK-15537][table-planner-blink] Type of keys should be `BinaryRow`… URL: https://github.com/apache/flink/pull/10815#issuecomment-572566376 ## CI report: * 19a4290f709495491fe460037c8c31d106984ea8 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143732723) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4229) * c3ef5ea345a343170806de8112163edb7df31f69 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144110200) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4284) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15489) Cannot update jobmanager/taskmanager logs
[ https://issues.apache.org/jira/browse/FLINK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-15489: --- Labels: pull-request-available (was: ) > Cannot update jobmanager/taskmanager logs > - > > Key: FLINK-15489 > URL: https://issues.apache.org/jira/browse/FLINK-15489 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.9.1, 1.10.0 >Reporter: Dawid Wysakowicz >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0 > > Attachments: 6E9A4D72-6664-4B00-B02F-57EB2695D967.png, > 7A1CF604-032D-46D8-AC70-FBB884E868B9.png, screenshot-1.png, screenshot-2.png > > > There is no way to query the latest state of logs of jobmanager/taskmanager. > The Web UI show only the first version that was ever displayed. > How to reproduce: > * (not sure if necessary) configure logback as described here: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html#use-logback-when-running-flink-on-a-cluster > * start a cluster > * show jobmanager logs in the Web UI > * run example job > * check again the jobmanager logs, there is no trace of the job. Clicking the > refresh button does not help -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] vthinkxie opened a new pull request #10842: [FLINK-15489][web]: add cache control no-cache to log api
vthinkxie opened a new pull request #10842: [FLINK-15489][web]: add cache control no-cache to log api URL: https://github.com/apache/flink/pull/10842 ## What is the purpose of the change fix https://issues.apache.org/jira/browse/FLINK-14270 ## Brief change log support more metric display at once ## Verifying this change - *Go to the log page* - *Click the refresh button* - *Check the network panel if the log with no-cache header* before: https://user-images.githubusercontent.com/1506722/72252288-f7674f00-3639-11ea-9861-93157e789082.png";> after: https://user-images.githubusercontent.com/1506722/72252293-fa623f80-3639-11ea-8a8a-049bf09b2677.png";> ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] curcur opened a new pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
curcur opened a new pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions URL: https://github.com/apache/flink/pull/10832 ## What is the purpose of the change `Execution#producedPartitions` are registered through an asynchronous interface `ShuffleMaster#registerPartitionWithProducer` to `ShuffleMaster`, however they are not always accessed through callbacks. So, it is possible that `Execution#producedPartitions` have not been available yet when accessed (in `Execution#deploy` for example). Since the only implementation of `ShuffleMaster` is `NettyShuffleMaster`, which indeed registers producedPartition in a synchronous way, this pull request enforces synchronous registrations under an asynchronous interface as a temporary fix to prevent future misuses. ## Brief change log - Make sure the future of registration is done immediately after registration. - Throws IllegalStateException otherwise. ## Verifying this change unit test: ExecutionTest#testSynchronousRegisterProducedPartitions ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? JavaDocs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] curcur removed a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
curcur removed a comment on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions URL: https://github.com/apache/flink/pull/10832#issuecomment-573614132 > Thanks for opening the PR @curcur . The change generally looks good to me. > I have a few comments for it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] curcur closed pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
curcur closed pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions URL: https://github.com/apache/flink/pull/10832 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] curcur commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
curcur commented on issue #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions URL: https://github.com/apache/flink/pull/10832#issuecomment-573614132 > Thanks for opening the PR @curcur . The change generally looks good to me. > I have a few comments for it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365749861
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphTestUtils.java
##
@@ -431,6 +433,18 @@ public static ExecutionJobVertex
getExecutionVertex(JobVertexID id) throws Excep
return getExecutionVertex(id, new
DirectScheduledExecutorService());
}
+ public static ExecutionEdge createMockExecutionEdge(int maxParallelism)
{
Review comment:
This is reused from ExecutionVertexDeploymentTest.
Let me have a try to see whether I can avoid Mockito and at the same time
not to make the code too complicated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365749861
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphTestUtils.java
##
@@ -431,6 +433,18 @@ public static ExecutionJobVertex
getExecutionVertex(JobVertexID id) throws Excep
return getExecutionVertex(id, new
DirectScheduledExecutorService());
}
+ public static ExecutionEdge createMockExecutionEdge(int maxParallelism)
{
Review comment:
This is reused from ExecutionVertexDeploymentTest.
Let me have a try to see whether we can avoid Mockito and at the same time
not to make the code too complicated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Comment Edited] (FLINK-14163) Execution#producedPartitions is possibly not assigned when used
[
https://issues.apache.org/jira/browse/FLINK-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014147#comment-17014147
]
Yuan Mei edited comment on FLINK-14163 at 1/13/20 11:04 AM:
Synced up with [~zjwang] offline.
In general, I would prefer not to change interfaces unless having to, since
interface changes have more profound impact on users than implementation
changes. Especially in this case, given the future design is not determined
(both Shuffle and Scheduler), changing async -> sycn and then sync -> async
again bring much more pain to users than keeping the asycn interfaces and
enforces a sync implementation underneath.
In the latter case, a user barely needs to do anything if we do want to support
async implementation in the future, while in the first case, every single user
that uses the interface has to make a new release to adjust to the updated
interface.
I do not have strong preferences other than the point of keeping the interface.
I think retain a timeout is a good compromise, but may introduce some
inconsistent system behavior. Hence even if we decide to retain a timeout, I
would prefer to document as `enforce a synchronous implementation under
asynchronous interface`.
was (Author: ym):
Synced up with [~zjwang] offline.
In general, I would prefer not to change interfaces unless having to, since
interface changes have more profound impact on users than implementation
changes. Especially in this case, given the future design is not determined
(both Shuffle and Scheduler), changing async -> sycn and then sync -> async
again bring much more pain to users than keeping the asycn interfaces and
enforces a sync implementation underneath.
In the latter case, a user barely needs to do anything if we do want to support
async implementation in the future, while in the first case, every single user
that uses the interface has to make a new release to adjust to the updated
interface.
I do not have strong preferences other than the point to keep the interface. I
think retain a timeout is a good compromise, but may introduce some
inconsistent system behavior. Hence even if we decide to retain a timeout, I
would prefer to document as `enforce a synchronous implementation under
asynchronous interface`.
> Execution#producedPartitions is possibly not assigned when used
> ---
>
> Key: FLINK-14163
> URL: https://issues.apache.org/jira/browse/FLINK-14163
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhu Zhu
>Assignee: Yuan Mei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently {{Execution#producedPartitions}} is assigned after the partitions
> have completed the registration to shuffle master in
> {{Execution#registerProducedPartitions(...)}}.
> The partition registration is an async interface
> ({{ShuffleMaster#registerPartitionWithProducer(...)}}), so
> {{Execution#producedPartitions}} is possible[1] not set when used.
> Usages includes:
> 1. deploying this task, so that the task may be deployed without its result
> partitions assigned, and the job would hang. (DefaultScheduler issue only,
> since legacy scheduler handled this case)
> 2. generating input descriptors for downstream tasks:
> 3. retrieve {{ResultPartitionID}} for partition releasing:
> [1] If a user uses Flink default shuffle master {{NettyShuffleMaster}}, it is
> not problematic at the moment since it returns a completed future on
> registration, so that it would be a synchronized process. However, if users
> implement their own shuffle service in which the
> {{ShuffleMaster#registerPartitionWithProducer}} returns an pending future, it
> can be a problem. This is possible since customizable shuffle service is open
> to users since 1.9 (via config "shuffle-service-factory.class").
> To avoid issues to happen, we may either
> 1. fix all the usages of {{Execution#producedPartitions}} regarding the async
> assigning, or
> 2. change {{ShuffleMaster#registerPartitionWithProducer(...)}} to a sync
> interface
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15174) FLINK security using PKI mutual auth needs certificate pinning or Private CA
[ https://issues.apache.org/jira/browse/FLINK-15174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014222#comment-17014222 ] Stephan Ewen commented on FLINK-15174: -- [~knaufk] We can continue the discussion about whether there are other possible workarounds, but the patch as such works and is meaningful in my opinion, so I would close this issue. > FLINK security using PKI mutual auth needs certificate pinning or Private CA > > > Key: FLINK-15174 > URL: https://issues.apache.org/jira/browse/FLINK-15174 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration, Runtime / REST >Affects Versions: 1.9.0, 1.9.1, 1.10.0 >Reporter: Bhagavan >Assignee: Bhagavan >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The current design for Flink security for internal/REST relies on PKI mutual > authentication. However, the design is not robust if CA used for generating > certificates are public CA or Firwide internal CA. This is due to how the > chain of trust works whilst validating the client certificate. i.e. Any > certificate signed by same CA would be able to make a connection to internal > Flink network. > Proposed improvement. > An environment where operators are constrained to use firmwide Internal > public CA, Allow the operator to specify the certificate fingerprint to > further protect the cluster allowing only specific certificate. > This change should be a backward compatible change where one can use just > certificate with private CA. > Changes are easy to implement as all network communications are done using > netty and netty provides FingerprintTrustManagerFactory. > Happy to send PR if we agree on the change. > Document corrections. > From security documentation. > [https://ci.apache.org/projects/flink/flink-docs-stable/ops/security-ssl.html] > _"All internal connections are SSL authenticated and encrypted. The > connections use *mutual authentication*, meaning both server and client-side > of each connection need to present the certificate to each other. The > certificate acts effectively as a shared secret."_ > _-_ This not exactly true. Any party who obtains the client certificate from > CA would be able to form the connection even though the certificate > public/private keys are different. So it's not *a* shared secret ( merely a > common signature) > _Further doc says - "A common setup is to generate a dedicated certificate > (maybe self-signed) for a Flink deployment._ > - I think this is the only way to make the cluster secure. i.e. create > private CA just for the cluster. > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15174) FLINK security using PKI mutual auth needs certificate pinning or Private CA
[ https://issues.apache.org/jira/browse/FLINK-15174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-15174. > FLINK security using PKI mutual auth needs certificate pinning or Private CA > > > Key: FLINK-15174 > URL: https://issues.apache.org/jira/browse/FLINK-15174 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration, Runtime / REST >Affects Versions: 1.9.0, 1.9.1, 1.10.0 >Reporter: Bhagavan >Assignee: Bhagavan >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The current design for Flink security for internal/REST relies on PKI mutual > authentication. However, the design is not robust if CA used for generating > certificates are public CA or Firwide internal CA. This is due to how the > chain of trust works whilst validating the client certificate. i.e. Any > certificate signed by same CA would be able to make a connection to internal > Flink network. > Proposed improvement. > An environment where operators are constrained to use firmwide Internal > public CA, Allow the operator to specify the certificate fingerprint to > further protect the cluster allowing only specific certificate. > This change should be a backward compatible change where one can use just > certificate with private CA. > Changes are easy to implement as all network communications are done using > netty and netty provides FingerprintTrustManagerFactory. > Happy to send PR if we agree on the change. > Document corrections. > From security documentation. > [https://ci.apache.org/projects/flink/flink-docs-stable/ops/security-ssl.html] > _"All internal connections are SSL authenticated and encrypted. The > connections use *mutual authentication*, meaning both server and client-side > of each connection need to present the certificate to each other. The > certificate acts effectively as a shared secret."_ > _-_ This not exactly true. Any party who obtains the client certificate from > CA would be able to form the connection even though the certificate > public/private keys are different. So it's not *a* shared secret ( merely a > common signature) > _Further doc says - "A common setup is to generate a dedicated certificate > (maybe self-signed) for a Flink deployment._ > - I think this is the only way to make the cluster secure. i.e. create > private CA just for the cluster. > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-15174) FLINK security using PKI mutual auth needs certificate pinning or Private CA
[ https://issues.apache.org/jira/browse/FLINK-15174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-15174. -- Fix Version/s: (was: 1.9.2) 1.11.0 Resolution: Fixed I resolved this issue for 1.10.0 and 1.11.0 > FLINK security using PKI mutual auth needs certificate pinning or Private CA > > > Key: FLINK-15174 > URL: https://issues.apache.org/jira/browse/FLINK-15174 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration, Runtime / REST >Affects Versions: 1.9.0, 1.9.1, 1.10.0 >Reporter: Bhagavan >Assignee: Bhagavan >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The current design for Flink security for internal/REST relies on PKI mutual > authentication. However, the design is not robust if CA used for generating > certificates are public CA or Firwide internal CA. This is due to how the > chain of trust works whilst validating the client certificate. i.e. Any > certificate signed by same CA would be able to make a connection to internal > Flink network. > Proposed improvement. > An environment where operators are constrained to use firmwide Internal > public CA, Allow the operator to specify the certificate fingerprint to > further protect the cluster allowing only specific certificate. > This change should be a backward compatible change where one can use just > certificate with private CA. > Changes are easy to implement as all network communications are done using > netty and netty provides FingerprintTrustManagerFactory. > Happy to send PR if we agree on the change. > Document corrections. > From security documentation. > [https://ci.apache.org/projects/flink/flink-docs-stable/ops/security-ssl.html] > _"All internal connections are SSL authenticated and encrypted. The > connections use *mutual authentication*, meaning both server and client-side > of each connection need to present the certificate to each other. The > certificate acts effectively as a shared secret."_ > _-_ This not exactly true. Any party who obtains the client certificate from > CA would be able to form the connection even though the certificate > public/private keys are different. So it's not *a* shared secret ( merely a > common signature) > _Further doc says - "A common setup is to generate a dedicated certificate > (maybe self-signed) for a Flink deployment._ > - I think this is the only way to make the cluster secure. i.e. create > private CA just for the cluster. > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365745754
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##
@@ -540,6 +557,65 @@ public void testSlotReleaseAtomicallyReleasesExecution()
throws Exception {
});
}
+ /**
+* Tests that producedPartitions are registered synchronously under an
asynchronous interface.
+*/
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws
Exception {
+ ExecutionJobVertex jobVertex = getExecutionVertex(
+ new JobVertexID(),
+ new DirectScheduledExecutorService(),
+ EAGER);
+ IntermediateResult result =
+ new IntermediateResult(new
IntermediateDataSetID(), jobVertex, 1, ResultPartitionType.PIPELINED);
+ ExecutionVertex vertex =
+ new ExecutionVertex(jobVertex, 0, new
IntermediateResult[]{result}, Time.minutes(1));
+ ExecutionEdge edge = createMockExecutionEdge(1);
+ result.getPartitions()[0].addConsumerGroup();
+ result.getPartitions()[0].addConsumer(edge, 0);
+ TaskManagerLocation location =
+ new TaskManagerLocation(ResourceID.generate(),
InetAddress.getLoopbackAddress(), 1);
+ ExecutionAttemptID attemptID = new ExecutionAttemptID();
+ Execution.registerProducedPartitions(vertex, location,
attemptID, EAGER.allowLazyDeployment());
+ }
+
+ @Nonnull
+ private ExecutionJobVertex getExecutionVertex(
Review comment:
that's a good idea, will do it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-15174) FLINK security using PKI mutual auth needs certificate pinning or Private CA
[ https://issues.apache.org/jira/browse/FLINK-15174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014220#comment-17014220 ] Stephan Ewen commented on FLINK-15174: -- The patch is merged in - 1.11.0 via 16eed2928bad97669e13ad521b2ca9fa5db3c582 and 69a727d1f9b6ef39ffb707f6d4ef46c534944ab5 - 1.10.0 via 83db7bc122e48248e8e66b816cbd3ed630878969 and 7ddd746b2b9d26c02826cc92c58ed9134f34bf0b > FLINK security using PKI mutual auth needs certificate pinning or Private CA > > > Key: FLINK-15174 > URL: https://issues.apache.org/jira/browse/FLINK-15174 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration, Runtime / REST >Affects Versions: 1.9.0, 1.9.1, 1.10.0 >Reporter: Bhagavan >Assignee: Bhagavan >Priority: Critical > Labels: pull-request-available > Fix For: 1.9.2, 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The current design for Flink security for internal/REST relies on PKI mutual > authentication. However, the design is not robust if CA used for generating > certificates are public CA or Firwide internal CA. This is due to how the > chain of trust works whilst validating the client certificate. i.e. Any > certificate signed by same CA would be able to make a connection to internal > Flink network. > Proposed improvement. > An environment where operators are constrained to use firmwide Internal > public CA, Allow the operator to specify the certificate fingerprint to > further protect the cluster allowing only specific certificate. > This change should be a backward compatible change where one can use just > certificate with private CA. > Changes are easy to implement as all network communications are done using > netty and netty provides FingerprintTrustManagerFactory. > Happy to send PR if we agree on the change. > Document corrections. > From security documentation. > [https://ci.apache.org/projects/flink/flink-docs-stable/ops/security-ssl.html] > _"All internal connections are SSL authenticated and encrypted. The > connections use *mutual authentication*, meaning both server and client-side > of each connection need to present the certificate to each other. The > certificate acts effectively as a shared secret."_ > _-_ This not exactly true. Any party who obtains the client certificate from > CA would be able to form the connection even though the certificate > public/private keys are different. So it's not *a* shared secret ( merely a > common signature) > _Further doc says - "A common setup is to generate a dedicated certificate > (maybe self-signed) for a Flink deployment._ > - I think this is the only way to make the cluster secure. i.e. create > private CA just for the cluster. > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] curcur commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
curcur commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365745182
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##
@@ -540,6 +557,65 @@ public void testSlotReleaseAtomicallyReleasesExecution()
throws Exception {
});
}
+ /**
+* Tests that producedPartitions are registered synchronously under an
asynchronous interface.
+*/
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws
Exception {
+ ExecutionJobVertex jobVertex = getExecutionVertex(
+ new JobVertexID(),
+ new DirectScheduledExecutorService(),
+ EAGER);
+ IntermediateResult result =
+ new IntermediateResult(new
IntermediateDataSetID(), jobVertex, 1, ResultPartitionType.PIPELINED);
+ ExecutionVertex vertex =
+ new ExecutionVertex(jobVertex, 0, new
IntermediateResult[]{result}, Time.minutes(1));
+ ExecutionEdge edge = createMockExecutionEdge(1);
+ result.getPartitions()[0].addConsumerGroup();
+ result.getPartitions()[0].addConsumer(edge, 0);
Review comment:
Yep, this part is needed, there has to be one consumer.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-15174) FLINK security using PKI mutual auth needs certificate pinning or Private CA
[ https://issues.apache.org/jira/browse/FLINK-15174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014219#comment-17014219 ] Stephan Ewen commented on FLINK-15174: -- [~knaufk] I tried to describe that above: The limitation in the JDK implementation is that any non-self-signed certificate needs the issuing CA in the TrustStore, which would then cause also all other certificates by that CA to be trusted. Just having the certificate or even only having the purpose-dedicated intermediate CA would not work (validation would fail). > FLINK security using PKI mutual auth needs certificate pinning or Private CA > > > Key: FLINK-15174 > URL: https://issues.apache.org/jira/browse/FLINK-15174 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration, Runtime / REST >Affects Versions: 1.9.0, 1.9.1, 1.10.0 >Reporter: Bhagavan >Assignee: Bhagavan >Priority: Critical > Labels: pull-request-available > Fix For: 1.9.2, 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The current design for Flink security for internal/REST relies on PKI mutual > authentication. However, the design is not robust if CA used for generating > certificates are public CA or Firwide internal CA. This is due to how the > chain of trust works whilst validating the client certificate. i.e. Any > certificate signed by same CA would be able to make a connection to internal > Flink network. > Proposed improvement. > An environment where operators are constrained to use firmwide Internal > public CA, Allow the operator to specify the certificate fingerprint to > further protect the cluster allowing only specific certificate. > This change should be a backward compatible change where one can use just > certificate with private CA. > Changes are easy to implement as all network communications are done using > netty and netty provides FingerprintTrustManagerFactory. > Happy to send PR if we agree on the change. > Document corrections. > From security documentation. > [https://ci.apache.org/projects/flink/flink-docs-stable/ops/security-ssl.html] > _"All internal connections are SSL authenticated and encrypted. The > connections use *mutual authentication*, meaning both server and client-side > of each connection need to present the certificate to each other. The > certificate acts effectively as a shared secret."_ > _-_ This not exactly true. Any party who obtains the client certificate from > CA would be able to form the connection even though the certificate > public/private keys are different. So it's not *a* shared secret ( merely a > common signature) > _Further doc says - "A common setup is to generate a dedicated certificate > (maybe self-signed) for a Flink deployment._ > - I think this is the only way to make the cluster secure. i.e. create > private CA just for the cluster. > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15568) RestartPipelinedRegionStrategy: not ensure the EXACTLY_ONCE semantics, in 1.8 versions
[ https://issues.apache.org/jira/browse/FLINK-15568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014212#comment-17014212 ] Zhu Zhu edited comment on FLINK-15568 at 1/13/20 10:51 AM: --- Hi [~andrew_lin], RestartPipelinedRegionStrategy is experimental in 1.8 with some known issues. It becomes production ready since 1.9 (FLINK-4256). State restore is also supported since then. There is no plan to backport the changes to 1.8 since the changes are huge and not bug fixes. was (Author: zhuzh): Hi [~andrew_lin], RestartPipelinedRegionStrategy is experimental in 1.8 with some known issues. It becomes production ready since 1.9 (FLINK-4256). There is no plan to backport the changes to 1.8 since the changes are huge and not bug fixes. > RestartPipelinedRegionStrategy: not ensure the EXACTLY_ONCE semantics, in 1.8 > versions > -- > > Key: FLINK-15568 > URL: https://issues.apache.org/jira/browse/FLINK-15568 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.8.0, 1.8.1, 1.8.3 >Reporter: Andrew.D.lin >Priority: Minor > Attachments: image-2020-01-13-16-40-47-888.png > > > In 1.8* versions, FailoverRegion.java restart method not restore from latest > checkpoint and marked TODO. > Should we support this feature (region restart) in flink 1.8? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15568) RestartPipelinedRegionStrategy: not ensure the EXACTLY_ONCE semantics, in 1.8 versions
[ https://issues.apache.org/jira/browse/FLINK-15568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014212#comment-17014212 ] Zhu Zhu commented on FLINK-15568: - Hi [~andrew_lin], RestartPipelinedRegionStrategy is experimental in 1.8 with some known issues. It becomes production ready since 1.9 (FLINK-4256). There is no plan to backport the changes to 1.8 since the changes are huge and not bug fixes. > RestartPipelinedRegionStrategy: not ensure the EXACTLY_ONCE semantics, in 1.8 > versions > -- > > Key: FLINK-15568 > URL: https://issues.apache.org/jira/browse/FLINK-15568 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.8.0, 1.8.1, 1.8.3 >Reporter: Andrew.D.lin >Priority: Minor > Attachments: image-2020-01-13-16-40-47-888.png > > > In 1.8* versions, FailoverRegion.java restart method not restore from latest > checkpoint and marked TODO. > Should we support this feature (region restart) in flink 1.8? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15066) Cannot run multiple `insert into csvTable values ()`
[
https://issues.apache.org/jira/browse/FLINK-15066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014211#comment-17014211
]
Jingsong Lee commented on FLINK-15066:
--
[~TsReaper] If you are interesting for this, you can take a look to:
https://issues.apache.org/jira/browse/FLINK-14266
https://issues.apache.org/jira/browse/FLINK-14267
Plan to introduce new Csv, and integrated to {{FileSystemOutputFormat}}.
> Cannot run multiple `insert into csvTable values ()`
>
>
> Key: FLINK-15066
> URL: https://issues.apache.org/jira/browse/FLINK-15066
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Reporter: Kurt Young
>Assignee: Jingsong Lee
>Priority: Major
> Fix For: 1.11.0
>
>
> I created a csv table in sql client, and tried to insert some data into this
> table.
> The first insert into success, but the second one failed with exception:
> {code:java}
> // Caused by: java.io.IOException: File or directory /.../xxx.csv already
> exists. Existing files and directories are not overwritten in NO_OVERWRITE
> mode. Use OVERWRITE mode to overwrite existing files and directories.at
> org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileSystem.java:817)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15066) Cannot run multiple `insert into csvTable values ()`
[
https://issues.apache.org/jira/browse/FLINK-15066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014210#comment-17014210
]
Caizhi Weng commented on FLINK-15066:
-
>From the perspective of a new user, especially a user from the traditional
>database, insert overwrite is a really strange behavior. It would be nice if
>the appending mode is made as the default behavior.
> Cannot run multiple `insert into csvTable values ()`
>
>
> Key: FLINK-15066
> URL: https://issues.apache.org/jira/browse/FLINK-15066
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Reporter: Kurt Young
>Assignee: Jingsong Lee
>Priority: Major
> Fix For: 1.11.0
>
>
> I created a csv table in sql client, and tried to insert some data into this
> table.
> The first insert into success, but the second one failed with exception:
> {code:java}
> // Caused by: java.io.IOException: File or directory /.../xxx.csv already
> exists. Existing files and directories are not overwritten in NO_OVERWRITE
> mode. Use OVERWRITE mode to overwrite existing files and directories.at
> org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileSystem.java:817)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15355) Nightly streaming file sink fails with unshaded hadoop
[
https://issues.apache.org/jira/browse/FLINK-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014209#comment-17014209
]
Till Rohrmann edited comment on FLINK-15355 at 1/13/20 10:40 AM:
-
Another instance of the problem:
https://api.travis-ci.org/v3/job/635957245/log.txt
Full build matrix: https://travis-ci.org/apache/flink/builds/635957218
was (Author: till.rohrmann):
Another instance of the problem:
https://api.travis-ci.org/v3/job/635957245/log.txt
> Nightly streaming file sink fails with unshaded hadoop
> --
>
> Key: FLINK-15355
> URL: https://issues.apache.org/jira/browse/FLINK-15355
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Arvid Heise
>Assignee: Arvid Heise
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {code:java}
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.client.JobSubmissionException: Failed to submit
> JobGraph.
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
> at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:664)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.client.JobSubmissionException: Failed to submit
> JobGraph.
> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:199)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1751)
> at
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:94)
> at
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:63)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1628)
> at StreamingFileSinkProgram.main(StreamingFileSinkProgram.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
> ... 11 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.client.JobSubmissionException: Failed to submit
> JobGraph.
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1746)
> ... 20 more
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to
> submit JobGraph.
> at
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$7(RestClusterClient.java:326)
> at
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
> at
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:274)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFu
[jira] [Closed] (FLINK-15570) Support inserting into non-empty csv tables
[
https://issues.apache.org/jira/browse/FLINK-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caizhi Weng closed FLINK-15570.
---
Resolution: Duplicate
> Support inserting into non-empty csv tables
> ---
>
> Key: FLINK-15570
> URL: https://issues.apache.org/jira/browse/FLINK-15570
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Reporter: Caizhi Weng
>Priority: Major
> Labels: Usability
>
> Currently we only support inserting into empty csv tables. It would be nice
> for a new user, especially for users from traditional database, to try out
> Flink if we support inserting into non-empty csv tables.
> In other words, it would be nice if the following SQL produces a valid result:
> {code:java}
> CREATE TABLE myTable(
> a INT,
> b DOUBLE
> ) WITH (
> 'connector.type' = 'filesystem',
> 'connector.path' = an existing csv file,
> 'format.type' = 'csv',
> 'format.derive-schema' = 'true'
> );
> SELECT * FROM myTable; // produces 3 rows
> INSERT INTO myTable VALUES (4, 4.4);
> SELECT * FROM myTable; // produces 4 rows{code}
> We currently only have two write modes, namely NO_OVERWRITE and OVERWRITE. In
> NO_OVERWRITE mode we can only insert into empty csv tables, while in
> OVERWRITE mode inserting into a csv table will wipe all existing data in it
> (in the above example, the last SELECT will produce only 1 instead of 4 rows)
> which is really strange for a mere INSERT operation.
> We need to add a new APPEND write mode, or change the behavior for OVERWRITE
> mode to something like appending to files. Supporting specifying write mode
> in the WITH clause is also needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15355) Nightly streaming file sink fails with unshaded hadoop
[
https://issues.apache.org/jira/browse/FLINK-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014209#comment-17014209
]
Till Rohrmann commented on FLINK-15355:
---
Another instance of the problem:
https://api.travis-ci.org/v3/job/635957245/log.txt
> Nightly streaming file sink fails with unshaded hadoop
> --
>
> Key: FLINK-15355
> URL: https://issues.apache.org/jira/browse/FLINK-15355
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Arvid Heise
>Assignee: Arvid Heise
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {code:java}
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.client.JobSubmissionException: Failed to submit
> JobGraph.
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
> at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:664)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.client.JobSubmissionException: Failed to submit
> JobGraph.
> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:199)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1751)
> at
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:94)
> at
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:63)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1628)
> at StreamingFileSinkProgram.main(StreamingFileSinkProgram.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
> ... 11 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.client.JobSubmissionException: Failed to submit
> JobGraph.
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1746)
> ... 20 more
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to
> submit JobGraph.
> at
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$7(RestClusterClient.java:326)
> at
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
> at
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:274)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561)
> at
> java.util.concurrent.CompletableFuture$Un
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365732020
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##
@@ -540,6 +557,65 @@ public void testSlotReleaseAtomicallyReleasesExecution()
throws Exception {
});
}
+ /**
+* Tests that producedPartitions are registered synchronously under an
asynchronous interface.
+*/
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws
Exception {
+ ExecutionJobVertex jobVertex = getExecutionVertex(
+ new JobVertexID(),
+ new DirectScheduledExecutorService(),
+ EAGER);
+ IntermediateResult result =
+ new IntermediateResult(new
IntermediateDataSetID(), jobVertex, 1, ResultPartitionType.PIPELINED);
+ ExecutionVertex vertex =
+ new ExecutionVertex(jobVertex, 0, new
IntermediateResult[]{result}, Time.minutes(1));
+ ExecutionEdge edge = createMockExecutionEdge(1);
+ result.getPartitions()[0].addConsumerGroup();
+ result.getPartitions()[0].addConsumer(edge, 0);
Review comment:
Are these lines necessary?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-15387) Expose missing RocksDB properties out via RocksDBNativeMetricOptions
[ https://issues.apache.org/jira/browse/FLINK-15387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Yao updated FLINK-15387: - Fix Version/s: (was: 1.11.0) > Expose missing RocksDB properties out via RocksDBNativeMetricOptions > > > Key: FLINK-15387 > URL: https://issues.apache.org/jira/browse/FLINK-15387 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.10.0, 1.11.0 >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When we implements FLINK-15368, we need to expose block cache related metrics > of RocksDB out by adding more available options to current > RocksDBNativeMetricOptions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15522) Misleading root cause exception when cancelling the job
[
https://issues.apache.org/jira/browse/FLINK-15522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-15522:
-
Fix Version/s: (was: 1.11.0)
> Misleading root cause exception when cancelling the job
> ---
>
> Key: FLINK-15522
> URL: https://issues.apache.org/jira/browse/FLINK-15522
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Runtime / Coordination
>Affects Versions: 1.9.1, 1.10.0, 1.11.0
>Reporter: Till Rohrmann
>Assignee: Zhu Zhu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.9.2, 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When cancelling a Flink job, the following stack trace gets displayed
> {code}
> The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: Job failed.
> (JobID: d0e8c2026709385166bcc0253c30742e)
> at
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:262)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338)
> at
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:60)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1507)
> at
> org.apache.flink.streaming.examples.statemachine.StateMachineExample.main(StateMachineExample.java:142)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
> at
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
> at
> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
> Caused by: org.apache.flink.runtime.client.JobCancellationException: Job was
> cancelled.
> at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:148)
> at
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:259)
> ... 18 more
> Caused by:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> No pooled slot available and request to ResourceManager for new slot failed
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.slotRequestToResourceManagerFailed(SlotPoolImpl.java:357)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.lambda$requestSlotFromResourceManager$1(SlotPoolImpl.java:345)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrEls
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365721185
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/ShuffleMaster.java
##
@@ -31,6 +31,10 @@
/**
* Asynchronously register a partition and its producer with the
shuffle service.
*
+* IMPORTANT: Please implement the asynchronous interface in a
synchronous way.
+* The default scheduler only supports synchronous registration for now.
+* Please refer to {@link
NettyShuffleMaster#registerPartitionWithProducer} for example.
Review comment:
I'd prefer to state that "one must return a completed future at the moment".
It is clear enough so the user do need to refer to NettyShuffleMaster for the
implementation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365720350
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphTestUtils.java
##
@@ -431,6 +433,18 @@ public static ExecutionJobVertex
getExecutionVertex(JobVertexID id) throws Excep
return getExecutionVertex(id, new
DirectScheduledExecutorService());
}
+ public static ExecutionEdge createMockExecutionEdge(int maxParallelism)
{
Review comment:
We are trying to get rid of Mockito in flink tests.
If we are touching this part, I'd prefer to introduce a mock ExecutionEdge
class for this purpose.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365723610
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java
##
@@ -605,6 +606,26 @@ public void setInitialState(@Nullable
JobManagerTaskRestore taskRestore) {
});
}
+ /**
+* Register producedPartitions to {@link ShuffleMaster}
+*
+* HACK: Please notice that this method simulates asynchronous
registration in a synchronous way
+* by making sure the returned {@link CompletableFuture} from {@link
ShuffleMaster#registerPartitionWithProducer}
+* is done immediately.
+*
+* {@link Execution#producedPartitions} are registered through an
asynchronous interface
+* {@link ShuffleMaster#registerPartitionWithProducer} to {@link
ShuffleMaster}, however they are not always
+* accessed through callbacks. So, it is possible that {@link
Execution#producedPartitions}
+* have not been available yet when accessed (in {@link
Execution#deploy} for example).
+*
+* Since the only implementation of {@link ShuffleMaster} is {@link
NettyShuffleMaster},
+* which indeed registers producedPartition in a synchronous way, hence
this method enforces
+* synchronous registration under an asynchronous interface for now.
+*
+* If asynchronous registration is needed in the future, use
callbacks to access {@link Execution#producedPartitions}.
Review comment:
This is a temporary fix. So we'd better have a TODO here as a reminder for
follow up fixes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10832: [FLINK-14163][runtime]Enforce synchronous registration of Execution#producedPartitions
zhuzhurk commented on a change in pull request #10832:
[FLINK-14163][runtime]Enforce synchronous registration of
Execution#producedPartitions
URL: https://github.com/apache/flink/pull/10832#discussion_r365729537
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionTest.java
##
@@ -540,6 +557,65 @@ public void testSlotReleaseAtomicallyReleasesExecution()
throws Exception {
});
}
+ /**
+* Tests that producedPartitions are registered synchronously under an
asynchronous interface.
+*/
+ @Test(expected = IllegalStateException.class)
+ public void testSynchronousRegisterProducedPartitions() throws
Exception {
+ ExecutionJobVertex jobVertex = getExecutionVertex(
+ new JobVertexID(),
+ new DirectScheduledExecutorService(),
+ EAGER);
+ IntermediateResult result =
+ new IntermediateResult(new
IntermediateDataSetID(), jobVertex, 1, ResultPartitionType.PIPELINED);
+ ExecutionVertex vertex =
+ new ExecutionVertex(jobVertex, 0, new
IntermediateResult[]{result}, Time.minutes(1));
+ ExecutionEdge edge = createMockExecutionEdge(1);
+ result.getPartitions()[0].addConsumerGroup();
+ result.getPartitions()[0].addConsumer(edge, 0);
+ TaskManagerLocation location =
+ new TaskManagerLocation(ResourceID.generate(),
InetAddress.getLoopbackAddress(), 1);
+ ExecutionAttemptID attemptID = new ExecutionAttemptID();
+ Execution.registerProducedPartitions(vertex, location,
attemptID, EAGER.allowLazyDeployment());
+ }
+
+ @Nonnull
+ private ExecutionJobVertex getExecutionVertex(
Review comment:
Could we extend ExecutionGraphTestUtils#getExecutionVertex(...) to customize
shuffleMaster and reuse it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] twalthr commented on issue #10838: [hotfix][table-common] Simplify formatted exceptions
twalthr commented on issue #10838: [hotfix][table-common] Simplify formatted exceptions URL: https://github.com/apache/flink/pull/10838#issuecomment-573595958 I'm fine with a builder approach. This also allows us to include error codes and other properties (such as severity) in the future. I will update the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15417) Remove the docker volume or mount when starting Mesos e2e cluster
[ https://issues.apache.org/jira/browse/FLINK-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Yao updated FLINK-15417: - Issue Type: Bug (was: Test) > Remove the docker volume or mount when starting Mesos e2e cluster > - > > Key: FLINK-15417 > URL: https://issues.apache.org/jira/browse/FLINK-15417 > Project: Flink > Issue Type: Bug >Reporter: Yangze Guo >Assignee: Yangze Guo >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > As discussed > [here|https://github.com/apache/flink/pull/10695#discussion_r361574394], > there is a potential risk of permission problems when cleanup logs and > output. We could found another way to let containers get the input and output > file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15570) Support inserting into non-empty csv tables
[
https://issues.apache.org/jira/browse/FLINK-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014190#comment-17014190
]
Kurt Young commented on FLINK-15570:
This is duplicated with FLINK-15066?
> Support inserting into non-empty csv tables
> ---
>
> Key: FLINK-15570
> URL: https://issues.apache.org/jira/browse/FLINK-15570
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Reporter: Caizhi Weng
>Priority: Major
> Labels: Usability
>
> Currently we only support inserting into empty csv tables. It would be nice
> for a new user, especially for users from traditional database, to try out
> Flink if we support inserting into non-empty csv tables.
> In other words, it would be nice if the following SQL produces a valid result:
> {code:java}
> CREATE TABLE myTable(
> a INT,
> b DOUBLE
> ) WITH (
> 'connector.type' = 'filesystem',
> 'connector.path' = an existing csv file,
> 'format.type' = 'csv',
> 'format.derive-schema' = 'true'
> );
> SELECT * FROM myTable; // produces 3 rows
> INSERT INTO myTable VALUES (4, 4.4);
> SELECT * FROM myTable; // produces 4 rows{code}
> We currently only have two write modes, namely NO_OVERWRITE and OVERWRITE. In
> NO_OVERWRITE mode we can only insert into empty csv tables, while in
> OVERWRITE mode inserting into a csv table will wipe all existing data in it
> (in the above example, the last SELECT will produce only 1 instead of 4 rows)
> which is really strange for a mere INSERT operation.
> We need to add a new APPEND write mode, or change the behavior for OVERWRITE
> mode to something like appending to files. Supporting specifying write mode
> in the WITH clause is also needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-15565) Incompatible types of expression and result type thrown in codegen
[ https://issues.apache.org/jira/browse/FLINK-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-15565: -- Assignee: Jingsong Lee > Incompatible types of expression and result type thrown in codegen > -- > > Key: FLINK-15565 > URL: https://issues.apache.org/jira/browse/FLINK-15565 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Assignee: Jingsong Lee >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > *The sql is:* > CREATE TABLE `over10k` ( > t tinyint, > si smallint, > i int, > b bigint, > f float, > d double, > bo boolean, > s varchar, > ts timestamp, > deci decimal(4,2), > bin varchar > ) WITH ( > > 'connector.path'='/daily_regression_batch_hive_1.10/test_window_with_specific_behavior/sources/over10k.csv', > 'format.empty-column-as-null'='true', > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > 'format.type'='csv' > ); > select s, rank() over (partition by s order by si), sum(b) over (partition > by s order by si) from over10k limit 100; > *The data is :* > 109|277|65620|4294967305|97.25|7.80|true|nick quirinius|2013-03-01 > 09:11:58.703226|27.72|undecided > 93|263|65725|4294967341|6.06|4.12|false|calvin king|2013-03-01 > 09:11:58.703299|32.44|values clariffication > 108|383|65629|4294967510|39.55|47.67|false|jessica zipper|2013-03-01 > 09:11:58.703133|74.23|nap time > 89|463|65537|4294967493|64.82|13.79|true|ethan white|2013-03-01 > 09:11:58.703243|89.52|nap time > 88|372|65645|4294967358|34.48|11.18|true|quinn thompson|2013-03-01 > 09:11:58.703168|84.86|forestry > 123|432|65626|4294967435|2.39|16.49|true|david white|2013-03-01 > 09:11:58.703136|61.24|joggying > 57|486|65551|4294967397|36.11|9.88|true|katie xylophone|2013-03-01 > 09:11:58.703142|57.10|zync studies > 59|343|65787|4294967312|66.89|6.54|true|mike laertes|2013-03-01 > 09:11:58.703209|27.56|xylophone band > 74|267|65671|4294967409|21.14|14.64|true|priscilla miller|2013-03-01 > 09:11:58.703197|89.06|undecided > 25|336|65587|4294967336|71.01|14.90|true|tom ichabod|2013-03-01 > 09:11:58.703127|74.32|zync studies > 48|346|65712|4294967315|45.01|16.08|true|zach brown|2013-03-01 > 09:11:58.703108|21.68|zync studies > 84|385|65776|4294967452|35.80|32.13|false|xavier zipper|2013-03-01 > 09:11:58.703311|99.46|education > 58|389|65766|4294967416|95.55|20.62|false|sarah miller|2013-03-01 > 09:11:58.703215|70.92|history > 22|403|65565|4294967381|99.65|35.42|false|yuri johnson|2013-03-01 > 09:11:58.703154|94.47|geology > 55|428|65733|4294967535|99.54|5.35|false|jessica king|2013-03-01 > 09:11:58.703233|30.30|forestry > 117|410|65706|4294967391|50.15|0.21|false|quinn johnson|2013-03-01 > 09:11:58.703248|65.99|yard duty > 95|423|65573|4294967378|47.59|17.37|true|alice robinson|2013-03-01 > 09:11:58.703133|54.57|linguistics > 87|332|65748|4294967320|19.83|41.67|false|fred ellison|2013-03-01 > 09:11:58.703289|79.02|mathematics > 114|263|65674|4294967405|84.44|33.18|true|victor van buren|2013-03-01 > 09:11:58.703092|63.74|linguistics > 5|369|65780|4294967488|92.02|38.59|true|zach polk|2013-03-01 > 09:11:58.703271|67.29|yard duty > -3|430|65667|4294967469|65.50|40.46|true|yuri xylophone|2013-03-01 > 09:11:58.703258|30.94|american history > 120|264|65769|4294967486|89.97|41.18|false|xavier hernandez|2013-03-01 > 09:11:58.703140|66.89|philosophy > 107|317|65634|4294967488|5.68|18.89|false|priscilla ichabod|2013-03-01 > 09:11:58.703196|39.42|joggying > 29|386|65723|4294967328|71.48|6.13|false|ulysses ichabod|2013-03-01 > 09:11:58.703215|86.65|xylophone band > 22|434|65768|4294967543|44.25|27.56|false|tom polk|2013-03-01 > 09:11:58.703306|12.30|kindergarten > -1|274|65755|4294967300|22.01|35.52|false|oscar king|2013-03-01 > 09:11:58.703141|33.35|chemistry > 6|365|65603|4294967522|18.51|5.60|false|gabriella king|2013-03-01 > 09:11:58.703104|34.20|geology > 97|414|65757|4294967325|31.82|22.37|false|rachel nixon|2013-03-01 > 09:11:58.703127|61.00|nap time > 72|448|65538|4294967524|80.09|7.73|true|luke brown|2013-03-01 > 09:11:58.703090|95.81|american history > 51|280|65589|4294967486|57.46|23.35|false|zach xylophone|2013-03-01 > 09:11:58.703299|11.54|education > 12|447|65583|4294967389|0.98|29.79|true|yuri polk|2013-03-01 > 09:11:58.703305|1.89|wind surfing > -1|360|65539|4294967464|4.08|39.51|false|oscar davidson|2013-03-01 > 09:11:58.703144|59.47|nap time > 0|380|65569|4294967425|0.94|28.93|false|sarah robinson|2013-03-01 > 09:11:58.703176|88.81|xylophone band > 66|478|65669|4294967339|23.66|38.3
[jira] [Commented] (FLINK-15565) Incompatible types of expression and result type thrown in codegen
[ https://issues.apache.org/jira/browse/FLINK-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014186#comment-17014186 ] Kurt Young commented on FLINK-15565: [~lzljs3620320] I've assigned to you. > Incompatible types of expression and result type thrown in codegen > -- > > Key: FLINK-15565 > URL: https://issues.apache.org/jira/browse/FLINK-15565 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Assignee: Jingsong Lee >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > *The sql is:* > CREATE TABLE `over10k` ( > t tinyint, > si smallint, > i int, > b bigint, > f float, > d double, > bo boolean, > s varchar, > ts timestamp, > deci decimal(4,2), > bin varchar > ) WITH ( > > 'connector.path'='/daily_regression_batch_hive_1.10/test_window_with_specific_behavior/sources/over10k.csv', > 'format.empty-column-as-null'='true', > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > 'format.type'='csv' > ); > select s, rank() over (partition by s order by si), sum(b) over (partition > by s order by si) from over10k limit 100; > *The data is :* > 109|277|65620|4294967305|97.25|7.80|true|nick quirinius|2013-03-01 > 09:11:58.703226|27.72|undecided > 93|263|65725|4294967341|6.06|4.12|false|calvin king|2013-03-01 > 09:11:58.703299|32.44|values clariffication > 108|383|65629|4294967510|39.55|47.67|false|jessica zipper|2013-03-01 > 09:11:58.703133|74.23|nap time > 89|463|65537|4294967493|64.82|13.79|true|ethan white|2013-03-01 > 09:11:58.703243|89.52|nap time > 88|372|65645|4294967358|34.48|11.18|true|quinn thompson|2013-03-01 > 09:11:58.703168|84.86|forestry > 123|432|65626|4294967435|2.39|16.49|true|david white|2013-03-01 > 09:11:58.703136|61.24|joggying > 57|486|65551|4294967397|36.11|9.88|true|katie xylophone|2013-03-01 > 09:11:58.703142|57.10|zync studies > 59|343|65787|4294967312|66.89|6.54|true|mike laertes|2013-03-01 > 09:11:58.703209|27.56|xylophone band > 74|267|65671|4294967409|21.14|14.64|true|priscilla miller|2013-03-01 > 09:11:58.703197|89.06|undecided > 25|336|65587|4294967336|71.01|14.90|true|tom ichabod|2013-03-01 > 09:11:58.703127|74.32|zync studies > 48|346|65712|4294967315|45.01|16.08|true|zach brown|2013-03-01 > 09:11:58.703108|21.68|zync studies > 84|385|65776|4294967452|35.80|32.13|false|xavier zipper|2013-03-01 > 09:11:58.703311|99.46|education > 58|389|65766|4294967416|95.55|20.62|false|sarah miller|2013-03-01 > 09:11:58.703215|70.92|history > 22|403|65565|4294967381|99.65|35.42|false|yuri johnson|2013-03-01 > 09:11:58.703154|94.47|geology > 55|428|65733|4294967535|99.54|5.35|false|jessica king|2013-03-01 > 09:11:58.703233|30.30|forestry > 117|410|65706|4294967391|50.15|0.21|false|quinn johnson|2013-03-01 > 09:11:58.703248|65.99|yard duty > 95|423|65573|4294967378|47.59|17.37|true|alice robinson|2013-03-01 > 09:11:58.703133|54.57|linguistics > 87|332|65748|4294967320|19.83|41.67|false|fred ellison|2013-03-01 > 09:11:58.703289|79.02|mathematics > 114|263|65674|4294967405|84.44|33.18|true|victor van buren|2013-03-01 > 09:11:58.703092|63.74|linguistics > 5|369|65780|4294967488|92.02|38.59|true|zach polk|2013-03-01 > 09:11:58.703271|67.29|yard duty > -3|430|65667|4294967469|65.50|40.46|true|yuri xylophone|2013-03-01 > 09:11:58.703258|30.94|american history > 120|264|65769|4294967486|89.97|41.18|false|xavier hernandez|2013-03-01 > 09:11:58.703140|66.89|philosophy > 107|317|65634|4294967488|5.68|18.89|false|priscilla ichabod|2013-03-01 > 09:11:58.703196|39.42|joggying > 29|386|65723|4294967328|71.48|6.13|false|ulysses ichabod|2013-03-01 > 09:11:58.703215|86.65|xylophone band > 22|434|65768|4294967543|44.25|27.56|false|tom polk|2013-03-01 > 09:11:58.703306|12.30|kindergarten > -1|274|65755|4294967300|22.01|35.52|false|oscar king|2013-03-01 > 09:11:58.703141|33.35|chemistry > 6|365|65603|4294967522|18.51|5.60|false|gabriella king|2013-03-01 > 09:11:58.703104|34.20|geology > 97|414|65757|4294967325|31.82|22.37|false|rachel nixon|2013-03-01 > 09:11:58.703127|61.00|nap time > 72|448|65538|4294967524|80.09|7.73|true|luke brown|2013-03-01 > 09:11:58.703090|95.81|american history > 51|280|65589|4294967486|57.46|23.35|false|zach xylophone|2013-03-01 > 09:11:58.703299|11.54|education > 12|447|65583|4294967389|0.98|29.79|true|yuri polk|2013-03-01 > 09:11:58.703305|1.89|wind surfing > -1|360|65539|4294967464|4.08|39.51|false|oscar davidson|2013-03-01 > 09:11:58.703144|59.47|nap time > 0|380|65569|4294967425|0.94|28.93|false|sarah robinson|2013-03-01 > 09:11:58.703
[GitHub] [flink] carp84 merged pull request #10805: [FLINK-15529] [docs] Update upgrade compatibility table in operations doc for 1.10.0
carp84 merged pull request #10805: [FLINK-15529] [docs] Update upgrade compatibility table in operations doc for 1.10.0 URL: https://github.com/apache/flink/pull/10805 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15572) Function DDL is not compliant with FLIP-64
[
https://issues.apache.org/jira/browse/FLINK-15572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014182#comment-17014182
]
Timo Walther commented on FLINK-15572:
--
CC [~phoenixjiangnan]
> Function DDL is not compliant with FLIP-64
> --
>
> Key: FLINK-15572
> URL: https://issues.apache.org/jira/browse/FLINK-15572
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
>
> The Function DDL does not follow the path resolution defined in FLIP-64. It
> always assumes fully qualified paths for catalog and database instead of
> resolving partially defined paths with the help of the current
> catalog/database. See {{createTemporaryView()}} as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14163) Execution#producedPartitions is possibly not assigned when used
[
https://issues.apache.org/jira/browse/FLINK-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014147#comment-17014147
]
Yuan Mei edited comment on FLINK-14163 at 1/13/20 9:59 AM:
---
Synced up with [~zjwang] offline.
In general, I would prefer not to change interfaces unless having to, since
interface changes have more profound impact on users than implementation
changes. Especially in this case, given the future design is not determined
(both Shuffle and Scheduler), changing async -> sycn and then sync -> async
again bring much more pain to users than keeping the asycn interfaces and
enforces a sync implementation underneath.
In the latter case, a user barely needs to do anything if we do want to support
async implementation in the future, while in the first case, every single user
that uses the interface has to make a new release to adjust to the updated
interface.
I do not have strong preferences other than the point to keep the interface. I
think retain a timeout is a good compromise, but may introduce some
inconsistent system behavior. Hence even if we decide to retain a timeout, I
would prefer to document as `enforce a synchronous implementation under
asynchronous interface`.
was (Author: ym):
Synced up with [~zjwang] offline.
In general, I would prefer not to change interfaces unless having to, since
interface changes have more profound impact on users than implementation
changes. Especially in this case, given the future design is not determined
(both Shuffle and Scheduler), changing async -> sycn and then sync -> async
again bring much more pain to users than keeping the asycn interfaces and
enforces a sync implementation underneath.
In the latter case, a user barely needs to do anything if we do want to support
async implementation in the future, while in the first case, every single user
that uses the interface has to make a new release to adjust to the updated
interface.
I do not have strong preferences other than keeping the interface. I think
retain a timeout is a good compromise, but may introduce some inconsistent
system behavior. Hence even if we decide to retain a timeout, I would prefer to
document as `enforce a synchronous implementation under asynchronous interface`.
> Execution#producedPartitions is possibly not assigned when used
> ---
>
> Key: FLINK-14163
> URL: https://issues.apache.org/jira/browse/FLINK-14163
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhu Zhu
>Assignee: Yuan Mei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently {{Execution#producedPartitions}} is assigned after the partitions
> have completed the registration to shuffle master in
> {{Execution#registerProducedPartitions(...)}}.
> The partition registration is an async interface
> ({{ShuffleMaster#registerPartitionWithProducer(...)}}), so
> {{Execution#producedPartitions}} is possible[1] not set when used.
> Usages includes:
> 1. deploying this task, so that the task may be deployed without its result
> partitions assigned, and the job would hang. (DefaultScheduler issue only,
> since legacy scheduler handled this case)
> 2. generating input descriptors for downstream tasks:
> 3. retrieve {{ResultPartitionID}} for partition releasing:
> [1] If a user uses Flink default shuffle master {{NettyShuffleMaster}}, it is
> not problematic at the moment since it returns a completed future on
> registration, so that it would be a synchronized process. However, if users
> implement their own shuffle service in which the
> {{ShuffleMaster#registerPartitionWithProducer}} returns an pending future, it
> can be a problem. This is possible since customizable shuffle service is open
> to users since 1.9 (via config "shuffle-service-factory.class").
> To avoid issues to happen, we may either
> 1. fix all the usages of {{Execution#producedPartitions}} regarding the async
> assigning, or
> 2. change {{ShuffleMaster#registerPartitionWithProducer(...)}} to a sync
> interface
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14163) Execution#producedPartitions is possibly not assigned when used
[
https://issues.apache.org/jira/browse/FLINK-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014147#comment-17014147
]
Yuan Mei edited comment on FLINK-14163 at 1/13/20 9:58 AM:
---
Synced up with [~zjwang] offline.
In general, I would prefer not to change interfaces unless having to, since
interface changes have more profound impact on users than implementation
changes. Especially in this case, given the future design is not determined
(both Shuffle and Scheduler), changing async -> sycn and then sync -> async
again bring much more pain to users than keeping the asycn interfaces and
enforces a sync implementation underneath.
In the latter case, a user barely needs to do anything if we do want to support
async implementation in the future, while in the first case, every single user
that uses the interface has to make a new release to adjust to the updated
interface.
I do not have strong preferences other than keeping the interface. I think
retain a timeout is a good compromise, but may introduce some inconsistent
system behavior. Hence even if we decide to retain a timeout, I would prefer to
document as `enforce a synchronous implementation under asynchronous interface`.
was (Author: ym):
Synced up with [~zjwang] offline.
In general, I would prefer not to change interfaces unless having to, since
interface changes have more profound impact on users than implementation
changes. Especially in this case, given the future design is not determined
(both Shuffle and Scheduler), changing async -> sycn and then sync -> async
again bring much more pain to users than keeping the asycn interfaces and
enforces a sync implementation underneath.
In the latter case, a user barely needs to do anything if we do want to support
async implementation in the future, while in the first case, every single user
that uses the interface has to make a new release to adjust to the updated
interface.
I do not have strong preferences other than the interface. I think retain a
timeout is a good compromise, but may introduce some inconsistent system
behavior. Hence even if we decide to retain a timeout, I would prefer to
document as `enforce a synchronous implementation under asynchronous interface`.
> Execution#producedPartitions is possibly not assigned when used
> ---
>
> Key: FLINK-14163
> URL: https://issues.apache.org/jira/browse/FLINK-14163
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhu Zhu
>Assignee: Yuan Mei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently {{Execution#producedPartitions}} is assigned after the partitions
> have completed the registration to shuffle master in
> {{Execution#registerProducedPartitions(...)}}.
> The partition registration is an async interface
> ({{ShuffleMaster#registerPartitionWithProducer(...)}}), so
> {{Execution#producedPartitions}} is possible[1] not set when used.
> Usages includes:
> 1. deploying this task, so that the task may be deployed without its result
> partitions assigned, and the job would hang. (DefaultScheduler issue only,
> since legacy scheduler handled this case)
> 2. generating input descriptors for downstream tasks:
> 3. retrieve {{ResultPartitionID}} for partition releasing:
> [1] If a user uses Flink default shuffle master {{NettyShuffleMaster}}, it is
> not problematic at the moment since it returns a completed future on
> registration, so that it would be a synchronized process. However, if users
> implement their own shuffle service in which the
> {{ShuffleMaster#registerPartitionWithProducer}} returns an pending future, it
> can be a problem. This is possible since customizable shuffle service is open
> to users since 1.9 (via config "shuffle-service-factory.class").
> To avoid issues to happen, we may either
> 1. fix all the usages of {{Execution#producedPartitions}} regarding the async
> assigning, or
> 2. change {{ShuffleMaster#registerPartitionWithProducer(...)}} to a sync
> interface
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15572) Function DDL is not compliant with FLIP-64
Timo Walther created FLINK-15572:
Summary: Function DDL is not compliant with FLIP-64
Key: FLINK-15572
URL: https://issues.apache.org/jira/browse/FLINK-15572
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Reporter: Timo Walther
The Function DDL does not follow the path resolution defined in FLIP-64. It
always assumes fully qualified paths for catalog and database instead of
resolving partially defined paths with the help of the current
catalog/database. See {{createTemporaryView()}} as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-15488) Cannot start a taskmanger if using logback
[
https://issues.apache.org/jira/browse/FLINK-15488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao closed FLINK-15488.
Resolution: Fixed
1.10: 6c886ade7c02840423552025f5d5da535ae40d02
master: d76f2166396e3758c08851d024e2c23c8a297062
> Cannot start a taskmanger if using logback
> --
>
> Key: FLINK-15488
> URL: https://issues.apache.org/jira/browse/FLINK-15488
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Deployment / Scripts
>Affects Versions: 1.10.0
>Reporter: Dawid Wysakowicz
>Assignee: Yangze Guo
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When using logback it is not possible to start the taskmanager using
> {{taskamanger.sh}} scripts. The same problem (probably) occurs when using
> slf4j that logs into the console.
> The problem is that when calculating memory configuration with
> {{BashJavaUtils}} class the result is returned through standard output. If
> something is logged into the console it may result in undefined behavior such
> as e.g.
> {code}
> Error: Could not find or load main class 13:51:23.961
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] GJL closed pull request #10804: [FLINK-15488] Obtain the JVM and TM param correctly
GJL closed pull request #10804: [FLINK-15488] Obtain the JVM and TM param correctly URL: https://github.com/apache/flink/pull/10804 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] jrthe42 closed pull request #6666: [FLINK-10290] [table] Fix conversion error in StreamScan and BatchScan
jrthe42 closed pull request #: [FLINK-10290] [table] Fix conversion error in StreamScan and BatchScan URL: https://github.com/apache/flink/pull/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] jrthe42 closed pull request #6847: [FLINK-10553] [sql] Unified sink and source table name in SQL statement
jrthe42 closed pull request #6847: [FLINK-10553] [sql] Unified sink and source table name in SQL statement URL: https://github.com/apache/flink/pull/6847 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] jrthe42 closed pull request #6301: [FLINK-9794] [jdbc] JDBCOutputFormat does not consider idle connection and multithreads synchronization
jrthe42 closed pull request #6301: [FLINK-9794] [jdbc] JDBCOutputFormat does not consider idle connection and multithreads synchronization URL: https://github.com/apache/flink/pull/6301 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10674: [FLINK-15220][Connector/Kafka][Table] Add startFromTimestamp in KafkaTableSource
flinkbot edited a comment on issue #10674: [FLINK-15220][Connector/Kafka][Table] Add startFromTimestamp in KafkaTableSource URL: https://github.com/apache/flink/pull/10674#issuecomment-568687684 ## CI report: * f59106d121f9dd0fc40b23640ec2ad8a663d6020 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142201948) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3880) * a69cf4921d9a947d437a77c173ae979998ad14a1 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142214808) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3889) * a8681f8b91fd819cbee65212479e1263fb88bf64 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144107176) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4283) * 7a75e1c85bc511af3be0eaf977288af816a12b78 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
flinkbot edited a comment on issue #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests URL: https://github.com/apache/flink/pull/10665#issuecomment-568440738 ## CI report: * 5a0d5d3d499347ca216e19175ff5f066a6d9b458 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142099952) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3861) * 255d89be8069b36be2b980ea6dba4798568160bb Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143507134) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4176) * 9168102e928bacaa8026407f77a33b80a8ddeae4 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143514708) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4179) * 1465ba2465b0fba3c91377735481db17abb1c21f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143667456) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4203) * 76f3dadc5062279dda9df063901a548315a8b8da Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143714109) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4219) * 677c4fcd1b82c766f79e296883f8464daafb6a44 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144091754) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4278) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-15571) Create a Redis Streams Connector for Flink
Tugdual Grall created FLINK-15571: - Summary: Create a Redis Streams Connector for Flink Key: FLINK-15571 URL: https://issues.apache.org/jira/browse/FLINK-15571 Project: Flink Issue Type: New Feature Components: Connectors / Common Reporter: Tugdual Grall Redis has a "log data structure" called Redis Streams, it would be nice to integrate Redis Streams and Apache Flink as: * Source * Sink See Redis Streams introduction: [https://redis.io/topics/streams-intro] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15570) Support inserting into non-empty csv tables
[
https://issues.apache.org/jira/browse/FLINK-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caizhi Weng updated FLINK-15570:
Priority: Major (was: Minor)
> Support inserting into non-empty csv tables
> ---
>
> Key: FLINK-15570
> URL: https://issues.apache.org/jira/browse/FLINK-15570
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Reporter: Caizhi Weng
>Priority: Major
> Labels: Usability
>
> Currently we only support inserting into empty csv tables. It would be nice
> for a new user, especially for users from traditional database, to try out
> Flink if we support inserting into non-empty csv tables.
> In other words, it would be nice if the following SQL produces a valid result:
> {code:java}
> CREATE TABLE myTable(
> a INT,
> b DOUBLE
> ) WITH (
> 'connector.type' = 'filesystem',
> 'connector.path' = an existing csv file,
> 'format.type' = 'csv',
> 'format.derive-schema' = 'true'
> );
> SELECT * FROM myTable; // produces 3 rows
> INSERT INTO myTable VALUES (4, 4.4);
> SELECT * FROM myTable; // produces 4 rows{code}
> We currently only have two write modes, namely NO_OVERWRITE and OVERWRITE. In
> NO_OVERWRITE mode we can only insert into empty csv tables, while in
> OVERWRITE mode inserting into a csv table will wipe all existing data in it
> (in the above example, the last SELECT will produce only 1 instead of 4 rows)
> which is really strange for a mere INSERT operation.
> We need to add a new APPEND write mode, or change the behavior for OVERWRITE
> mode to something like appending to files. Supporting specifying write mode
> in the WITH clause is also needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15562) Unable to create walk through project
[
https://issues.apache.org/jira/browse/FLINK-15562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014152#comment-17014152
]
Yangze Guo commented on FLINK-15562:
Hi, [~chesnay], I think it will benefit the user experience if we give a
suggestion of locking the version of the plugin to 2.4 here. Indeed, some user
fix the issue in that way in Chinese community. WDYT?
> Unable to create walk through project
> -
>
> Key: FLINK-15562
> URL: https://issues.apache.org/jira/browse/FLINK-15562
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Jeff Zhang
>Priority: Major
>
> I try to follow the instruction here to create flink walk though project, but
> hit the following errors.
> [https://ci.apache.org/projects/flink/flink-docs-master/getting-started/walkthroughs/table_api.html]
> {code:java}
> ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli)
> on project standalone-pom: archetypeCatalog
> 'https://repository.apache.org/content/repositories/snapshots/' is not
> supported anymore. Please read the plugin documentation for details. -> [Help
> 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-15562) Unable to create walk through project
[
https://issues.apache.org/jira/browse/FLINK-15562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-15562.
Resolution: Not A Problem
The issue is mentioned in the documentation.
> Unable to create walk through project
> -
>
> Key: FLINK-15562
> URL: https://issues.apache.org/jira/browse/FLINK-15562
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Jeff Zhang
>Priority: Major
>
> I try to follow the instruction here to create flink walk though project, but
> hit the following errors.
> [https://ci.apache.org/projects/flink/flink-docs-master/getting-started/walkthroughs/table_api.html]
> {code:java}
> ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli)
> on project standalone-pom: archetypeCatalog
> 'https://repository.apache.org/content/repositories/snapshots/' is not
> supported anymore. Please read the plugin documentation for details. -> [Help
> 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #10841: [FLINK-15565][table-planner-blink] Fix error conversion of TinyInt, SmallInt literals between Flink and Calcite
flinkbot commented on issue #10841: [FLINK-15565][table-planner-blink] Fix error conversion of TinyInt,SmallInt literals between Flink and Calcite URL: https://github.com/apache/flink/pull/10841#issuecomment-573572659 ## CI report: * d3abca3b9c9ca0ad030f7e208843502d14ff13a5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL
flinkbot edited a comment on issue #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL URL: https://github.com/apache/flink/pull/10839#issuecomment-573563076 ## CI report: * c5595ea0c994dcf664cb68ec45a6f500e705a3b7 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144113663) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4285) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10840: [FLINK-15444][runtime] Make the component AbstractInvokable in CheckpointBarrierHandler NonNull
flinkbot edited a comment on issue #10840: [FLINK-15444][runtime] Make the component AbstractInvokable in CheckpointBarrierHandler NonNull URL: https://github.com/apache/flink/pull/10840#issuecomment-573563119 ## CI report: * 61e9e675667e05e44e9c4b5ffd7565f60aec6d62 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144113694) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4286) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10838: [hotfix][table-common] Simplify formatted exceptions
flinkbot edited a comment on issue #10838: [hotfix][table-common] Simplify formatted exceptions URL: https://github.com/apache/flink/pull/10838#issuecomment-573544312 ## CI report: * 6d7cd665ed96239643ccbc81bafacc8fd0e12e80 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144107161) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4282) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW merged pull request #10735: [hotfix][runtime] Fix variable name typos in TaskInformation.java and…
zhijiangW merged pull request #10735: [hotfix][runtime] Fix variable name typos in TaskInformation.java and… URL: https://github.com/apache/flink/pull/10735 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15556) Add a switch for PushProjectIntoTableSourceScanRule
[ https://issues.apache.org/jira/browse/FLINK-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014148#comment-17014148 ] godfrey he commented on FLINK-15556: [~jark], I think if the query only contains one kind of TableSource, the answer is yes. But, if there are more than one kind of TableSource in a query, the answer may be no. > Add a switch for PushProjectIntoTableSourceScanRule > --- > > Key: FLINK-15556 > URL: https://issues.apache.org/jira/browse/FLINK-15556 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: hailong wang >Priority: Minor > Fix For: 1.11.0 > > > In some case, For the project push down, The digest of tableSource maybe > different. For example, If we create a jdbc tablesource, and is used twice > after it for different column, the source can not be reused for the different > digest. In this case, the query io will more precious, So we can add a switch > to turn off project push down, and the reused the source. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14163) Execution#producedPartitions is possibly not assigned when used
[
https://issues.apache.org/jira/browse/FLINK-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014147#comment-17014147
]
Yuan Mei commented on FLINK-14163:
--
Synced up with [~zjwang] offline.
In general, I would prefer not to change interfaces unless having to, since
interface changes have more profound impact on users than implementation
changes. Especially in this case, given the future design is not determined
(both Shuffle and Scheduler), changing async -> sycn and then sync -> async
again bring much more pain to users than keeping the asycn interfaces and
enforces a sync implementation underneath.
In the latter case, a user barely needs to do anything if we do want to support
async implementation in the future, while in the first case, every single user
that uses the interface has to make a new release to adjust to the updated
interface.
I do not have strong preferences other than the interface. I think retain a
timeout is a good compromise, but may introduce some inconsistent system
behavior. Hence even if we decide to retain a timeout, I would prefer to
document as `enforce a synchronous implementation under asynchronous interface`.
> Execution#producedPartitions is possibly not assigned when used
> ---
>
> Key: FLINK-14163
> URL: https://issues.apache.org/jira/browse/FLINK-14163
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhu Zhu
>Assignee: Yuan Mei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently {{Execution#producedPartitions}} is assigned after the partitions
> have completed the registration to shuffle master in
> {{Execution#registerProducedPartitions(...)}}.
> The partition registration is an async interface
> ({{ShuffleMaster#registerPartitionWithProducer(...)}}), so
> {{Execution#producedPartitions}} is possible[1] not set when used.
> Usages includes:
> 1. deploying this task, so that the task may be deployed without its result
> partitions assigned, and the job would hang. (DefaultScheduler issue only,
> since legacy scheduler handled this case)
> 2. generating input descriptors for downstream tasks:
> 3. retrieve {{ResultPartitionID}} for partition releasing:
> [1] If a user uses Flink default shuffle master {{NettyShuffleMaster}}, it is
> not problematic at the moment since it returns a completed future on
> registration, so that it would be a synchronized process. However, if users
> implement their own shuffle service in which the
> {{ShuffleMaster#registerPartitionWithProducer}} returns an pending future, it
> can be a problem. This is possible since customizable shuffle service is open
> to users since 1.9 (via config "shuffle-service-factory.class").
> To avoid issues to happen, we may either
> 1. fix all the usages of {{Execution#producedPartitions}} regarding the async
> assigning, or
> 2. change {{ShuffleMaster#registerPartitionWithProducer(...)}} to a sync
> interface
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14163) Execution#producedPartitions is possibly not assigned when used
[
https://issues.apache.org/jira/browse/FLINK-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014147#comment-17014147
]
Yuan Mei edited comment on FLINK-14163 at 1/13/20 9:20 AM:
---
Synced up with [~zjwang] offline.
In general, I would prefer not to change interfaces unless having to, since
interface changes have more profound impact on users than implementation
changes. Especially in this case, given the future design is not determined
(both Shuffle and Scheduler), changing async -> sycn and then sync -> async
again bring much more pain to users than keeping the asycn interfaces and
enforces a sync implementation underneath.
In the latter case, a user barely needs to do anything if we do want to support
async implementation in the future, while in the first case, every single user
that uses the interface has to make a new release to adjust to the updated
interface.
I do not have strong preferences other than the interface. I think retain a
timeout is a good compromise, but may introduce some inconsistent system
behavior. Hence even if we decide to retain a timeout, I would prefer to
document as `enforce a synchronous implementation under asynchronous interface`.
was (Author: ym):
Synced up with [~zjwang] offline.
In general, I would prefer not to change interfaces unless having to, since
interface changes have more profound impact on users than implementation
changes. Especially in this case, given the future design is not determined
(both Shuffle and Scheduler), changing async -> sycn and then sync -> async
again bring much more pain to users than keeping the asycn interfaces and
enforces a sync implementation underneath.
In the latter case, a user barely needs to do anything if we do want to support
async implementation in the future, while in the first case, every single user
that uses the interface has to make a new release to adjust to the updated
interface.
I do not have strong preferences other than the interface. I think retain a
timeout is a good compromise, but may introduce some inconsistent system
behavior. Hence even if we decide to retain a timeout, I would prefer to
document as `enforce a synchronous implementation under asynchronous interface`.
> Execution#producedPartitions is possibly not assigned when used
> ---
>
> Key: FLINK-14163
> URL: https://issues.apache.org/jira/browse/FLINK-14163
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhu Zhu
>Assignee: Yuan Mei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently {{Execution#producedPartitions}} is assigned after the partitions
> have completed the registration to shuffle master in
> {{Execution#registerProducedPartitions(...)}}.
> The partition registration is an async interface
> ({{ShuffleMaster#registerPartitionWithProducer(...)}}), so
> {{Execution#producedPartitions}} is possible[1] not set when used.
> Usages includes:
> 1. deploying this task, so that the task may be deployed without its result
> partitions assigned, and the job would hang. (DefaultScheduler issue only,
> since legacy scheduler handled this case)
> 2. generating input descriptors for downstream tasks:
> 3. retrieve {{ResultPartitionID}} for partition releasing:
> [1] If a user uses Flink default shuffle master {{NettyShuffleMaster}}, it is
> not problematic at the moment since it returns a completed future on
> registration, so that it would be a synchronized process. However, if users
> implement their own shuffle service in which the
> {{ShuffleMaster#registerPartitionWithProducer}} returns an pending future, it
> can be a problem. This is possible since customizable shuffle service is open
> to users since 1.9 (via config "shuffle-service-factory.class").
> To avoid issues to happen, we may either
> 1. fix all the usages of {{Execution#producedPartitions}} regarding the async
> assigning, or
> 2. change {{ShuffleMaster#registerPartitionWithProducer(...)}} to a sync
> interface
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15570) Support inserting into non-empty csv tables
[
https://issues.apache.org/jira/browse/FLINK-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caizhi Weng updated FLINK-15570:
Description:
Currently we only support inserting into empty csv tables. It would be nice for
a new user, especially for users from traditional database, to try out Flink if
we support inserting into non-empty csv tables.
In other words, it would be nice if the following SQL produces a valid result:
{code:java}
CREATE TABLE myTable(
a INT,
b DOUBLE
) WITH (
'connector.type' = 'filesystem',
'connector.path' = an existing csv file,
'format.type' = 'csv',
'format.derive-schema' = 'true'
);
SELECT * FROM myTable; // produces 3 rows
INSERT INTO myTable VALUES (4, 4.4);
SELECT * FROM myTable; // produces 4 rows{code}
We currently only have two write modes, namely NO_OVERWRITE and OVERWRITE. In
NO_OVERWRITE mode we can only insert into empty csv tables, while in OVERWRITE
mode inserting into a csv table will wipe all existing data in it (in the above
example, the last SELECT will produce only 1 instead of 4 rows) which is really
strange for a mere INSERT operation.
We need to add a new APPEND write mode, or change the behavior for OVERWRITE
mode to something like appending to files. Supporting specifying write mode in
the WITH clause is also needed.
was:
Currently we only support inserting into empty csv tables. It would be nice for
a new user from traditional database to try out Flink if we support inserting
into non-empty csv tables.
In other words, it would be nice if the following SQL produces a valid result:
{code:java}
CREATE TABLE myTable(
a INT,
b DOUBLE
) WITH (
'connector.type' = 'filesystem',
'connector.path' = an existing csv file,
'format.type' = 'csv',
'format.derive-schema' = 'true'
);
SELECT * FROM myTable; // produces 3 rows
INSERT INTO myTable VALUES (4, 4.4);
SELECT * FROM myTable; // produces 4 rows{code}
We currently only have two write modes, namely NO_OVERWRITE and OVERWRITE. In
NO_OVERWRITE mode we can only insert into empty csv tables, while in OVERWRITE
mode inserting into a csv table will wipe all existing data in it (in the above
example, the last SELECT will produce only 1 instead of 4 rows) which is really
strange for a mere INSERT operation.
We need to add a new APPEND write mode, or change the behavior for OVERWRITE
mode to something like appending to files. Supporting specifying write mode in
the WITH clause is also needed.
> Support inserting into non-empty csv tables
> ---
>
> Key: FLINK-15570
> URL: https://issues.apache.org/jira/browse/FLINK-15570
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Reporter: Caizhi Weng
>Priority: Minor
>
> Currently we only support inserting into empty csv tables. It would be nice
> for a new user, especially for users from traditional database, to try out
> Flink if we support inserting into non-empty csv tables.
> In other words, it would be nice if the following SQL produces a valid result:
> {code:java}
> CREATE TABLE myTable(
> a INT,
> b DOUBLE
> ) WITH (
> 'connector.type' = 'filesystem',
> 'connector.path' = an existing csv file,
> 'format.type' = 'csv',
> 'format.derive-schema' = 'true'
> );
> SELECT * FROM myTable; // produces 3 rows
> INSERT INTO myTable VALUES (4, 4.4);
> SELECT * FROM myTable; // produces 4 rows{code}
> We currently only have two write modes, namely NO_OVERWRITE and OVERWRITE. In
> NO_OVERWRITE mode we can only insert into empty csv tables, while in
> OVERWRITE mode inserting into a csv table will wipe all existing data in it
> (in the above example, the last SELECT will produce only 1 instead of 4 rows)
> which is really strange for a mere INSERT operation.
> We need to add a new APPEND write mode, or change the behavior for OVERWRITE
> mode to something like appending to files. Supporting specifying write mode
> in the WITH clause is also needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15570) Support inserting into non-empty csv tables
[
https://issues.apache.org/jira/browse/FLINK-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caizhi Weng updated FLINK-15570:
Labels: Usability (was: )
> Support inserting into non-empty csv tables
> ---
>
> Key: FLINK-15570
> URL: https://issues.apache.org/jira/browse/FLINK-15570
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Reporter: Caizhi Weng
>Priority: Minor
> Labels: Usability
>
> Currently we only support inserting into empty csv tables. It would be nice
> for a new user, especially for users from traditional database, to try out
> Flink if we support inserting into non-empty csv tables.
> In other words, it would be nice if the following SQL produces a valid result:
> {code:java}
> CREATE TABLE myTable(
> a INT,
> b DOUBLE
> ) WITH (
> 'connector.type' = 'filesystem',
> 'connector.path' = an existing csv file,
> 'format.type' = 'csv',
> 'format.derive-schema' = 'true'
> );
> SELECT * FROM myTable; // produces 3 rows
> INSERT INTO myTable VALUES (4, 4.4);
> SELECT * FROM myTable; // produces 4 rows{code}
> We currently only have two write modes, namely NO_OVERWRITE and OVERWRITE. In
> NO_OVERWRITE mode we can only insert into empty csv tables, while in
> OVERWRITE mode inserting into a csv table will wipe all existing data in it
> (in the above example, the last SELECT will produce only 1 instead of 4 rows)
> which is really strange for a mere INSERT operation.
> We need to add a new APPEND write mode, or change the behavior for OVERWRITE
> mode to something like appending to files. Supporting specifying write mode
> in the WITH clause is also needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15559) Missing {{site.baseurl}} in some docs
[
https://issues.apache.org/jira/browse/FLINK-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014144#comment-17014144
]
Benchao Li commented on FLINK-15559:
The reason is following:
Currently {{build_docs.sh}} does not use {{baseurl}}, for a url like
{{/dev/table/streaming/query_configuration.html}}, local url is
{{localhost:4000/dev/table/streaming/query_configuration.html}} which is a
valid url.
When we has a baseurl, for example {{/projects/flink/flink-docs-master}}, then
{{localhost:4000/dev/table/streaming/query_configuration.html}} is a broken
url. In this case, it needs to be
{{localhost:4000/projects/flink/flink-docs-master/dev/table/streaming/query_configuration.html}}.
You can check
[https://ci.apache.org/projects/flink/flink-docs-master/dev/table/functions/systemFunctions.html#type-conversion-functions]
The "See the supported types here"'s like is broken due to lacking {{baseurl}}
> Missing {{site.baseurl}} in some docs
> -
>
> Key: FLINK-15559
> URL: https://issues.apache.org/jira/browse/FLINK-15559
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Benchao Li
>Assignee: Benchao Li
>Priority: Major
>
> For example, in
> * dev/table/functions/systemFunctions.md
> * dev/table/functions/systemFunctions.zh.md
> * dev/table/sourceSinks.md
> * dev/table/sourceSinks.zh.md
> * dev/table/sql/queries.zh.md
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15559) Missing {{site.baseurl}} in some docs
[
https://issues.apache.org/jira/browse/FLINK-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014144#comment-17014144
]
Benchao Li edited comment on FLINK-15559 at 1/13/20 9:19 AM:
-
The reason is following:
Currently {{build_docs.sh}} does not use {{baseurl}}, for a url like
{{/dev/table/streaming/query_configuration.html}}, local url is
{{localhost:4000/dev/table/streaming/query_configuration.html}} which is a
valid url.
When we has a baseurl, for example {{/projects/flink/flink-docs-master}}, then
{{localhost:4000/dev/table/streaming/query_configuration.html}} is a broken
url. In this case, it needs to be
{{localhost:4000/projects/flink/flink-docs-master/dev/table/streaming/query_configuration.html}}.
You can check
[https://ci.apache.org/projects/flink/flink-docs-master/dev/table/functions/systemFunctions.html#type-conversion-functions]
The "See the supported types here"'s link is broken due to lacking {{baseurl}}
was (Author: libenchao):
The reason is following:
Currently {{build_docs.sh}} does not use {{baseurl}}, for a url like
{{/dev/table/streaming/query_configuration.html}}, local url is
{{localhost:4000/dev/table/streaming/query_configuration.html}} which is a
valid url.
When we has a baseurl, for example {{/projects/flink/flink-docs-master}}, then
{{localhost:4000/dev/table/streaming/query_configuration.html}} is a broken
url. In this case, it needs to be
{{localhost:4000/projects/flink/flink-docs-master/dev/table/streaming/query_configuration.html}}.
You can check
[https://ci.apache.org/projects/flink/flink-docs-master/dev/table/functions/systemFunctions.html#type-conversion-functions]
The "See the supported types here"'s like is broken due to lacking {{baseurl}}
> Missing {{site.baseurl}} in some docs
> -
>
> Key: FLINK-15559
> URL: https://issues.apache.org/jira/browse/FLINK-15559
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Benchao Li
>Assignee: Benchao Li
>Priority: Major
>
> For example, in
> * dev/table/functions/systemFunctions.md
> * dev/table/functions/systemFunctions.zh.md
> * dev/table/sourceSinks.md
> * dev/table/sourceSinks.zh.md
> * dev/table/sql/queries.zh.md
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10815: [FLINK-15537][table-planner-blink] Type of keys should be `BinaryRow`…
flinkbot edited a comment on issue #10815: [FLINK-15537][table-planner-blink] Type of keys should be `BinaryRow`… URL: https://github.com/apache/flink/pull/10815#issuecomment-572566376 ## CI report: * 19a4290f709495491fe460037c8c31d106984ea8 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143732723) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4229) * c3ef5ea345a343170806de8112163edb7df31f69 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144110200) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4284) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15570) Support inserting into non-empty csv tables
[
https://issues.apache.org/jira/browse/FLINK-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caizhi Weng updated FLINK-15570:
Description:
Currently we only support inserting into empty csv tables. It would be nice for
a new user from traditional database to try out Flink if we support inserting
into non-empty csv tables.
In other words, it would be nice if the following SQL produces a valid result:
{code:java}
CREATE TABLE myTable(
a INT,
b DOUBLE
) WITH (
'connector.type' = 'filesystem',
'connector.path' = an existing csv file,
'format.type' = 'csv',
'format.derive-schema' = 'true'
);
SELECT * FROM myTable; // produces 3 rows
INSERT INTO myTable VALUES (4, 4.4);
SELECT * FROM myTable; // produces 4 rows{code}
We currently only have two write modes, namely NO_OVERWRITE and OVERWRITE. In
NO_OVERWRITE mode we can only insert into empty csv tables, while in OVERWRITE
mode inserting into a csv table will wipe all existing data in it (in the above
example, the last SELECT will produce only 1 instead of 4 rows) which is really
strange for a mere INSERT operation.
We need to add a new APPEND write mode, or change the behavior for OVERWRITE
mode to something like appending to files. Supporting specifying write mode in
the WITH clause is also needed.
was:
Currently we only support inserting into empty csv tables. It would be nice for
a new user from traditional database to try out Flink if we support inserting
into non-empty csv tables.
In other words, it would be nice if the following SQL produces a valid result:
{code:java}
CREATE TABLE myTable(
a INT,
b DOUBLE
) WITH (
'connector.type' = 'filesystem',
'connector.path' = an existing csv file,
'format.type' = 'csv',
'format.derive-schema' = 'true'
);
SELECT * FROM myTable; // produces 3 rows
INSERT INTO myTable VALUES (4, 4.4);
SELECT * FROM myTable; // produces 4 rows{code}
We currently only have two write modes, namely NO_OVERWRITE and OVERWRITE. In
NO_OVERWRITE mode we can only insert into empty csv tables, while in OVERWRITE
mode inserting into a csv table will wipe all existing data in it (in the above
example, the last SELECT will produce only 1 instead of 4 rows) which is really
strange for a mere INSERT operation. We need to add a new APPEND write mode, or
change the behavior for OVERWRITE mode to something like appending to files.
> Support inserting into non-empty csv tables
> ---
>
> Key: FLINK-15570
> URL: https://issues.apache.org/jira/browse/FLINK-15570
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Reporter: Caizhi Weng
>Priority: Minor
>
> Currently we only support inserting into empty csv tables. It would be nice
> for a new user from traditional database to try out Flink if we support
> inserting into non-empty csv tables.
> In other words, it would be nice if the following SQL produces a valid result:
> {code:java}
> CREATE TABLE myTable(
> a INT,
> b DOUBLE
> ) WITH (
> 'connector.type' = 'filesystem',
> 'connector.path' = an existing csv file,
> 'format.type' = 'csv',
> 'format.derive-schema' = 'true'
> );
> SELECT * FROM myTable; // produces 3 rows
> INSERT INTO myTable VALUES (4, 4.4);
> SELECT * FROM myTable; // produces 4 rows{code}
> We currently only have two write modes, namely NO_OVERWRITE and OVERWRITE. In
> NO_OVERWRITE mode we can only insert into empty csv tables, while in
> OVERWRITE mode inserting into a csv table will wipe all existing data in it
> (in the above example, the last SELECT will produce only 1 instead of 4 rows)
> which is really strange for a mere INSERT operation.
> We need to add a new APPEND write mode, or change the behavior for OVERWRITE
> mode to something like appending to files. Supporting specifying write mode
> in the WITH clause is also needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15570) Support inserting into non-empty csv tables
Caizhi Weng created FLINK-15570:
---
Summary: Support inserting into non-empty csv tables
Key: FLINK-15570
URL: https://issues.apache.org/jira/browse/FLINK-15570
Project: Flink
Issue Type: Improvement
Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
ORC, SequenceFile)
Reporter: Caizhi Weng
Currently we only support inserting into empty csv tables. It would be nice for
a new user from traditional database to try out Flink if we support inserting
into non-empty csv tables.
In other words, it would be nice if the following SQL produces a valid result:
{code:java}
CREATE TABLE myTable(
a INT,
b DOUBLE
) WITH (
'connector.type' = 'filesystem',
'connector.path' = an existing csv file,
'format.type' = 'csv',
'format.derive-schema' = 'true'
);
SELECT * FROM myTable; // produces 3 rows
INSERT INTO myTable VALUES (4, 4.4);
SELECT * FROM myTable; // produces 4 rows{code}
We currently only have two write modes, namely NO_OVERWRITE and OVERWRITE. In
NO_OVERWRITE mode we can only insert into empty csv tables, while in OVERWRITE
mode inserting into a csv table will wipe all existing data in it (in the above
example, the last SELECT will produce only 1 instead of 4 rows) which is really
strange for a mere INSERT operation. We need to add a new APPEND write mode, or
change the behavior for OVERWRITE mode to something like appending to files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14837) Support Function DDL in TableEnvironment
[ https://issues.apache.org/jira/browse/FLINK-14837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014141#comment-17014141 ] Xianxun Ye commented on FLINK-14837: Is there a development plan for this issue? Will it be resolved in 1.10.0 version? > Support Function DDL in TableEnvironment > > > Key: FLINK-14837 > URL: https://issues.apache.org/jira/browse/FLINK-14837 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.10.0 >Reporter: Zhenqiu Huang >Assignee: Zhenqiu Huang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15569) Incorrect sample code in udf document
Jeff Zhang created FLINK-15569: -- Summary: Incorrect sample code in udf document Key: FLINK-15569 URL: https://issues.apache.org/jira/browse/FLINK-15569 Project: Flink Issue Type: Improvement Components: Documentation Affects Versions: 1.10.0 Reporter: Jeff Zhang Attachments: image-2020-01-13-16-59-00-022.png Should use JTuple2 instead of JTuple1 !image-2020-01-13-16-59-00-022.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #10841: [FLINK-15565][table-planner-blink] Fix error conversion of TinyInt, SmallInt literals between Flink and Calcite
flinkbot commented on issue #10841: [FLINK-15565][table-planner-blink] Fix error conversion of TinyInt,SmallInt literals between Flink and Calcite URL: https://github.com/apache/flink/pull/10841#issuecomment-573564277 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit d3abca3b9c9ca0ad030f7e208843502d14ff13a5 (Mon Jan 13 08:57:31 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10840: [FLINK-15444][runtime] Make the component AbstractInvokable in CheckpointBarrierHandler NonNull
flinkbot commented on issue #10840: [FLINK-15444][runtime] Make the component AbstractInvokable in CheckpointBarrierHandler NonNull URL: https://github.com/apache/flink/pull/10840#issuecomment-573563119 ## CI report: * 61e9e675667e05e44e9c4b5ffd7565f60aec6d62 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10837: [FLINK-15541] Fix unstable case FlinkKinesisConsumerTest
flinkbot edited a comment on issue #10837: [FLINK-15541] Fix unstable case FlinkKinesisConsumerTest URL: https://github.com/apache/flink/pull/10837#issuecomment-573544281 ## CI report: * 4d045d632eb72a162e5d56aa52a6819042d2f1aa Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144107145) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4281) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL
flinkbot commented on issue #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL URL: https://github.com/apache/flink/pull/10839#issuecomment-573563076 ## CI report: * c5595ea0c994dcf664cb68ec45a6f500e705a3b7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15565) Incompatible types of expression and result type thrown in codegen
[ https://issues.apache.org/jira/browse/FLINK-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-15565: --- Labels: pull-request-available (was: ) > Incompatible types of expression and result type thrown in codegen > -- > > Key: FLINK-15565 > URL: https://issues.apache.org/jira/browse/FLINK-15565 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0 > > > *The sql is:* > CREATE TABLE `over10k` ( > t tinyint, > si smallint, > i int, > b bigint, > f float, > d double, > bo boolean, > s varchar, > ts timestamp, > deci decimal(4,2), > bin varchar > ) WITH ( > > 'connector.path'='/daily_regression_batch_hive_1.10/test_window_with_specific_behavior/sources/over10k.csv', > 'format.empty-column-as-null'='true', > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > 'format.type'='csv' > ); > select s, rank() over (partition by s order by si), sum(b) over (partition > by s order by si) from over10k limit 100; > *The data is :* > 109|277|65620|4294967305|97.25|7.80|true|nick quirinius|2013-03-01 > 09:11:58.703226|27.72|undecided > 93|263|65725|4294967341|6.06|4.12|false|calvin king|2013-03-01 > 09:11:58.703299|32.44|values clariffication > 108|383|65629|4294967510|39.55|47.67|false|jessica zipper|2013-03-01 > 09:11:58.703133|74.23|nap time > 89|463|65537|4294967493|64.82|13.79|true|ethan white|2013-03-01 > 09:11:58.703243|89.52|nap time > 88|372|65645|4294967358|34.48|11.18|true|quinn thompson|2013-03-01 > 09:11:58.703168|84.86|forestry > 123|432|65626|4294967435|2.39|16.49|true|david white|2013-03-01 > 09:11:58.703136|61.24|joggying > 57|486|65551|4294967397|36.11|9.88|true|katie xylophone|2013-03-01 > 09:11:58.703142|57.10|zync studies > 59|343|65787|4294967312|66.89|6.54|true|mike laertes|2013-03-01 > 09:11:58.703209|27.56|xylophone band > 74|267|65671|4294967409|21.14|14.64|true|priscilla miller|2013-03-01 > 09:11:58.703197|89.06|undecided > 25|336|65587|4294967336|71.01|14.90|true|tom ichabod|2013-03-01 > 09:11:58.703127|74.32|zync studies > 48|346|65712|4294967315|45.01|16.08|true|zach brown|2013-03-01 > 09:11:58.703108|21.68|zync studies > 84|385|65776|4294967452|35.80|32.13|false|xavier zipper|2013-03-01 > 09:11:58.703311|99.46|education > 58|389|65766|4294967416|95.55|20.62|false|sarah miller|2013-03-01 > 09:11:58.703215|70.92|history > 22|403|65565|4294967381|99.65|35.42|false|yuri johnson|2013-03-01 > 09:11:58.703154|94.47|geology > 55|428|65733|4294967535|99.54|5.35|false|jessica king|2013-03-01 > 09:11:58.703233|30.30|forestry > 117|410|65706|4294967391|50.15|0.21|false|quinn johnson|2013-03-01 > 09:11:58.703248|65.99|yard duty > 95|423|65573|4294967378|47.59|17.37|true|alice robinson|2013-03-01 > 09:11:58.703133|54.57|linguistics > 87|332|65748|4294967320|19.83|41.67|false|fred ellison|2013-03-01 > 09:11:58.703289|79.02|mathematics > 114|263|65674|4294967405|84.44|33.18|true|victor van buren|2013-03-01 > 09:11:58.703092|63.74|linguistics > 5|369|65780|4294967488|92.02|38.59|true|zach polk|2013-03-01 > 09:11:58.703271|67.29|yard duty > -3|430|65667|4294967469|65.50|40.46|true|yuri xylophone|2013-03-01 > 09:11:58.703258|30.94|american history > 120|264|65769|4294967486|89.97|41.18|false|xavier hernandez|2013-03-01 > 09:11:58.703140|66.89|philosophy > 107|317|65634|4294967488|5.68|18.89|false|priscilla ichabod|2013-03-01 > 09:11:58.703196|39.42|joggying > 29|386|65723|4294967328|71.48|6.13|false|ulysses ichabod|2013-03-01 > 09:11:58.703215|86.65|xylophone band > 22|434|65768|4294967543|44.25|27.56|false|tom polk|2013-03-01 > 09:11:58.703306|12.30|kindergarten > -1|274|65755|4294967300|22.01|35.52|false|oscar king|2013-03-01 > 09:11:58.703141|33.35|chemistry > 6|365|65603|4294967522|18.51|5.60|false|gabriella king|2013-03-01 > 09:11:58.703104|34.20|geology > 97|414|65757|4294967325|31.82|22.37|false|rachel nixon|2013-03-01 > 09:11:58.703127|61.00|nap time > 72|448|65538|4294967524|80.09|7.73|true|luke brown|2013-03-01 > 09:11:58.703090|95.81|american history > 51|280|65589|4294967486|57.46|23.35|false|zach xylophone|2013-03-01 > 09:11:58.703299|11.54|education > 12|447|65583|4294967389|0.98|29.79|true|yuri polk|2013-03-01 > 09:11:58.703305|1.89|wind surfing > -1|360|65539|4294967464|4.08|39.51|false|oscar davidson|2013-03-01 > 09:11:58.703144|59.47|nap time > 0|380|65569|4294967425|0.94|28.93|false|sarah robinson|2013-03-01 > 09:11:58.703176|88.81|xylophone band > 66|478|65669|4294967339|23.66|38.34|true|yuri carson|2013-03-01 > 09:11:58.703228|64.68|opthamology > 1
[GitHub] [flink] JingsongLi opened a new pull request #10841: [FLINK-15565][table-planner-blink] Fix error conversion of TinyInt, SmallInt literals between Flink and Calcite
JingsongLi opened a new pull request #10841: [FLINK-15565][table-planner-blink] Fix error conversion of TinyInt,SmallInt literals between Flink and Calcite URL: https://github.com/apache/flink/pull/10841 ## What is the purpose of the change In ExpressionConverter, we convert Flink expression to Calcite RexNode. For literal, Flink TinyInt/SmallInt literals are wrongly converted to Calcite Int/Int literals. This lead to incompatible types of expression and result type. ## Brief change log Fix tinyint/smallint/int literals in `ExpressionConverter.visit(ValueLiteralExpression)`. ## Verifying this change - `ExpressionConverter` - `OverWindowITCase.testOverWindowWithUDAG11G` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector:no ## Documentation - Does this pull request introduce a new feature? no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10815: [FLINK-15537][table-planner-blink] Type of keys should be `BinaryRow`…
flinkbot edited a comment on issue #10815: [FLINK-15537][table-planner-blink] Type of keys should be `BinaryRow`… URL: https://github.com/apache/flink/pull/10815#issuecomment-572566376 ## CI report: * 19a4290f709495491fe460037c8c31d106984ea8 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143732723) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4229) * c3ef5ea345a343170806de8112163edb7df31f69 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144110200) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4284) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15565) Incompatible types of expression and result type thrown in codegen
[
https://issues.apache.org/jira/browse/FLINK-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014128#comment-17014128
]
Jingsong Lee commented on FLINK-15565:
--
The bug is:
In {{ExpressionConverter}}, we convert Flink expression to Calcite RexNode. For
literal, Flink TinyInt/SmallInt literals are wrongly converted to Calcite
Int/Int literals. This lead to incompatible types of expression and result type.
> Incompatible types of expression and result type thrown in codegen
> --
>
> Key: FLINK-15565
> URL: https://issues.apache.org/jira/browse/FLINK-15565
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.10.0
>Reporter: xiaojin.wy
>Priority: Major
> Fix For: 1.10.0
>
>
> *The sql is:*
> CREATE TABLE `over10k` (
> t tinyint,
> si smallint,
> i int,
> b bigint,
> f float,
> d double,
> bo boolean,
> s varchar,
> ts timestamp,
> deci decimal(4,2),
> bin varchar
> ) WITH (
>
> 'connector.path'='/daily_regression_batch_hive_1.10/test_window_with_specific_behavior/sources/over10k.csv',
> 'format.empty-column-as-null'='true',
> 'format.field-delimiter'='|',
> 'connector.type'='filesystem',
> 'format.derive-schema'='true',
> 'format.type'='csv'
> );
> select s, rank() over (partition by s order by si), sum(b) over (partition
> by s order by si) from over10k limit 100;
> *The data is :*
> 109|277|65620|4294967305|97.25|7.80|true|nick quirinius|2013-03-01
> 09:11:58.703226|27.72|undecided
> 93|263|65725|4294967341|6.06|4.12|false|calvin king|2013-03-01
> 09:11:58.703299|32.44|values clariffication
> 108|383|65629|4294967510|39.55|47.67|false|jessica zipper|2013-03-01
> 09:11:58.703133|74.23|nap time
> 89|463|65537|4294967493|64.82|13.79|true|ethan white|2013-03-01
> 09:11:58.703243|89.52|nap time
> 88|372|65645|4294967358|34.48|11.18|true|quinn thompson|2013-03-01
> 09:11:58.703168|84.86|forestry
> 123|432|65626|4294967435|2.39|16.49|true|david white|2013-03-01
> 09:11:58.703136|61.24|joggying
> 57|486|65551|4294967397|36.11|9.88|true|katie xylophone|2013-03-01
> 09:11:58.703142|57.10|zync studies
> 59|343|65787|4294967312|66.89|6.54|true|mike laertes|2013-03-01
> 09:11:58.703209|27.56|xylophone band
> 74|267|65671|4294967409|21.14|14.64|true|priscilla miller|2013-03-01
> 09:11:58.703197|89.06|undecided
> 25|336|65587|4294967336|71.01|14.90|true|tom ichabod|2013-03-01
> 09:11:58.703127|74.32|zync studies
> 48|346|65712|4294967315|45.01|16.08|true|zach brown|2013-03-01
> 09:11:58.703108|21.68|zync studies
> 84|385|65776|4294967452|35.80|32.13|false|xavier zipper|2013-03-01
> 09:11:58.703311|99.46|education
> 58|389|65766|4294967416|95.55|20.62|false|sarah miller|2013-03-01
> 09:11:58.703215|70.92|history
> 22|403|65565|4294967381|99.65|35.42|false|yuri johnson|2013-03-01
> 09:11:58.703154|94.47|geology
> 55|428|65733|4294967535|99.54|5.35|false|jessica king|2013-03-01
> 09:11:58.703233|30.30|forestry
> 117|410|65706|4294967391|50.15|0.21|false|quinn johnson|2013-03-01
> 09:11:58.703248|65.99|yard duty
> 95|423|65573|4294967378|47.59|17.37|true|alice robinson|2013-03-01
> 09:11:58.703133|54.57|linguistics
> 87|332|65748|4294967320|19.83|41.67|false|fred ellison|2013-03-01
> 09:11:58.703289|79.02|mathematics
> 114|263|65674|4294967405|84.44|33.18|true|victor van buren|2013-03-01
> 09:11:58.703092|63.74|linguistics
> 5|369|65780|4294967488|92.02|38.59|true|zach polk|2013-03-01
> 09:11:58.703271|67.29|yard duty
> -3|430|65667|4294967469|65.50|40.46|true|yuri xylophone|2013-03-01
> 09:11:58.703258|30.94|american history
> 120|264|65769|4294967486|89.97|41.18|false|xavier hernandez|2013-03-01
> 09:11:58.703140|66.89|philosophy
> 107|317|65634|4294967488|5.68|18.89|false|priscilla ichabod|2013-03-01
> 09:11:58.703196|39.42|joggying
> 29|386|65723|4294967328|71.48|6.13|false|ulysses ichabod|2013-03-01
> 09:11:58.703215|86.65|xylophone band
> 22|434|65768|4294967543|44.25|27.56|false|tom polk|2013-03-01
> 09:11:58.703306|12.30|kindergarten
> -1|274|65755|4294967300|22.01|35.52|false|oscar king|2013-03-01
> 09:11:58.703141|33.35|chemistry
> 6|365|65603|4294967522|18.51|5.60|false|gabriella king|2013-03-01
> 09:11:58.703104|34.20|geology
> 97|414|65757|4294967325|31.82|22.37|false|rachel nixon|2013-03-01
> 09:11:58.703127|61.00|nap time
> 72|448|65538|4294967524|80.09|7.73|true|luke brown|2013-03-01
> 09:11:58.703090|95.81|american history
> 51|280|65589|4294967486|57.46|23.35|false|zach xylophone|2013-03-01
> 09:11:58.703299|11.54|education
> 12|447|65583|4294967389|0.98|29.79|true|yuri polk|2013-03-01
> 09:11:58.703305|1.89|wind surfing
> -1|360|65539|4294967464|4.08|39.51|false|oscar davidson|2013-03-01
> 09:11:58.703144|59.47|nap time
> 0|3
[jira] [Updated] (FLINK-15565) Incompatible types of expression and result type thrown in codegen
[ https://issues.apache.org/jira/browse/FLINK-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-15565: - Priority: Critical (was: Major) > Incompatible types of expression and result type thrown in codegen > -- > > Key: FLINK-15565 > URL: https://issues.apache.org/jira/browse/FLINK-15565 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Priority: Critical > Fix For: 1.10.0 > > > *The sql is:* > CREATE TABLE `over10k` ( > t tinyint, > si smallint, > i int, > b bigint, > f float, > d double, > bo boolean, > s varchar, > ts timestamp, > deci decimal(4,2), > bin varchar > ) WITH ( > > 'connector.path'='/daily_regression_batch_hive_1.10/test_window_with_specific_behavior/sources/over10k.csv', > 'format.empty-column-as-null'='true', > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > 'format.type'='csv' > ); > select s, rank() over (partition by s order by si), sum(b) over (partition > by s order by si) from over10k limit 100; > *The data is :* > 109|277|65620|4294967305|97.25|7.80|true|nick quirinius|2013-03-01 > 09:11:58.703226|27.72|undecided > 93|263|65725|4294967341|6.06|4.12|false|calvin king|2013-03-01 > 09:11:58.703299|32.44|values clariffication > 108|383|65629|4294967510|39.55|47.67|false|jessica zipper|2013-03-01 > 09:11:58.703133|74.23|nap time > 89|463|65537|4294967493|64.82|13.79|true|ethan white|2013-03-01 > 09:11:58.703243|89.52|nap time > 88|372|65645|4294967358|34.48|11.18|true|quinn thompson|2013-03-01 > 09:11:58.703168|84.86|forestry > 123|432|65626|4294967435|2.39|16.49|true|david white|2013-03-01 > 09:11:58.703136|61.24|joggying > 57|486|65551|4294967397|36.11|9.88|true|katie xylophone|2013-03-01 > 09:11:58.703142|57.10|zync studies > 59|343|65787|4294967312|66.89|6.54|true|mike laertes|2013-03-01 > 09:11:58.703209|27.56|xylophone band > 74|267|65671|4294967409|21.14|14.64|true|priscilla miller|2013-03-01 > 09:11:58.703197|89.06|undecided > 25|336|65587|4294967336|71.01|14.90|true|tom ichabod|2013-03-01 > 09:11:58.703127|74.32|zync studies > 48|346|65712|4294967315|45.01|16.08|true|zach brown|2013-03-01 > 09:11:58.703108|21.68|zync studies > 84|385|65776|4294967452|35.80|32.13|false|xavier zipper|2013-03-01 > 09:11:58.703311|99.46|education > 58|389|65766|4294967416|95.55|20.62|false|sarah miller|2013-03-01 > 09:11:58.703215|70.92|history > 22|403|65565|4294967381|99.65|35.42|false|yuri johnson|2013-03-01 > 09:11:58.703154|94.47|geology > 55|428|65733|4294967535|99.54|5.35|false|jessica king|2013-03-01 > 09:11:58.703233|30.30|forestry > 117|410|65706|4294967391|50.15|0.21|false|quinn johnson|2013-03-01 > 09:11:58.703248|65.99|yard duty > 95|423|65573|4294967378|47.59|17.37|true|alice robinson|2013-03-01 > 09:11:58.703133|54.57|linguistics > 87|332|65748|4294967320|19.83|41.67|false|fred ellison|2013-03-01 > 09:11:58.703289|79.02|mathematics > 114|263|65674|4294967405|84.44|33.18|true|victor van buren|2013-03-01 > 09:11:58.703092|63.74|linguistics > 5|369|65780|4294967488|92.02|38.59|true|zach polk|2013-03-01 > 09:11:58.703271|67.29|yard duty > -3|430|65667|4294967469|65.50|40.46|true|yuri xylophone|2013-03-01 > 09:11:58.703258|30.94|american history > 120|264|65769|4294967486|89.97|41.18|false|xavier hernandez|2013-03-01 > 09:11:58.703140|66.89|philosophy > 107|317|65634|4294967488|5.68|18.89|false|priscilla ichabod|2013-03-01 > 09:11:58.703196|39.42|joggying > 29|386|65723|4294967328|71.48|6.13|false|ulysses ichabod|2013-03-01 > 09:11:58.703215|86.65|xylophone band > 22|434|65768|4294967543|44.25|27.56|false|tom polk|2013-03-01 > 09:11:58.703306|12.30|kindergarten > -1|274|65755|4294967300|22.01|35.52|false|oscar king|2013-03-01 > 09:11:58.703141|33.35|chemistry > 6|365|65603|4294967522|18.51|5.60|false|gabriella king|2013-03-01 > 09:11:58.703104|34.20|geology > 97|414|65757|4294967325|31.82|22.37|false|rachel nixon|2013-03-01 > 09:11:58.703127|61.00|nap time > 72|448|65538|4294967524|80.09|7.73|true|luke brown|2013-03-01 > 09:11:58.703090|95.81|american history > 51|280|65589|4294967486|57.46|23.35|false|zach xylophone|2013-03-01 > 09:11:58.703299|11.54|education > 12|447|65583|4294967389|0.98|29.79|true|yuri polk|2013-03-01 > 09:11:58.703305|1.89|wind surfing > -1|360|65539|4294967464|4.08|39.51|false|oscar davidson|2013-03-01 > 09:11:58.703144|59.47|nap time > 0|380|65569|4294967425|0.94|28.93|false|sarah robinson|2013-03-01 > 09:11:58.703176|88.81|xylophone band > 66|478|65669|4294967339|23.66|38.34|true|yuri carson|2013-03-01 > 09:11:58.703228|64.68|opthamology > 12|322|65771|4294967545|84.87|10.76|false|sarah allen|2013
[GitHub] [flink] flinkbot commented on issue #10840: [FLINK-15444][runtime] Make the component AbstractInvokable in CheckpointBarrierHandler NonNull
flinkbot commented on issue #10840: [FLINK-15444][runtime] Make the component AbstractInvokable in CheckpointBarrierHandler NonNull URL: https://github.com/apache/flink/pull/10840#issuecomment-573561483 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 61e9e675667e05e44e9c4b5ffd7565f60aec6d62 (Mon Jan 13 08:48:18 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15533) Writing DataStream as text file fails due to output path already exists
[
https://issues.apache.org/jira/browse/FLINK-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014126#comment-17014126
]
Kostas Kloudas commented on FLINK-15533:
Thanks for the feedback [~lirui]! I opened a PR for the issue.
> Writing DataStream as text file fails due to output path already exists
> ---
>
> Key: FLINK-15533
> URL: https://issues.apache.org/jira/browse/FLINK-15533
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Affects Versions: 1.10.0
>Reporter: Rui Li
>Assignee: Kostas Kloudas
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following program reproduces the issue.
> {code}
> Configuration configuration = GlobalConfiguration.loadConfiguration();
> configuration.set(DeploymentOptions.TARGET, RemoteExecutor.NAME);
> StreamExecutionEnvironment streamEnv = new
> StreamExecutionEnvironment(configuration);
> DataStream dataStream = streamEnv.fromCollection(Arrays.asList(1,2,3));
> dataStream.writeAsText("hdfs://localhost:8020/tmp/output");
> streamEnv.execute();
> {code}
> The job will fail with the follow error, even though the output path doesn't
> exist before job submission:
> {noformat}
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException):
> /tmp/output already exists as a directory
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15568) RestartPipelinedRegionStrategy: not ensure the EXACTLY_ONCE semantics, in 1.8 versions
[ https://issues.apache.org/jira/browse/FLINK-15568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew.D.lin updated FLINK-15568: - Summary: RestartPipelinedRegionStrategy: not ensure the EXACTLY_ONCE semantics, in 1.8 versions (was: RestartPipelinedRegionStrategy: not ensure the EXACTLY_ONCE semantics) > RestartPipelinedRegionStrategy: not ensure the EXACTLY_ONCE semantics, in 1.8 > versions > -- > > Key: FLINK-15568 > URL: https://issues.apache.org/jira/browse/FLINK-15568 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.8.0, 1.8.1, 1.8.3 >Reporter: Andrew.D.lin >Priority: Minor > Attachments: image-2020-01-13-16-40-47-888.png > > > In 1.8* versions, FailoverRegion.java restart method not restore from latest > checkpoint and marked TODO. > Should we support this feature (region restart) in flink 1.8? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15568) RestartPipelinedRegionStrategy: not ensure the EXACTLY_ONCE semantics
[ https://issues.apache.org/jira/browse/FLINK-15568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew.D.lin updated FLINK-15568: - Description: In 1.8* versions, FailoverRegion.java restart method not restore from latest checkpoint and marked TODO. Should we support this feature (region restart) in flink 1.8? was: !image-2020-01-13-16-40-47-888.png! In 1.8* versions, FailoverRegion.java restart method not restore from latest checkpoint and marked TODO. Should we support this feature (region restart) in flink 1.8? > RestartPipelinedRegionStrategy: not ensure the EXACTLY_ONCE semantics > - > > Key: FLINK-15568 > URL: https://issues.apache.org/jira/browse/FLINK-15568 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.8.0, 1.8.1, 1.8.3 >Reporter: Andrew.D.lin >Priority: Minor > Attachments: image-2020-01-13-16-40-47-888.png > > > In 1.8* versions, FailoverRegion.java restart method not restore from latest > checkpoint and marked TODO. > Should we support this feature (region restart) in flink 1.8? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15568) RestartPipelinedRegionStrategy: not ensure the EXACTLY_ONCE semantics
Andrew.D.lin created FLINK-15568: Summary: RestartPipelinedRegionStrategy: not ensure the EXACTLY_ONCE semantics Key: FLINK-15568 URL: https://issues.apache.org/jira/browse/FLINK-15568 Project: Flink Issue Type: Improvement Components: Runtime / Checkpointing Affects Versions: 1.8.3, 1.8.1, 1.8.0 Reporter: Andrew.D.lin Attachments: image-2020-01-13-16-40-47-888.png !image-2020-01-13-16-40-47-888.png! In 1.8* versions, FailoverRegion.java restart method not restore from latest checkpoint and marked TODO. Should we support this feature (region restart) in flink 1.8? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15565) Incompatible types of expression and result type thrown in codegen
[ https://issues.apache.org/jira/browse/FLINK-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014124#comment-17014124 ] Jingsong Lee commented on FLINK-15565: -- [~ykt836] [~jark] can you assign this to me? > Incompatible types of expression and result type thrown in codegen > -- > > Key: FLINK-15565 > URL: https://issues.apache.org/jira/browse/FLINK-15565 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Priority: Major > Fix For: 1.10.0 > > > *The sql is:* > CREATE TABLE `over10k` ( > t tinyint, > si smallint, > i int, > b bigint, > f float, > d double, > bo boolean, > s varchar, > ts timestamp, > deci decimal(4,2), > bin varchar > ) WITH ( > > 'connector.path'='/daily_regression_batch_hive_1.10/test_window_with_specific_behavior/sources/over10k.csv', > 'format.empty-column-as-null'='true', > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > 'format.type'='csv' > ); > select s, rank() over (partition by s order by si), sum(b) over (partition > by s order by si) from over10k limit 100; > *The data is :* > 109|277|65620|4294967305|97.25|7.80|true|nick quirinius|2013-03-01 > 09:11:58.703226|27.72|undecided > 93|263|65725|4294967341|6.06|4.12|false|calvin king|2013-03-01 > 09:11:58.703299|32.44|values clariffication > 108|383|65629|4294967510|39.55|47.67|false|jessica zipper|2013-03-01 > 09:11:58.703133|74.23|nap time > 89|463|65537|4294967493|64.82|13.79|true|ethan white|2013-03-01 > 09:11:58.703243|89.52|nap time > 88|372|65645|4294967358|34.48|11.18|true|quinn thompson|2013-03-01 > 09:11:58.703168|84.86|forestry > 123|432|65626|4294967435|2.39|16.49|true|david white|2013-03-01 > 09:11:58.703136|61.24|joggying > 57|486|65551|4294967397|36.11|9.88|true|katie xylophone|2013-03-01 > 09:11:58.703142|57.10|zync studies > 59|343|65787|4294967312|66.89|6.54|true|mike laertes|2013-03-01 > 09:11:58.703209|27.56|xylophone band > 74|267|65671|4294967409|21.14|14.64|true|priscilla miller|2013-03-01 > 09:11:58.703197|89.06|undecided > 25|336|65587|4294967336|71.01|14.90|true|tom ichabod|2013-03-01 > 09:11:58.703127|74.32|zync studies > 48|346|65712|4294967315|45.01|16.08|true|zach brown|2013-03-01 > 09:11:58.703108|21.68|zync studies > 84|385|65776|4294967452|35.80|32.13|false|xavier zipper|2013-03-01 > 09:11:58.703311|99.46|education > 58|389|65766|4294967416|95.55|20.62|false|sarah miller|2013-03-01 > 09:11:58.703215|70.92|history > 22|403|65565|4294967381|99.65|35.42|false|yuri johnson|2013-03-01 > 09:11:58.703154|94.47|geology > 55|428|65733|4294967535|99.54|5.35|false|jessica king|2013-03-01 > 09:11:58.703233|30.30|forestry > 117|410|65706|4294967391|50.15|0.21|false|quinn johnson|2013-03-01 > 09:11:58.703248|65.99|yard duty > 95|423|65573|4294967378|47.59|17.37|true|alice robinson|2013-03-01 > 09:11:58.703133|54.57|linguistics > 87|332|65748|4294967320|19.83|41.67|false|fred ellison|2013-03-01 > 09:11:58.703289|79.02|mathematics > 114|263|65674|4294967405|84.44|33.18|true|victor van buren|2013-03-01 > 09:11:58.703092|63.74|linguistics > 5|369|65780|4294967488|92.02|38.59|true|zach polk|2013-03-01 > 09:11:58.703271|67.29|yard duty > -3|430|65667|4294967469|65.50|40.46|true|yuri xylophone|2013-03-01 > 09:11:58.703258|30.94|american history > 120|264|65769|4294967486|89.97|41.18|false|xavier hernandez|2013-03-01 > 09:11:58.703140|66.89|philosophy > 107|317|65634|4294967488|5.68|18.89|false|priscilla ichabod|2013-03-01 > 09:11:58.703196|39.42|joggying > 29|386|65723|4294967328|71.48|6.13|false|ulysses ichabod|2013-03-01 > 09:11:58.703215|86.65|xylophone band > 22|434|65768|4294967543|44.25|27.56|false|tom polk|2013-03-01 > 09:11:58.703306|12.30|kindergarten > -1|274|65755|4294967300|22.01|35.52|false|oscar king|2013-03-01 > 09:11:58.703141|33.35|chemistry > 6|365|65603|4294967522|18.51|5.60|false|gabriella king|2013-03-01 > 09:11:58.703104|34.20|geology > 97|414|65757|4294967325|31.82|22.37|false|rachel nixon|2013-03-01 > 09:11:58.703127|61.00|nap time > 72|448|65538|4294967524|80.09|7.73|true|luke brown|2013-03-01 > 09:11:58.703090|95.81|american history > 51|280|65589|4294967486|57.46|23.35|false|zach xylophone|2013-03-01 > 09:11:58.703299|11.54|education > 12|447|65583|4294967389|0.98|29.79|true|yuri polk|2013-03-01 > 09:11:58.703305|1.89|wind surfing > -1|360|65539|4294967464|4.08|39.51|false|oscar davidson|2013-03-01 > 09:11:58.703144|59.47|nap time > 0|380|65569|4294967425|0.94|28.93|false|sarah robinson|2013-03-01 > 09:11:58.703176|88.81|xylophone band > 66|478|65669|4294967339|23.66|38.34|true|yuri carson|2013-03-01 > 09:11:58.703228|64.68|opthamolog
[jira] [Commented] (FLINK-15559) Missing {{site.baseurl}} in some docs
[
https://issues.apache.org/jira/browse/FLINK-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014123#comment-17014123
]
Jark Wu commented on FLINK-15559:
-
Why the links are broken because of changing base_url ?
AFAIK, if a link is broken, it should always be broken no matter what the
base_url is. That says we don't need a baseurl option.
> Missing {{site.baseurl}} in some docs
> -
>
> Key: FLINK-15559
> URL: https://issues.apache.org/jira/browse/FLINK-15559
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Benchao Li
>Assignee: Benchao Li
>Priority: Major
>
> For example, in
> * dev/table/functions/systemFunctions.md
> * dev/table/functions/systemFunctions.zh.md
> * dev/table/sourceSinks.md
> * dev/table/sourceSinks.zh.md
> * dev/table/sql/queries.zh.md
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhijiangW opened a new pull request #10840: [FLINK-15444][runtime] Make the component AbstractInvokable in CheckpointBarrierHandler NonNull
zhijiangW opened a new pull request #10840: [FLINK-15444][runtime] Make the component AbstractInvokable in CheckpointBarrierHandler NonNull URL: https://github.com/apache/flink/pull/10840 ## What is the purpose of the change *The current component `AbstractInvokable` in `CheckpointBarrierHandler` is annotated as `@Nullable`. Actually in real code path it is passed via the constructor and never be null. The nullable annotation is only used for unit test purpose. But this way would mislead the real usage in practice and bring extra troubles, because you have to alway check whether it is null before usage in related processes.* *We can refactor the related unit tests to implement a dummy `AbstractInvokable` for tests and remove the @Nullable annotation from the related class constructors.* ## Brief change log - *Remove `Nullable` annotation from `CheckpointedInputGate`* - *Remove `Nullable` annotation from `CheckpointBarrierHandler`* - *Introduce `DummyCheckpointInvokable` for tests* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15444) Make the component AbstractInvokable in CheckpointBarrierHandler NonNull
[
https://issues.apache.org/jira/browse/FLINK-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-15444:
---
Labels: pull-request-available (was: )
> Make the component AbstractInvokable in CheckpointBarrierHandler NonNull
> -
>
> Key: FLINK-15444
> URL: https://issues.apache.org/jira/browse/FLINK-15444
> Project: Flink
> Issue Type: Task
> Components: Runtime / Checkpointing
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> The current component {{AbstractInvokable}} in {{CheckpointBarrierHandler}}
> is annotated as {{@Nullable}}. Actually in real code path it is passed via
> the constructor and never be null. The nullable annotation is only used for
> unit test purpose. But this way would mislead the real usage in practice and
> bring extra troubles, because you have to alway check whether it is null
> before usage in related processes.
> We can refactor the related unit tests to implement a dummy
> {{AbstractInvokable}} for tests and remove the {{@Nullable}} annotation from
> the related class constructors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
