[jira] [Commented] (FLINK-16143) Turn on more date time functions of blink planner
[ https://issues.apache.org/jira/browse/FLINK-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039787#comment-17039787 ] Zhenghua Gao commented on FLINK-16143: -- [~jark] Actually the TIMESTAMPDIFF/TIMESTAMPADD with DATE parameter supports DATE type: timestampadd(DAY, 1, date '2016-06-15') gets date '2016-06-16' timestampdiff(DAY, date '2016-06-15', date '2016-06-14') gets -1 > Turn on more date time functions of blink planner > - > > Key: FLINK-16143 > URL: https://issues.apache.org/jira/browse/FLINK-16143 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Planner >Reporter: Zili Chen >Priority: Major > Fix For: 1.11.0 > > > Currently blink planner has a series of built-in functions such as > DATEDIFF > DATE_ADD > DATE_SUB > which haven't been into used so far. We could add the necessary register, > generate and convert code to make it available in production scope. > > what do you think [~jark]? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306 ## CI report: * 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706) * d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740) * d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759) * d846f33ba278b1d75a1eac672c7efebee1350e8f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147664218) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885) * 570a3b05bc858f96a98da192d03e147585ce44cc Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147761655) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922) * f522e48ac9e3478ac88c6de8389d529a17c73377 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/149583795) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5297) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] aljoscha closed pull request #11042: FLINK-15744 Some TaskManager Task exceptions are logged as info
aljoscha closed pull request #11042: FLINK-15744 Some TaskManager Task exceptions are logged as info URL: https://github.com/apache/flink/pull/11042 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] aljoscha commented on issue #11042: FLINK-15744 Some TaskManager Task exceptions are logged as info
aljoscha commented on issue #11042: FLINK-15744 Some TaskManager Task exceptions are logged as info URL: https://github.com/apache/flink/pull/11042#issuecomment-588092905 Thanks, I now merged this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-15744) Some TaskManager Task exception are logged as info
[
https://issues.apache.org/jira/browse/FLINK-15744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-15744.
Fix Version/s: 1.11.0
Resolution: Fixed
master: b38a70201ac34bdced145e00fd9d517d9da46d17
> Some TaskManager Task exception are logged as info
> --
>
> Key: FLINK-15744
> URL: https://issues.apache.org/jira/browse/FLINK-15744
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.9.1
> Environment: 1.9.1 on a recent ubuntu release
>Reporter: Jason Kania
>Assignee: Jason Kania
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After investigating to find the cause for a submitted job alternating between
> the deploying and running states, I tried setting the logging to info level
> and realized that some deployment exceptions were being logged as info when
> they really should be a warning or error level.
> In particular, the following line (at line 960) of the 1.9.1 branch of the
> Task class logs as info:
> LOG.info("{} ({}) switched from {} to {}.", taskNameWithSubtask, executionId,
> currentState, newState, cause);
> This log line always has a non null exception cause so it would make sense to
> log at warning or error level for this state transition so it is more easily
> found.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11071: [FLINK-16019][runtime] fix ContinuousFileReaderOperator error reporting
flinkbot edited a comment on issue #11071: [FLINK-16019][runtime] fix ContinuousFileReaderOperator error reporting URL: https://github.com/apache/flink/pull/11071#issuecomment-585252480 ## CI report: * 7415250749dd788f3ed8abc7106593b7946db170 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148593892) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5109) * 0224fb6b62a0bedc24c804c025cccf08a71cd6b2 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148629364) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5114) * 7467973a1bf580c213e241ff8bf756ed5a2c1ad6 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148988978) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5185) * 318cdc32ca2599c0790201477f2305c40aea27f9 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149507383) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5291) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] aljoscha commented on issue #10005: [FLINK-14533][flink-table-planner] Fix LOWER/UPPER not being pushed down to TableSource
aljoscha commented on issue #10005: [FLINK-14533][flink-table-planner] Fix LOWER/UPPER not being pushed down to TableSource URL: https://github.com/apache/flink/pull/10005#issuecomment-588093428 Cool that this can finally be resolved. @wuchong do you want to merge? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11116: [FLINK-15349] add 'create catalog' DDL to blink planner
flinkbot edited a comment on issue #6: [FLINK-15349] add 'create catalog' DDL to blink planner URL: https://github.com/apache/flink/pull/6#issuecomment-587279527 ## CI report: * 7d09ea38e74e801907d3f1da660c41f5cf739a29 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/149381924) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5262) * 53361a9c2aeaefc852862ca8ed9ed5082fe31fe2 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/149507422) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5292) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11130: [FLINK-15972][python][table-planner][table-planner-blink] Add Python building blocks to make sure the basic functionality of Python TableFun
flinkbot edited a comment on issue #11130: [FLINK-15972][python][table-planner][table-planner-blink] Add Python building blocks to make sure the basic functionality of Python TableFunction could work URL: https://github.com/apache/flink/pull/11130#issuecomment-588081932 ## CI report: * a34a306f023189434d990edd1a85eecfa324842a Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/149583826) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10151: [FLINK-14231] Handle the pending processing-time timers to make endInput semantics on the operator chain strict
flinkbot edited a comment on issue #10151: [FLINK-14231] Handle the pending processing-time timers to make endInput semantics on the operator chain strict URL: https://github.com/apache/flink/pull/10151#issuecomment-552442468 ## CI report: * c6d7f5e864076448dca590035a6a590dc5e25c44 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/135928709) * 682da0aec5dee14c09583468d15115e2a512c827 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140139283) * 9c66fbe1e4d81c3656eba38d56d39dfe0c065f4f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142108012) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3864) * 1b23c2232e9717218a7c61c930c481cbcf2e6f2e Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/142214794) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3888) * 063b5e87dcef4a363f01a48e4af4fb9d3670429f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142218414) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3890) * d5dfd65a163634584e8eaeee452d5454b2d4fe45 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143576795) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4193) * d1f5b89012dc266fc0c664085c1a2aef0c8b95ec Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143669093) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4204) * f61c52bcb1420572c2ad94e3d2f1caafbf7f6081 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143708993) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4215) * 9ea2c427c8f7046d78c122eea8f2d0c10c200224 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144479909) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4362) * fd4f362ceed0a4785c449c807cb46dac15f712ea Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148094252) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4978) * 8e6f927d76fed13e4c095619dee0d72e6a27359f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148125056) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4983) * 12a890c54ecc602a3d46e07e8ded4263469c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148185789) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5005) * 1a8f90917d5fe5b928f4f70bad62a06f1ef9e3fa Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148207317) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5015) * d0f2df460885f14ad4d4f0331bde5ff2de9664ec Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148235635) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5018) * 97bdd27325be54d1ebfcdc2a4bb21b31d910d039 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/148335745) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5040) * 4a56e798775f7f320a3e7b1436d28b00392caa09 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/148395560) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5068) * 564913126f4ac833b52f587e2ccd9de2183dfd5f Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/148509000) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5081) * b2951a6743c376d92371d6eaf3f9be7048a3c29c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149072760) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5193) * 2cb2da33e913914dfb01dda1b8f70626c39eb25d Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/149478672) * 4d58bd199ac7f27e09a3cf60ae7e4cea29b97650 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149492639) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5290) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the messag
[GitHub] [flink] flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306 ## CI report: * 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706) * d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740) * d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759) * d846f33ba278b1d75a1eac672c7efebee1350e8f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147664218) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885) * 570a3b05bc858f96a98da192d03e147585ce44cc Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147761655) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922) * f522e48ac9e3478ac88c6de8389d529a17c73377 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/149583795) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5297) * 4efb9fcbd694ec4b60dd8c8790a076bac9439438 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-16160) Schema#proctime and Schema#rowtime don't work in TableEnvironment#connect code path
Zhenghua Gao created FLINK-16160: Summary: Schema#proctime and Schema#rowtime don't work in TableEnvironment#connect code path Key: FLINK-16160 URL: https://issues.apache.org/jira/browse/FLINK-16160 Project: Flink Issue Type: Sub-task Reporter: Zhenghua Gao In ConnectTableDescriptor#createTemporaryTable, the proctime/rowtime properties are ignored so the generated catalog table is not correct. We should fix this to let TableEnvironment#connect() support watermark. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11047: [FLINK-15912][table] Add Context to TableSourceFactory and TableSinkFactory
flinkbot edited a comment on issue #11047: [FLINK-15912][table] Add Context to TableSourceFactory and TableSinkFactory URL: https://github.com/apache/flink/pull/11047#issuecomment-584007380 ## CI report: * 86c4939042e8af2da1ac1e5900225f4f0310fa04 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148149333) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4991) * 18beb9a3c1ed78a6cbc1bc5ba9f96b51bbf8eeea Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148181501) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5000) * a9125de0859c43262d41c4cecee53ceeb807cf27 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148340324) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5041) * 52bc7dcf579ef6e5e466ec93451cbf56451b1d41 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148547890) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5093) * a307994428657e1025786975b7cf11544afa3ab8 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148567327) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5104) * 4749ac089576b36668e412acdbe56f55b4ffbdc6 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148576458) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5106) * 8c7c673b8bf46297d499db5725c127986533c8ff Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148711047) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5119) * b1f4018140c5b3b881de6da6d5f69422a4b09831 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/149097149) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5200) * 342826120f528c06b9b811e4218a33ee3e4ba2d6 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149392860) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5263) * 2db873816bdf108a21f5405d1d7fbbcba4ef164c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/149414269) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5267) * daf7fda9ab89377adcf5888beebe379300c4140e Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/149456383) * eba92d7d11c5de1a6e948e5e649fe2714bfe1f0d Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149570715) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5295) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink-statefun] igalshilman commented on a change in pull request #25: [FLINK-16106] Add PersistedList state primitive
igalshilman commented on a change in pull request #25: [FLINK-16106] Add
PersistedList state primitive
URL: https://github.com/apache/flink-statefun/pull/25#discussion_r380577504
##
File path:
statefun-sdk/src/main/java/org/apache/flink/statefun/sdk/state/PersistedAppendingBuffer.java
##
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.statefun.sdk.state;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.flink.statefun.sdk.StatefulFunction;
+import org.apache.flink.statefun.sdk.annotations.ForRuntime;
+import org.apache.flink.statefun.sdk.annotations.Persisted;
+
+/**
+ * A {@link PersistedAppendingBuffer} is an append-only buffer registered
within {@link
+ * StatefulFunction}s and is persisted and maintained by the system for
fault-tolerance. Persisted
+ * elements in the buffer may only be updated with bulk replacements.
+ *
+ * Created persisted buffers must be registered by using the {@link
Persisted} annotation. Please
+ * see the class-level Javadoc of {@link StatefulFunction} for an example on
how to do that.
+ *
+ * @see StatefulFunction
+ * @param type of the list elements.
+ */
+public final class PersistedAppendingBuffer {
+ private final String name;
+ private final Class elementType;
+ private AppendingBufferAccessor accessor;
+
+ private PersistedAppendingBuffer(
+ String name, Class elementType, AppendingBufferAccessor accessor) {
+this.name = Objects.requireNonNull(name);
+this.elementType = Objects.requireNonNull(elementType);
+this.accessor = Objects.requireNonNull(accessor);
+ }
+
+ /**
+ * Creates a {@link PersistedAppendingBuffer} instance that may be used to
access persisted state
+ * managed by the system. Access to the persisted buffer is identified by an
unique name and type
+ * of the elements. These may not change across multiple executions of the
application.
+ *
+ * @param name the unique name of the persisted buffer state
+ * @param elementType the type of the elements of this {@code
PersistedAppendingBuffer}.
+ * @param the type of the elements.
+ * @return a {@code PersistedAppendingBuffer} instance.
+ */
+ public static PersistedAppendingBuffer of(String name, Class
elementType) {
+return new PersistedAppendingBuffer<>(name, elementType, new
NonFaultTolerantAccessor<>());
+ }
+
+ /**
+ * Returns the unique name of the persisted buffer.
+ *
+ * @return unique name of the persisted buffer.
+ */
+ public String name() {
+return name;
+ }
+
+ /**
+ * Returns the type of the persisted buffer elements.
+ *
+ * @return the type of the persisted buffer elements.
+ */
+ public Class elementType() {
+return elementType;
+ }
+
+ /**
+ * Appends an element to the persisted buffer.
+ *
+ * If {@code null} is passed in, then this method has no effect and the
persisted buffer
+ * remains the same.
+ *
+ * @param element the element to add to the persisted buffer.
+ */
+ public void append(@Nullable E element) {
+if (element != null) {
+ accessor.append(element);
+}
+ }
+
+ /**
+ * Adds all elements of a list to the persisted buffer.
+ *
+ * If {@code null} or an empty list is passed in, then this method has no
effect and the
+ * persisted buffer remains the same.
+ *
+ * @param elements a list of elements to add to the persisted buffer.
+ */
+ public void appendAll(@Nullable List elements) {
+if (elements != null && !elements.isEmpty()) {
+ accessor.appendAll(elements);
+}
+ }
+
+ /**
+ * Replace the elements in the persisted buffer with the provided list of
elements.
+ *
+ * If an empty list or {@code null} is passed in, this method will have
the same effect as
+ * {@link #clear()}.
+ *
+ * @param elements list of elements to replace the elements in the persisted
buffer with.
+ */
+ public void replaceWith(@Nullable List elements) {
+if (elements != null && !elements.isEmpty()) {
+ accessor.replaceWith(elements);
+} else {
+ accessor.clear();
+}
+ }
+
+ /**
+ * Gets the elements of
[GitHub] [flink] flinkbot edited a comment on issue #11130: [FLINK-15972][python][table-planner][table-planner-blink] Add Python building blocks to make sure the basic functionality of Python TableFun
flinkbot edited a comment on issue #11130: [FLINK-15972][python][table-planner][table-planner-blink] Add Python building blocks to make sure the basic functionality of Python TableFunction could work URL: https://github.com/apache/flink/pull/11130#issuecomment-588081932 ## CI report: * a34a306f023189434d990edd1a85eecfa324842a Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/149583826) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5298) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-16161) Statistics zero should be unknown in HiveCatalog
Jingsong Lee created FLINK-16161: Summary: Statistics zero should be unknown in HiveCatalog Key: FLINK-16161 URL: https://issues.apache.org/jira/browse/FLINK-16161 Project: Flink Issue Type: Bug Components: Connectors / Hive Reporter: Jingsong Lee Fix For: 1.10.1 In hive, treat statistics zero as unknown, but in Flink HiveCatalog, treat zero as real value. This lead wrong inputs to CBO. Previous discussed in [https://github.com/apache/flink/pull/10380] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16162) Bump flink-shaded to 10.0
Chesnay Schepler created FLINK-16162: Summary: Bump flink-shaded to 10.0 Key: FLINK-16162 URL: https://issues.apache.org/jira/browse/FLINK-16162 Project: Flink Issue Type: Bug Components: Build System Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16163) Migrate to flink-shaded-zookeeper
Chesnay Schepler created FLINK-16163: Summary: Migrate to flink-shaded-zookeeper Key: FLINK-16163 URL: https://issues.apache.org/jira/browse/FLINK-16163 Project: Flink Issue Type: Bug Components: Build System Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.11.0 Migrate to Zookeeper 3.4 provided by flink-shaded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16162) Bump flink-shaded to 10.0
[ https://issues.apache.org/jira/browse/FLINK-16162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-16162: --- Labels: pull-request-available (was: ) > Bump flink-shaded to 10.0 > - > > Key: FLINK-16162 > URL: https://issues.apache.org/jira/browse/FLINK-16162 > Project: Flink > Issue Type: Bug > Components: Build System >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol opened a new pull request #11131: [FLINK-16162][build] Bump flink-shaded to 10.0
zentol opened a new pull request #11131: [FLINK-16162][build] Bump flink-shaded to 10.0 URL: https://github.com/apache/flink/pull/11131 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306 ## CI report: * 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706) * d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740) * d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759) * d846f33ba278b1d75a1eac672c7efebee1350e8f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147664218) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885) * 570a3b05bc858f96a98da192d03e147585ce44cc Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147761655) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922) * f522e48ac9e3478ac88c6de8389d529a17c73377 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/149583795) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5297) * 4efb9fcbd694ec4b60dd8c8790a076bac9439438 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/149591443) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11131: [FLINK-16162][build] Bump flink-shaded to 10.0
flinkbot commented on issue #11131: [FLINK-16162][build] Bump flink-shaded to 10.0 URL: https://github.com/apache/flink/pull/11131#issuecomment-588120434 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 3fc1d46420dfd3b316cb6639af7c861f5dac5653 (Wed Feb 19 09:26:58 UTC 2020) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11129: [FLINK-16127][docs]Translate "Fault Tolerance Guarantees" page of connectors into Chinese
flinkbot edited a comment on issue #11129: [FLINK-16127][docs]Translate "Fault Tolerance Guarantees" page of connectors into Chinese URL: https://github.com/apache/flink/pull/11129#issuecomment-588061442 ## CI report: * d85ac6aa7166a24fff2bd1def31023877a95daeb Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149578134) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5296) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11130: [FLINK-15972][python][table-planner][table-planner-blink] Add Python building blocks to make sure the basic functionality of Python TableFun
flinkbot edited a comment on issue #11130: [FLINK-15972][python][table-planner][table-planner-blink] Add Python building blocks to make sure the basic functionality of Python TableFunction could work URL: https://github.com/apache/flink/pull/11130#issuecomment-588081932 ## CI report: * a34a306f023189434d990edd1a85eecfa324842a Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149583826) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5298) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] aljoscha commented on issue #10836: [FLINK-11589][Security] Support security module and context discovery via ServiceLoader
aljoscha commented on issue #10836: [FLINK-11589][Security] Support security module and context discovery via ServiceLoader URL: https://github.com/apache/flink/pull/10836#issuecomment-588125551 @walterddr I rebased this and pushed some changes to your PR (I hope that's ok). I removed the `factories` package and instead moved the factories to the same module as either the contexts or modules. Before, the `factories` package had factories for both modules and contexts. I updated the documentation of the new config options, please also have a look at that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-16164) Disable maven-site-plugin
Chesnay Schepler created FLINK-16164: Summary: Disable maven-site-plugin Key: FLINK-16164 URL: https://issues.apache.org/jira/browse/FLINK-16164 Project: Flink Issue Type: Bug Reporter: Chesnay Schepler -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16164) Disable maven-site-plugin
[ https://issues.apache.org/jira/browse/FLINK-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-16164: - Component/s: Build System > Disable maven-site-plugin > - > > Key: FLINK-16164 > URL: https://issues.apache.org/jira/browse/FLINK-16164 > Project: Flink > Issue Type: Improvement > Components: Build System >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16164) Disable maven-site-plugin
[ https://issues.apache.org/jira/browse/FLINK-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-16164: - Priority: Minor (was: Major) > Disable maven-site-plugin > - > > Key: FLINK-16164 > URL: https://issues.apache.org/jira/browse/FLINK-16164 > Project: Flink > Issue Type: Bug >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-16164) Disable maven-site-plugin
[ https://issues.apache.org/jira/browse/FLINK-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reassigned FLINK-16164: Assignee: Chesnay Schepler > Disable maven-site-plugin > - > > Key: FLINK-16164 > URL: https://issues.apache.org/jira/browse/FLINK-16164 > Project: Flink > Issue Type: Bug >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16164) Disable maven-site-plugin
[ https://issues.apache.org/jira/browse/FLINK-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-16164: - Fix Version/s: 1.11.0 > Disable maven-site-plugin > - > > Key: FLINK-16164 > URL: https://issues.apache.org/jira/browse/FLINK-16164 > Project: Flink > Issue Type: Improvement > Components: Build System >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Minor > Fix For: 1.11.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16164) Disable maven-site-plugin
[ https://issues.apache.org/jira/browse/FLINK-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-16164: - Issue Type: Improvement (was: Bug) > Disable maven-site-plugin > - > > Key: FLINK-16164 > URL: https://issues.apache.org/jira/browse/FLINK-16164 > Project: Flink > Issue Type: Improvement >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16164) Disable maven-site-plugin
[ https://issues.apache.org/jira/browse/FLINK-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-16164: - Description: The maven-site-plugin can be used to generate a site for the project, like the sites used by various maven modules. We inherit an execution for this plugin, but don't actually use it. Hence, we can disable it to remove some noise from the build. > Disable maven-site-plugin > - > > Key: FLINK-16164 > URL: https://issues.apache.org/jira/browse/FLINK-16164 > Project: Flink > Issue Type: Improvement > Components: Build System >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Minor > Fix For: 1.11.0 > > > The maven-site-plugin can be used to generate a site for the project, like > the sites used by various maven modules. > We inherit an execution for this plugin, but don't actually use it. > Hence, we can disable it to remove some noise from the build. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol opened a new pull request #11132: [FLINK-16164][build] Disable maven-site-plugin
zentol opened a new pull request #11132: [FLINK-16164][build] Disable maven-site-plugin URL: https://github.com/apache/flink/pull/11132 The maven-site-plugin can be used to generate a site for the project, like the sites used by various maven modules. We inherit an execution for this plugin, but don't actually use it. This PR disables the execution to remove some noise from the build. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wsry commented on issue #11088: [FLINK-16012][runtime] Reduce the default number of buffers per channel from 2 to 1
wsry commented on issue #11088: [FLINK-16012][runtime] Reduce the default number of buffers per channel from 2 to 1 URL: https://github.com/apache/flink/pull/11088#issuecomment-588129045 I have tested reducing the exclusive buffer from 2 to 1, and then put them into floating buffers instead. It has no evident influence on the performance. I also tried to tune the ```floating buffer per gate``` to see how many floating buffers can compensate the regression, unfortunately, the results show that we need about 1000 floating buffers which means we can't reduce the buffer of upstream if we expect for no performance regression. Can we implementation dynamic policy directly without first reducing the ```buffer per channel``` statically? Any suggestions? @pnowojski @zhijiangW This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16164) Disable maven-site-plugin
[ https://issues.apache.org/jira/browse/FLINK-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-16164: --- Labels: pull-request-available (was: ) > Disable maven-site-plugin > - > > Key: FLINK-16164 > URL: https://issues.apache.org/jira/browse/FLINK-16164 > Project: Flink > Issue Type: Improvement > Components: Build System >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Minor > Labels: pull-request-available > Fix For: 1.11.0 > > > The maven-site-plugin can be used to generate a site for the project, like > the sites used by various maven modules. > We inherit an execution for this plugin, but don't actually use it. > Hence, we can disable it to remove some noise from the build. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #11132: [FLINK-16164][build] Disable maven-site-plugin
flinkbot commented on issue #11132: [FLINK-16164][build] Disable maven-site-plugin URL: https://github.com/apache/flink/pull/11132#issuecomment-588130350 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 0e171258e4aa6e35dd4d685fff401ce49c829094 (Wed Feb 19 09:49:55 UTC 2020) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] yanghua opened a new pull request #11133: [FLINK-15716] Update WindowOperatorMigrationTest to restore from 1.10 savepoint
yanghua opened a new pull request #11133: [FLINK-15716] Update WindowOperatorMigrationTest to restore from 1.10 savepoint URL: https://github.com/apache/flink/pull/11133 ## What is the purpose of the change *This pull request updates WindowOperatorMigrationTest to restore from 1.10 savepoint* ## Brief change log - *Update WindowOperatorMigrationTest to restore from 1.10 savepoint* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15716) Update WindowOperatorMigrationTest to restore from 1.10 savepoint
[
https://issues.apache.org/jira/browse/FLINK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-15716:
---
Labels: pull-request-available (was: )
> Update WindowOperatorMigrationTest to restore from 1.10 savepoint
> -
>
> Key: FLINK-15716
> URL: https://issues.apache.org/jira/browse/FLINK-15716
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.11.0
>Reporter: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Update {{WindowOperatorMigrationTest}} to restore from 1.10 savepoint
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #11133: [FLINK-15716] Update WindowOperatorMigrationTest to restore from 1.10 savepoint
flinkbot commented on issue #11133: [FLINK-15716] Update WindowOperatorMigrationTest to restore from 1.10 savepoint URL: https://github.com/apache/flink/pull/11133#issuecomment-588132445 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 1f683ffa3d28b0d367dafe4ad3ebac4c3582e378 (Wed Feb 19 09:54:47 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-15716).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10836: [FLINK-11589][Security] Support security module and context discovery via ServiceLoader
flinkbot edited a comment on issue #10836: [FLINK-11589][Security] Support security module and context discovery via ServiceLoader URL: https://github.com/apache/flink/pull/10836#issuecomment-573438212 ## CI report: * 0e53d682360fe30462917c820c9aa866caa957b5 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144062568) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4270) * c5927b76a270ceaf3ae6442826e582d97487c52d Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144863310) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4415) * 4abd8a69ee904427937fb2a1df0c2c70743e7f86 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144966253) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4445) * b7fc04de87f5a4ce4330eed335441cd0f05c4b80 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148241552) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5019) * e9b600c4247ed0e732ffdb22da29a4420968fe30 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149315280) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5257) * d89ec0b2ac3a2f1547369443af8262a2fa6b8173 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zentol merged pull request #10984: [FLINK-15835][build][oss] Exclude hadoop-common
zentol merged pull request #10984: [FLINK-15835][build][oss] Exclude hadoop-common URL: https://github.com/apache/flink/pull/10984 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-15835) OSS filesystems bundles entire hadoop-common
[
https://issues.apache.org/jira/browse/FLINK-15835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-15835.
Resolution: Fixed
master: abf8010721a0614354e19ba84ce64f4a1a84de30
> OSS filesystems bundles entire hadoop-common
>
>
> Key: FLINK-15835
> URL: https://issues.apache.org/jira/browse/FLINK-15835
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.9.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The {{flink-oss-fs-hadoop}} filesystem has a dependency on
> {{flink-fs-hadoop-shaded}} (a trimmed down version of hadoop-common) and
> {{hadoop-aliyun}}, the latter also having a dependency on {{hadoop-common}}.
> Since {{flink-fs-hadoop-shaded}} bundles everything and creates a
> dependency-reduced pom (that as a result contains no dependencies) the
> transitive hadoop-common dependency pulled in from {{hadoop-aliyun}} is not
> being altered in any way, and is hence included in it's entirety in the jar.
> We should add an explicit exclusion for the transitive hadoop-common
> dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306 ## CI report: * 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706) * d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740) * d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759) * d846f33ba278b1d75a1eac672c7efebee1350e8f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147664218) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885) * 570a3b05bc858f96a98da192d03e147585ce44cc Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147761655) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922) * f522e48ac9e3478ac88c6de8389d529a17c73377 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/149583795) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5297) * 4efb9fcbd694ec4b60dd8c8790a076bac9439438 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149591443) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5301) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-15833) Fix 'Kerberized YARN on Docker' e2e test on Azure Pipelines
[ https://issues.apache.org/jira/browse/FLINK-15833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger reassigned FLINK-15833: -- Assignee: Robert Metzger > Fix 'Kerberized YARN on Docker' e2e test on Azure Pipelines > --- > > Key: FLINK-15833 > URL: https://issues.apache.org/jira/browse/FLINK-15833 > Project: Flink > Issue Type: Task > Components: Build System, Deployment / YARN, Tests >Reporter: Robert Metzger >Assignee: Robert Metzger >Priority: Major > > AZP support has been introduced in FLINK-13978. > The YARN on Docker tests are disabled in the PR introducing Azure support. > The issue is likely occurring due to different resources available on the > Azure machines, compared to Travis. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16105) Translate "User-defined Sources & Sinks" page of "Table API & SQL" into Chinese
[
https://issues.apache.org/jira/browse/FLINK-16105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039874#comment-17039874
]
Benchao Li commented on FLINK-16105:
[~jark] I can help translate this one.
> Translate "User-defined Sources & Sinks" page of "Table API & SQL" into
> Chinese
>
>
> Key: FLINK-16105
> URL: https://issues.apache.org/jira/browse/FLINK-16105
> Project: Flink
> Issue Type: Sub-task
> Components: chinese-translation, Documentation
>Reporter: Jark Wu
>Priority: Major
>
> The page url is
> https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/sourceSinks.html
> The markdown file is located in {{flink/docs/dev/table/sourceSinks.zh.md}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16094) Translate "User-defined Functions" page of "Functions" into Chinese
[
https://issues.apache.org/jira/browse/FLINK-16094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039873#comment-17039873
]
Benchao Li commented on FLINK-16094:
[~jark] I can help translate this one.
> Translate "User-defined Functions" page of "Functions" into Chinese
>
>
> Key: FLINK-16094
> URL: https://issues.apache.org/jira/browse/FLINK-16094
> Project: Flink
> Issue Type: Sub-task
> Components: chinese-translation, Documentation
>Reporter: Jark Wu
>Priority: Major
>
> The page url is
> https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/functions/udfs.html
> The markdown file is located in {{flink/docs/dev/table/functions/udfs.zh.md}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-15491) Fix incorrect path while missing trailing slash of high-availability.zookeeper.path.running-registry parameter.
[
https://issues.apache.org/jira/browse/FLINK-15491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qingxin Wu closed FLINK-15491.
--
Resolution: Abandoned
> Fix incorrect path while missing trailing slash of
> high-availability.zookeeper.path.running-registry parameter.
> ---
>
> Key: FLINK-15491
> URL: https://issues.apache.org/jira/browse/FLINK-15491
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.9.0
>Reporter: Qingxin Wu
>Priority: Minor
> Attachments: 03a8857d50e911f9d09c68df0.png
>
>
> By default, the value of configuration
> *high-availability.zookeeper.path.running-registry* must end with a slash. If
> we set this parameter as following:
> {code:java}
> high-availability.zookeeper.path.running-registry: /running_job_registry{code}
> an incorrect path will be created, as shown in the following picture
> !03a8857d50e911f9d09c68df0.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wangyang0918 edited a comment on issue #11117: [FLINK-16115][filesystem] Make aliyun oss filesystem could work with plugin mechanism
wangyang0918 edited a comment on issue #7: [FLINK-16115][filesystem] Make aliyun oss filesystem could work with plugin mechanism URL: https://github.com/apache/flink/pull/7#issuecomment-588000261 @AHeise Thanks for your additional explanation. Your analysis is on point. @zentol @pnowojski Do you have other concerns and could you help with merging? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11131: [FLINK-16162][build] Bump flink-shaded to 10.0
flinkbot commented on issue #11131: [FLINK-16162][build] Bump flink-shaded to 10.0 URL: https://github.com/apache/flink/pull/11131#issuecomment-588143394 ## CI report: * 3fc1d46420dfd3b316cb6639af7c861f5dac5653 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11132: [FLINK-16164][build] Disable maven-site-plugin
flinkbot commented on issue #11132: [FLINK-16164][build] Disable maven-site-plugin URL: https://github.com/apache/flink/pull/11132#issuecomment-588143489 ## CI report: * 0e171258e4aa6e35dd4d685fff401ce49c829094 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15959) Add min/max number of slots configuration to limit total number of slots
[
https://issues.apache.org/jira/browse/FLINK-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039875#comment-17039875
]
YufeiLiu commented on FLINK-15959:
--
[~xintongsong] I'm worried about the uncertainty, lost a TM won't make much
difference, but the situation could get worse after a few times restart. If
source operator parallelism is less than others, the source tasks are tend to
converge on a few TMs after several times TM lost.
I think combine the function of {{SlotPool}} and {{ResourceManager}} is a good
idea for the long term, and these config will work in any case if JobMaster
didn't caching slot.
> Add min/max number of slots configuration to limit total number of slots
>
>
> Key: FLINK-15959
> URL: https://issues.apache.org/jira/browse/FLINK-15959
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: YufeiLiu
>Priority: Major
>
> Flink removed `-n` option after FLIP-6, change to ResourceManager start a new
> worker when required. But I think maintain a certain amount of slots is
> necessary. These workers will start immediately when ResourceManager starts
> and would not release even if all slots are free.
> Here are some resons:
> # Users actually know how many resources are needed when run a single job,
> initialize all workers when cluster starts can speed up startup process.
> # Job schedule in topology order, next operator won't schedule until prior
> execution slot allocated. The TaskExecutors will start in several batchs in
> some cases, it might slow down the startup speed.
> # Flink support
> [FLINK-12122|https://issues.apache.org/jira/browse/FLINK-12122] [Spread out
> tasks evenly across all available registered TaskManagers], but it will only
> effect if all TMs are registered. Start all TMs at begining can slove this
> problem.
> *suggestion:*
> * Add config "taskmanager.minimum.numberOfTotalSlots" and
> "taskmanager.maximum.numberOfTotalSlots", default behavior is still like
> before.
> * Start plenty number of workers to satisfy minimum slots when
> ResourceManager accept leadership(subtract recovered workers).
> * Don't comlete slot request until minimum number of slots are registered,
> and throw exeception when exceed maximum.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zentol commented on issue #11117: [FLINK-16115][filesystem] Make aliyun oss filesystem could work with plugin mechanism
zentol commented on issue #7: [FLINK-16115][filesystem] Make aliyun oss filesystem could work with plugin mechanism URL: https://github.com/apache/flink/pull/7#issuecomment-588145838 I agree with @AHeise that this FS is likely just broken. The only way this could've worked is if another filesystem is in /lib that bundles flink-hadoop-fs, without relocating contained classes and relocates hadoop. I don't know whether any filesystem qualifies. I think the proposed changed makes sense. AFAIK we cannot add an end-to-end test since Aliyun OSS cannot be setup locally and we don't have any credentials. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] rmetzger opened a new pull request #11134: [FLINK-15833] Fix 'Kerberized YARN on Docker' e2e test
rmetzger opened a new pull request #11134: [FLINK-15833] Fix 'Kerberized YARN on Docker' e2e test URL: https://github.com/apache/flink/pull/11134 ## What is the purpose of the change The Kerberized YARN on Docker tests did not pass on Azure Pipelines. ## Brief change log Mostly removed the `-it` argument from docker exec. `-it` is meant to be used for interactive shell sessions, not in scripts on a CI system. Because of the `-it` argument, the docker command printed `the input device is not a TTY` to the output. That's why some checks / assumptions in the scripts failed. ## Verifying this change Run this end to end test locally: `FLINK_DIR=../build-target ./run-single-test.sh ./test-scripts/test_yarn_kerberos_docker.sh` Check the output of this build job: https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5309&view=logs&j=b1623ac9-0979-5b0d-2e5e-1377d695c991 ## Does this pull request potentially affect one of the following parts: only affects test scripts ## Documentation only affects test scripts This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15833) Fix 'Kerberized YARN on Docker' e2e test on Azure Pipelines
[ https://issues.apache.org/jira/browse/FLINK-15833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-15833: --- Labels: pull-request-available (was: ) > Fix 'Kerberized YARN on Docker' e2e test on Azure Pipelines > --- > > Key: FLINK-15833 > URL: https://issues.apache.org/jira/browse/FLINK-15833 > Project: Flink > Issue Type: Task > Components: Build System, Deployment / YARN, Tests >Reporter: Robert Metzger >Assignee: Robert Metzger >Priority: Major > Labels: pull-request-available > > AZP support has been introduced in FLINK-13978. > The YARN on Docker tests are disabled in the PR introducing Azure support. > The issue is likely occurring due to different resources available on the > Azure machines, compared to Travis. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15959) Add min/max number of slots configuration to limit total number of slots
[
https://issues.apache.org/jira/browse/FLINK-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039879#comment-17039879
]
Xintong Song commented on FLINK-15959:
--
[~liuyufei]
bq. but the situation could get worse after a few times restart.
I don't understand why is that. More failovers does not necessarily make {{
SlotPool }} caches less slots/TMs. Shouldn't it only fail to cache slots from
recent lost TMs? Flink should have already launched new TMs for those lost in
previous attempts, right?
> Add min/max number of slots configuration to limit total number of slots
>
>
> Key: FLINK-15959
> URL: https://issues.apache.org/jira/browse/FLINK-15959
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: YufeiLiu
>Priority: Major
>
> Flink removed `-n` option after FLIP-6, change to ResourceManager start a new
> worker when required. But I think maintain a certain amount of slots is
> necessary. These workers will start immediately when ResourceManager starts
> and would not release even if all slots are free.
> Here are some resons:
> # Users actually know how many resources are needed when run a single job,
> initialize all workers when cluster starts can speed up startup process.
> # Job schedule in topology order, next operator won't schedule until prior
> execution slot allocated. The TaskExecutors will start in several batchs in
> some cases, it might slow down the startup speed.
> # Flink support
> [FLINK-12122|https://issues.apache.org/jira/browse/FLINK-12122] [Spread out
> tasks evenly across all available registered TaskManagers], but it will only
> effect if all TMs are registered. Start all TMs at begining can slove this
> problem.
> *suggestion:*
> * Add config "taskmanager.minimum.numberOfTotalSlots" and
> "taskmanager.maximum.numberOfTotalSlots", default behavior is still like
> before.
> * Start plenty number of workers to satisfy minimum slots when
> ResourceManager accept leadership(subtract recovered workers).
> * Don't comlete slot request until minimum number of slots are registered,
> and throw exeception when exceed maximum.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15959) Add min/max number of slots configuration to limit total number of slots
[
https://issues.apache.org/jira/browse/FLINK-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039879#comment-17039879
]
Xintong Song edited comment on FLINK-15959 at 2/19/20 10:23 AM:
[~liuyufei]
bq. but the situation could get worse after a few times restart.
I don't understand why is that. More failovers does not necessarily make
{{SlotPool}} caches less slots/TMs. Shouldn't it only fail to cache slots from
recent lost TMs? Flink should have already launched new TMs for those lost in
previous attempts, right?
was (Author: xintongsong):
[~liuyufei]
bq. but the situation could get worse after a few times restart.
I don't understand why is that. More failovers does not necessarily make {{
SlotPool }} caches less slots/TMs. Shouldn't it only fail to cache slots from
recent lost TMs? Flink should have already launched new TMs for those lost in
previous attempts, right?
> Add min/max number of slots configuration to limit total number of slots
>
>
> Key: FLINK-15959
> URL: https://issues.apache.org/jira/browse/FLINK-15959
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: YufeiLiu
>Priority: Major
>
> Flink removed `-n` option after FLIP-6, change to ResourceManager start a new
> worker when required. But I think maintain a certain amount of slots is
> necessary. These workers will start immediately when ResourceManager starts
> and would not release even if all slots are free.
> Here are some resons:
> # Users actually know how many resources are needed when run a single job,
> initialize all workers when cluster starts can speed up startup process.
> # Job schedule in topology order, next operator won't schedule until prior
> execution slot allocated. The TaskExecutors will start in several batchs in
> some cases, it might slow down the startup speed.
> # Flink support
> [FLINK-12122|https://issues.apache.org/jira/browse/FLINK-12122] [Spread out
> tasks evenly across all available registered TaskManagers], but it will only
> effect if all TMs are registered. Start all TMs at begining can slove this
> problem.
> *suggestion:*
> * Add config "taskmanager.minimum.numberOfTotalSlots" and
> "taskmanager.maximum.numberOfTotalSlots", default behavior is still like
> before.
> * Start plenty number of workers to satisfy minimum slots when
> ResourceManager accept leadership(subtract recovered workers).
> * Don't comlete slot request until minimum number of slots are registered,
> and throw exeception when exceed maximum.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #11134: [FLINK-15833] Fix 'Kerberized YARN on Docker' e2e test
flinkbot commented on issue #11134: [FLINK-15833] Fix 'Kerberized YARN on Docker' e2e test URL: https://github.com/apache/flink/pull/11134#issuecomment-588149009 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit d1c300925c2e102650311664585ad63425ab7d37 (Wed Feb 19 10:27:49 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zentol opened a new pull request #11135: [FLINK-7727] [REST] Improve error handling in StaticFileServerHandlers
zentol opened a new pull request #11135: [FLINK-7727] [REST] Improve error handling in StaticFileServerHandlers URL: https://github.com/apache/flink/pull/11135 Adjusts the error handling in the `StaticFileServerHandlers` to a) prevent probing of file existence outside of web directory, b) return METHOD_NOT_ALLOWED when the requested resource is not a file, c) log errors. These changes are in part for security purposes (a), but mostly to ease debugging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11135: [FLINK-7727] [REST] Improve error handling in StaticFileServerHandlers
flinkbot commented on issue #11135: [FLINK-7727] [REST] Improve error handling in StaticFileServerHandlers URL: https://github.com/apache/flink/pull/11135#issuecomment-588150200 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 8fd95233a26eb69eeb7947122335098ec718ba4b (Wed Feb 19 10:30:29 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #11118: [FLINK-16072][python] Optimize the performance of the write/read null mask
dianfu commented on a change in pull request #8: [FLINK-16072][python] Optimize the performance of the write/read null mask URL: https://github.com/apache/flink/pull/8#discussion_r381200524 ## File path: flink-python/pyflink/fn_execution/coder_impl.py ## @@ -25,56 +25,331 @@ class FlattenRowCoderImpl(StreamCoderImpl): +null_mask_search_table = ((False, False, False, False, False, False, False, False), + (False, False, False, False, False, False, False, True), + (False, False, False, False, False, False, True, False), + (False, False, False, False, False, False, True, True), + (False, False, False, False, False, True, False, False), + (False, False, False, False, False, True, False, True), + (False, False, False, False, False, True, True, False), + (False, False, False, False, False, True, True, True), + (False, False, False, False, True, False, False, False), + (False, False, False, False, True, False, False, True), + (False, False, False, False, True, False, True, False), + (False, False, False, False, True, False, True, True), + (False, False, False, False, True, True, False, False), + (False, False, False, False, True, True, False, True), + (False, False, False, False, True, True, True, False), + (False, False, False, False, True, True, True, True), + (False, False, False, True, False, False, False, False), + (False, False, False, True, False, False, False, True), + (False, False, False, True, False, False, True, False), + (False, False, False, True, False, False, True, True), + (False, False, False, True, False, True, False, False), + (False, False, False, True, False, True, False, True), + (False, False, False, True, False, True, True, False), + (False, False, False, True, False, True, True, True), + (False, False, False, True, True, False, False, False), + (False, False, False, True, True, False, False, True), + (False, False, False, True, True, False, True, False), + (False, False, False, True, True, False, True, True), + (False, False, False, True, True, True, False, False), + (False, False, False, True, True, True, False, True), + (False, False, False, True, True, True, True, False), + (False, False, False, True, True, True, True, True), + (False, False, True, False, False, False, False, False), + (False, False, True, False, False, False, False, True), + (False, False, True, False, False, False, True, False), + (False, False, True, False, False, False, True, True), + (False, False, True, False, False, True, False, False), + (False, False, True, False, False, True, False, True), + (False, False, True, False, False, True, True, False), + (False, False, True, False, False, True, True, True), + (False, False, True, False, True, False, False, False), + (False, False, True, False, True, False, False, True), + (False, False, True, False, True, False, True, False), + (False, False, True, False, True, False, True, True), + (False, False, True, False, True, True, False, False), + (False, False, True, False, True, True, False, True), + (False, False, True, False, True, True, True, False), + (False, False, True, False, True, True, True, True), + (False, False, True, True, False, False, False, False), + (False, False, True, True, False, False, False, True), + (False, False, True, True, False, False, True, False), + (False, False, True, True, False, False, True, True), + (False, False, True, True, False, True, False, False), +
[GitHub] [flink] dianfu commented on a change in pull request #11118: [FLINK-16072][python] Optimize the performance of the write/read null mask
dianfu commented on a change in pull request #8: [FLINK-16072][python] Optimize the performance of the write/read null mask URL: https://github.com/apache/flink/pull/8#discussion_r381188340 ## File path: flink-python/pyflink/fn_execution/coder_impl.py ## @@ -25,56 +25,331 @@ class FlattenRowCoderImpl(StreamCoderImpl): +null_mask_search_table = ((False, False, False, False, False, False, False, False), Review comment: Could we generate this search table using one function? Besides, it would be great to add some comments on how to generate the search table. It will make it more readable and maintainable. What's your thought? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #11118: [FLINK-16072][python] Optimize the performance of the write/read null mask
dianfu commented on a change in pull request #8: [FLINK-16072][python] Optimize the performance of the write/read null mask URL: https://github.com/apache/flink/pull/8#discussion_r381195297 ## File path: flink-python/pyflink/fn_execution/coder_impl.py ## @@ -25,56 +25,331 @@ class FlattenRowCoderImpl(StreamCoderImpl): +null_mask_search_table = ((False, False, False, False, False, False, False, False), + (False, False, False, False, False, False, False, True), + (False, False, False, False, False, False, True, False), + (False, False, False, False, False, False, True, True), + (False, False, False, False, False, True, False, False), + (False, False, False, False, False, True, False, True), + (False, False, False, False, False, True, True, False), + (False, False, False, False, False, True, True, True), + (False, False, False, False, True, False, False, False), + (False, False, False, False, True, False, False, True), + (False, False, False, False, True, False, True, False), + (False, False, False, False, True, False, True, True), + (False, False, False, False, True, True, False, False), + (False, False, False, False, True, True, False, True), + (False, False, False, False, True, True, True, False), + (False, False, False, False, True, True, True, True), + (False, False, False, True, False, False, False, False), + (False, False, False, True, False, False, False, True), + (False, False, False, True, False, False, True, False), + (False, False, False, True, False, False, True, True), + (False, False, False, True, False, True, False, False), + (False, False, False, True, False, True, False, True), + (False, False, False, True, False, True, True, False), + (False, False, False, True, False, True, True, True), + (False, False, False, True, True, False, False, False), + (False, False, False, True, True, False, False, True), + (False, False, False, True, True, False, True, False), + (False, False, False, True, True, False, True, True), + (False, False, False, True, True, True, False, False), + (False, False, False, True, True, True, False, True), + (False, False, False, True, True, True, True, False), + (False, False, False, True, True, True, True, True), + (False, False, True, False, False, False, False, False), + (False, False, True, False, False, False, False, True), + (False, False, True, False, False, False, True, False), + (False, False, True, False, False, False, True, True), + (False, False, True, False, False, True, False, False), + (False, False, True, False, False, True, False, True), + (False, False, True, False, False, True, True, False), + (False, False, True, False, False, True, True, True), + (False, False, True, False, True, False, False, False), + (False, False, True, False, True, False, False, True), + (False, False, True, False, True, False, True, False), + (False, False, True, False, True, False, True, True), + (False, False, True, False, True, True, False, False), + (False, False, True, False, True, True, False, True), + (False, False, True, False, True, True, True, False), + (False, False, True, False, True, True, True, True), + (False, False, True, True, False, False, False, False), + (False, False, True, True, False, False, False, True), + (False, False, True, True, False, False, True, False), + (False, False, True, True, False, False, True, True), + (False, False, True, True, False, True, False, False), +
[GitHub] [flink] dianfu commented on a change in pull request #11118: [FLINK-16072][python] Optimize the performance of the write/read null mask
dianfu commented on a change in pull request #8: [FLINK-16072][python] Optimize the performance of the write/read null mask URL: https://github.com/apache/flink/pull/8#discussion_r381193279 ## File path: flink-python/pyflink/fn_execution/coder_impl.py ## @@ -25,56 +25,331 @@ class FlattenRowCoderImpl(StreamCoderImpl): +null_mask_search_table = ((False, False, False, False, False, False, False, False), + (False, False, False, False, False, False, False, True), + (False, False, False, False, False, False, True, False), + (False, False, False, False, False, False, True, True), + (False, False, False, False, False, True, False, False), + (False, False, False, False, False, True, False, True), + (False, False, False, False, False, True, True, False), + (False, False, False, False, False, True, True, True), + (False, False, False, False, True, False, False, False), + (False, False, False, False, True, False, False, True), + (False, False, False, False, True, False, True, False), + (False, False, False, False, True, False, True, True), + (False, False, False, False, True, True, False, False), + (False, False, False, False, True, True, False, True), + (False, False, False, False, True, True, True, False), + (False, False, False, False, True, True, True, True), + (False, False, False, True, False, False, False, False), + (False, False, False, True, False, False, False, True), + (False, False, False, True, False, False, True, False), + (False, False, False, True, False, False, True, True), + (False, False, False, True, False, True, False, False), + (False, False, False, True, False, True, False, True), + (False, False, False, True, False, True, True, False), + (False, False, False, True, False, True, True, True), + (False, False, False, True, True, False, False, False), + (False, False, False, True, True, False, False, True), + (False, False, False, True, True, False, True, False), + (False, False, False, True, True, False, True, True), + (False, False, False, True, True, True, False, False), + (False, False, False, True, True, True, False, True), + (False, False, False, True, True, True, True, False), + (False, False, False, True, True, True, True, True), + (False, False, True, False, False, False, False, False), + (False, False, True, False, False, False, False, True), + (False, False, True, False, False, False, True, False), + (False, False, True, False, False, False, True, True), + (False, False, True, False, False, True, False, False), + (False, False, True, False, False, True, False, True), + (False, False, True, False, False, True, True, False), + (False, False, True, False, False, True, True, True), + (False, False, True, False, True, False, False, False), + (False, False, True, False, True, False, False, True), + (False, False, True, False, True, False, True, False), + (False, False, True, False, True, False, True, True), + (False, False, True, False, True, True, False, False), + (False, False, True, False, True, True, False, True), + (False, False, True, False, True, True, True, False), + (False, False, True, False, True, True, True, True), + (False, False, True, True, False, False, False, False), + (False, False, True, True, False, False, False, True), + (False, False, True, True, False, False, True, False), + (False, False, True, True, False, False, True, True), + (False, False, True, True, False, True, False, False), +
[GitHub] [flink] flinkbot edited a comment on issue #10836: [FLINK-11589][Security] Support security module and context discovery via ServiceLoader
flinkbot edited a comment on issue #10836: [FLINK-11589][Security] Support security module and context discovery via ServiceLoader URL: https://github.com/apache/flink/pull/10836#issuecomment-573438212 ## CI report: * 0e53d682360fe30462917c820c9aa866caa957b5 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144062568) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4270) * c5927b76a270ceaf3ae6442826e582d97487c52d Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144863310) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4415) * 4abd8a69ee904427937fb2a1df0c2c70743e7f86 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144966253) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4445) * b7fc04de87f5a4ce4330eed335441cd0f05c4b80 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/148241552) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5019) * e9b600c4247ed0e732ffdb22da29a4420968fe30 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149315280) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5257) * d89ec0b2ac3a2f1547369443af8262a2fa6b8173 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/149602081) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5303) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangyang0918 commented on issue #11117: [FLINK-16115][filesystem] Make aliyun oss filesystem could work with plugin mechanism
wangyang0918 commented on issue #7: [FLINK-16115][filesystem] Make aliyun oss filesystem could work with plugin mechanism URL: https://github.com/apache/flink/pull/7#issuecomment-588154724 @zentol Yes, you are right. We can not set up a local oss and we could not add a oss token in the test script. I have manually run a standalone cluster with oss filesystem enable. And it works fine. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangyang0918 edited a comment on issue #11117: [FLINK-16115][filesystem] Make aliyun oss filesystem could work with plugin mechanism
wangyang0918 edited a comment on issue #7: [FLINK-16115][filesystem] Make aliyun oss filesystem could work with plugin mechanism URL: https://github.com/apache/flink/pull/7#issuecomment-588154724 @zentol Yes, you are right. We can not set up a local oss and we could not add a oss token in the test script. I have manually run a standalone cluster with oss filesystem enabled. And it works fine. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] rmetzger commented on issue #11134: [FLINK-15833] Fix 'Kerberized YARN on Docker' e2e test
rmetzger commented on issue #11134: [FLINK-15833] Fix 'Kerberized YARN on Docker' e2e test URL: https://github.com/apache/flink/pull/11134#issuecomment-588156475 Thanks for the quick review :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-16165) NestedLoopJoin fallback to HashJoin when build records number is very large
Jingsong Lee created FLINK-16165: Summary: NestedLoopJoin fallback to HashJoin when build records number is very large Key: FLINK-16165 URL: https://issues.apache.org/jira/browse/FLINK-16165 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Reporter: Jingsong Lee Fix For: 1.11.0 Now, if the statistic is not so accurate, maybe choose wrong join type to nested loop join. If build records number is very large, this lead to very slow join, the user looks like the join is in Hang. It is a stability problem, We should fallback to hash join in runtime to avoid this hang. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14106) Make SlotManager pluggable
[ https://issues.apache.org/jira/browse/FLINK-14106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039901#comment-17039901 ] Xintong Song commented on FLINK-14106: -- The WIP FLIP-56 (Dynamic Slot Allocation) tries to introduce significant changes to the slot allocations. Given the efforts needed and potential risk of changing the current slot allocation behaviors directly, after an offline discussion with [~sewen], [~trohrmann] and some other people, we decided to first make it a plugin beside the current slot allocation strategy. Therefore, we are reviving this issue again to make Flink's slot allocation pluggable. A design doc of proposed changes is linked to the ticket. > Make SlotManager pluggable > -- > > Key: FLINK-14106 > URL: https://issues.apache.org/jira/browse/FLINK-14106 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0 >Reporter: Xintong Song >Priority: Major > Fix For: 1.11.0 > > > As we are enabling fine grained resource management in 1.10, we can have > various resource scheduling strategies. Such strategies generally should make > the following three decisions. > * When to launch new / release existing TMs? (How many TMs) > * What and how many resources should TMs be started with? > * How to allocate between slot requests and TM resources? > We may want to make above decisions differently in different scenarios > (active/reactive mode, perjob/session mode, etc.). Therefore, we propose to > make the scheduling strategies pluggable. > We propose to make the following changes: > * Make SlotManager an interface, and implements it differently for different > strategies strategies. > * Modify ResourceManager-SlotManager interfaces to cover all the three > decisions mentioned above in SlotManager. In particular, SlotManager needs to > allocate TM resources instead of slot resources from ResourceActions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11131: [FLINK-16162][build] Bump flink-shaded to 10.0
flinkbot edited a comment on issue #11131: [FLINK-16162][build] Bump flink-shaded to 10.0 URL: https://github.com/apache/flink/pull/11131#issuecomment-588143394 ## CI report: * 3fc1d46420dfd3b316cb6639af7c861f5dac5653 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/149602116) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5304) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11133: [FLINK-15716] Update WindowOperatorMigrationTest to restore from 1.10 savepoint
flinkbot commented on issue #11133: [FLINK-15716] Update WindowOperatorMigrationTest to restore from 1.10 savepoint URL: https://github.com/apache/flink/pull/11133#issuecomment-588160633 ## CI report: * 1f683ffa3d28b0d367dafe4ad3ebac4c3582e378 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11132: [FLINK-16164][build] Disable maven-site-plugin
flinkbot edited a comment on issue #11132: [FLINK-16164][build] Disable maven-site-plugin URL: https://github.com/apache/flink/pull/11132#issuecomment-588143489 ## CI report: * 0e171258e4aa6e35dd4d685fff401ce49c829094 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/149602147) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5305) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11134: [FLINK-15833] Fix 'Kerberized YARN on Docker' e2e test
flinkbot commented on issue #11134: [FLINK-15833] Fix 'Kerberized YARN on Docker' e2e test URL: https://github.com/apache/flink/pull/11134#issuecomment-588160681 ## CI report: * d1c300925c2e102650311664585ad63425ab7d37 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-16166) Javadoc generation fails on Java 11
Chesnay Schepler created FLINK-16166:
Summary: Javadoc generation fails on Java 11
Key: FLINK-16166
URL: https://issues.apache.org/jira/browse/FLINK-16166
Project: Flink
Issue Type: Bug
Components: Build System, Documentation
Reporter: Chesnay Schepler
Apparently it is no longer possible to disable {{doclint}} on Java 11, as a
result of which {{javadoc}} complains about various errors in our javadocs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16143) Turn on more date time functions of blink planner
[ https://issues.apache.org/jira/browse/FLINK-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039913#comment-17039913 ] Zili Chen commented on FLINK-16143: --- [~docete] If we don't support DATE_DIFF, we should remove those dead code from our codebase. Actually there are plenty of dummy functions inside blink planner, including those listed above and BIT_* and more. > Turn on more date time functions of blink planner > - > > Key: FLINK-16143 > URL: https://issues.apache.org/jira/browse/FLINK-16143 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Planner >Reporter: Zili Chen >Priority: Major > Fix For: 1.11.0 > > > Currently blink planner has a series of built-in functions such as > DATEDIFF > DATE_ADD > DATE_SUB > which haven't been into used so far. We could add the necessary register, > generate and convert code to make it available in production scope. > > what do you think [~jark]? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] fhueske commented on issue #9694: [hotfix] [docs] Fixes in glossar
fhueske commented on issue #9694: [hotfix] [docs] Fixes in glossar URL: https://github.com/apache/flink/pull/9694#issuecomment-588165634 Thanks for the update @afedulov. Will merge this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] asfgit closed pull request #9694: [hotfix] [docs] Fixes in glossar
asfgit closed pull request #9694: [hotfix] [docs] Fixes in glossar URL: https://github.com/apache/flink/pull/9694 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16069) Creation of TaskDeploymentDescriptor can block main thread for long time
[
https://issues.apache.org/jira/browse/FLINK-16069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039916#comment-17039916
]
Till Rohrmann commented on FLINK-16069:
---
Whenever we access state which is not immutable belonging to the main thread
(e.g. the {{ExecutionGraph}}), then we must do the access from the main thread.
If this is not guaranteed, then we potentially produce a race condition.
> Creation of TaskDeploymentDescriptor can block main thread for long time
>
>
> Key: FLINK-16069
> URL: https://issues.apache.org/jira/browse/FLINK-16069
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: huweihua
>Priority: Major
>
> The deploy of tasks will take long time when we submit a high parallelism
> job. And Execution#deploy run in mainThread, so it will block JobMaster
> process other akka messages, such as Heartbeat. The creation of
> TaskDeploymentDescriptor take most of time. We can put the creation in future.
> For example, A job [source(8000)->sink(8000)], the total 16000 tasks from
> SCHEDULED to DEPLOYING took more than 1mins. This caused the heartbeat of
> TaskManager timeout and job never success.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16143) Turn on more date time functions of blink planner
[
https://issues.apache.org/jira/browse/FLINK-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039921#comment-17039921
]
Zili Chen commented on FLINK-16143:
---
[~docete] Another problem is what does it produce
{{timestampdiff(DAY, 1, date '2016-06-15 00:01:02')}}
?
I think DATE_DIFF gives only date part but not sure what behavior timestamp
diff obeys.
> Turn on more date time functions of blink planner
> -
>
> Key: FLINK-16143
> URL: https://issues.apache.org/jira/browse/FLINK-16143
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Planner
>Reporter: Zili Chen
>Priority: Major
> Fix For: 1.11.0
>
>
> Currently blink planner has a series of built-in functions such as
> DATEDIFF
> DATE_ADD
> DATE_SUB
> which haven't been into used so far. We could add the necessary register,
> generate and convert code to make it available in production scope.
>
> what do you think [~jark]?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-16143) Turn on more date time functions of blink planner
[
https://issues.apache.org/jira/browse/FLINK-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039921#comment-17039921
]
Zili Chen edited comment on FLINK-16143 at 2/19/20 11:17 AM:
-
[~docete] Another problem is what does it produce
{{timestampdiff(DAY, 1, date '2016-06-15 00:01:02')}}
?
I think DATESUB gives only date part but not sure what behavior timestamp diff
obeys.
was (Author: tison):
[~docete] Another problem is what does it produce
{{timestampdiff(DAY, 1, date '2016-06-15 00:01:02')}}
?
I think DATE_DIFF gives only date part but not sure what behavior timestamp
diff obeys.
> Turn on more date time functions of blink planner
> -
>
> Key: FLINK-16143
> URL: https://issues.apache.org/jira/browse/FLINK-16143
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Planner
>Reporter: Zili Chen
>Priority: Major
> Fix For: 1.11.0
>
>
> Currently blink planner has a series of built-in functions such as
> DATEDIFF
> DATE_ADD
> DATE_SUB
> which haven't been into used so far. We could add the necessary register,
> generate and convert code to make it available in production scope.
>
> what do you think [~jark]?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] jingwen-ywb opened a new pull request #11136: 修改执行命令行路径错误
jingwen-ywb opened a new pull request #11136: 修改执行命令行路径错误 URL: https://github.com/apache/flink/pull/11136 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11136: 修改执行命令行路径错误
flinkbot commented on issue #11136: 修改执行命令行路径错误 URL: https://github.com/apache/flink/pull/11136#issuecomment-588174107 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 14cf67fe033a5bd9e3bbe2fb87b93e7e2bd6cd1a (Wed Feb 19 11:29:16 UTC 2020) **Warnings:** * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11131: [FLINK-16162][build] Bump flink-shaded to 10.0
flinkbot edited a comment on issue #11131: [FLINK-16162][build] Bump flink-shaded to 10.0 URL: https://github.com/apache/flink/pull/11131#issuecomment-588143394 ## CI report: * 3fc1d46420dfd3b316cb6639af7c861f5dac5653 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149602116) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5304) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11134: [FLINK-15833] Fix 'Kerberized YARN on Docker' e2e test
flinkbot edited a comment on issue #11134: [FLINK-15833] Fix 'Kerberized YARN on Docker' e2e test URL: https://github.com/apache/flink/pull/11134#issuecomment-588160681 ## CI report: * d1c300925c2e102650311664585ad63425ab7d37 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/149608459) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11133: [FLINK-15716] Update WindowOperatorMigrationTest to restore from 1.10 savepoint
flinkbot edited a comment on issue #11133: [FLINK-15716] Update WindowOperatorMigrationTest to restore from 1.10 savepoint URL: https://github.com/apache/flink/pull/11133#issuecomment-588160633 ## CI report: * 1f683ffa3d28b0d367dafe4ad3ebac4c3582e378 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/149608426) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11132: [FLINK-16164][build] Disable maven-site-plugin
flinkbot edited a comment on issue #11132: [FLINK-16164][build] Disable maven-site-plugin URL: https://github.com/apache/flink/pull/11132#issuecomment-588143489 ## CI report: * 0e171258e4aa6e35dd4d685fff401ce49c829094 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149602147) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5305) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11135: [FLINK-7727] [REST] Improve error handling in StaticFileServerHandlers
flinkbot commented on issue #11135: [FLINK-7727] [REST] Improve error handling in StaticFileServerHandlers URL: https://github.com/apache/flink/pull/11135#issuecomment-588176266 ## CI report: * 8fd95233a26eb69eeb7947122335098ec718ba4b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-16167) update python_shell document execution
yuwenbing created FLINK-16167: - Summary: update python_shell document execution Key: FLINK-16167 URL: https://issues.apache.org/jira/browse/FLINK-16167 Project: Flink Issue Type: Improvement Components: Documentation Reporter: yuwenbing Hi gays , [https://ci.apache.org/projects/flink/flink-docs-master/ops/python_shell.html] , the execution "bin/pyflink-shell.sh local" above the line "in the root directory of your binary Flink directory. To run the Shell on a cluster, please see the Setup section below." needs to be updated to "pyflink-shell.sh local" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15959) Add min/max number of slots configuration to limit total number of slots
[ https://issues.apache.org/jira/browse/FLINK-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039932#comment-17039932 ] YufeiLiu commented on FLINK-15959: -- [~xintongsong] Sorry, I was wrong about that, it won't get worse no matter how many times it's been restarted, only affect by the number of lost TMs. Sorry about wasting your time. > Add min/max number of slots configuration to limit total number of slots > > > Key: FLINK-15959 > URL: https://issues.apache.org/jira/browse/FLINK-15959 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.11.0 >Reporter: YufeiLiu >Priority: Major > > Flink removed `-n` option after FLIP-6, change to ResourceManager start a new > worker when required. But I think maintain a certain amount of slots is > necessary. These workers will start immediately when ResourceManager starts > and would not release even if all slots are free. > Here are some resons: > # Users actually know how many resources are needed when run a single job, > initialize all workers when cluster starts can speed up startup process. > # Job schedule in topology order, next operator won't schedule until prior > execution slot allocated. The TaskExecutors will start in several batchs in > some cases, it might slow down the startup speed. > # Flink support > [FLINK-12122|https://issues.apache.org/jira/browse/FLINK-12122] [Spread out > tasks evenly across all available registered TaskManagers], but it will only > effect if all TMs are registered. Start all TMs at begining can slove this > problem. > *suggestion:* > * Add config "taskmanager.minimum.numberOfTotalSlots" and > "taskmanager.maximum.numberOfTotalSlots", default behavior is still like > before. > * Start plenty number of workers to satisfy minimum slots when > ResourceManager accept leadership(subtract recovered workers). > * Don't comlete slot request until minimum number of slots are registered, > and throw exeception when exceed maximum. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] AHeise commented on issue #11117: [FLINK-16115][filesystem] Make aliyun oss filesystem could work with plugin mechanism
AHeise commented on issue #7: [FLINK-16115][filesystem] Make aliyun oss filesystem could work with plugin mechanism URL: https://github.com/apache/flink/pull/7#issuecomment-588183595 Then LGTM and ready to be merged from my side @zentol . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-16105) Translate "User-defined Sources & Sinks" page of "Table API & SQL" into Chinese
[
https://issues.apache.org/jira/browse/FLINK-16105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-16105:
---
Assignee: Benchao Li
> Translate "User-defined Sources & Sinks" page of "Table API & SQL" into
> Chinese
>
>
> Key: FLINK-16105
> URL: https://issues.apache.org/jira/browse/FLINK-16105
> Project: Flink
> Issue Type: Sub-task
> Components: chinese-translation, Documentation
>Reporter: Jark Wu
>Assignee: Benchao Li
>Priority: Major
>
> The page url is
> https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/sourceSinks.html
> The markdown file is located in {{flink/docs/dev/table/sourceSinks.zh.md}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-16094) Translate "User-defined Functions" page of "Functions" into Chinese
[
https://issues.apache.org/jira/browse/FLINK-16094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-16094:
---
Assignee: Benchao Li
> Translate "User-defined Functions" page of "Functions" into Chinese
>
>
> Key: FLINK-16094
> URL: https://issues.apache.org/jira/browse/FLINK-16094
> Project: Flink
> Issue Type: Sub-task
> Components: chinese-translation, Documentation
>Reporter: Jark Wu
>Assignee: Benchao Li
>Priority: Major
>
> The page url is
> https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/functions/udfs.html
> The markdown file is located in {{flink/docs/dev/table/functions/udfs.zh.md}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16168) Add build debugging profile
Chesnay Schepler created FLINK-16168: Summary: Add build debugging profile Key: FLINK-16168 URL: https://issues.apache.org/jira/browse/FLINK-16168 Project: Flink Issue Type: Improvement Components: Build System Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.11.0 We should add a profile for debugging build. An initial version could print the dependency tree and list the currently active profiles. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16142) Memory Leak causes Metaspace OOM error on repeated job submission
[
https://issues.apache.org/jira/browse/FLINK-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039939#comment-17039939
]
Piotr Nowojski commented on FLINK-16142:
If you can not take a memory dump on OOM, can you try to attach some memory
profiler to a running TM? I think things like VisualVM allows you to compare
two memory snapshots - take one after first job submission, restart job enough
number of times to "almost" trigger the issue (couple? dozen? tens? hundreds?)
and take the second snapshot. Probably ideally it would be to take the
snapshots when no job is running.
> Memory Leak causes Metaspace OOM error on repeated job submission
> -
>
> Key: FLINK-16142
> URL: https://issues.apache.org/jira/browse/FLINK-16142
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Affects Versions: 1.10.0
>Reporter: Thomas Wozniakowski
>Priority: Blocker
> Fix For: 1.10.1, 1.11.0
>
>
> Hi Guys,
> We've just tried deploying 1.10.0 as it has lots of shiny stuff that fits our
> use-case exactly (RocksDB state backend running in a containerised cluster).
> Unfortunately, it seems like there is a memory leak somewhere in the job
> submission logic. We are getting this error:
> {code:java}
> 2020-02-18 10:22:10,020 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph - OPERATOR_NAME
> switched from RUNNING to FAILED.
> java.lang.OutOfMemoryError: Metaspace
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
> at
> org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.jmx.SdkMBeanRegistrySupport.registerMetricAdminMBean(SdkMBeanRegistrySupport.java:27)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.metrics.AwsSdkMetrics.registerMetricAdminMBean(AwsSdkMetrics.java:398)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.metrics.AwsSdkMetrics.(AwsSdkMetrics.java:359)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.AmazonWebServiceClient.requestMetricCollector(AmazonWebServiceClient.java:728)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.AmazonWebServiceClient.isRMCEnabledAtClientOrSdkLevel(AmazonWebServiceClient.java:660)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.AmazonWebServiceClient.isRequestMetricsEnabled(AmazonWebServiceClient.java:652)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.AmazonWebServiceClient.createExecutionContext(AmazonWebServiceClient.java:611)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.AmazonWebServiceClient.createExecutionContext(AmazonWebServiceClient.java:606)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.executeListShards(AmazonKinesisClient.java:1534)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.listShards(AmazonKinesisClient.java:1528)
> at
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.listShards(KinesisProxy.java:439)
> at
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getShardsOfStream(KinesisProxy.java:389)
> at
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getShardList(KinesisProxy.java:279)
> at
> org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.discoverNewShardsToSubscribe(KinesisDataFetcher.java:686)
> at
> org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.run(FlinkKinesisConsumer.java:287)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
> {code}
> (The only change in the above text is the OPERATOR_NAME text where I removed
> some of the internal specifics of our system).
> This will reliably happen on a fresh cluster after submitting and cancelling
> our job 3 times.
> We are using the presto-s3 plugin, the CEP library and the Kinesis connector.
> Please let me know what other diagnostics would be useful.
> Tom
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zentol opened a new pull request #11137: [FLINK-16168][build] Add build-debug profile
zentol opened a new pull request #11137: [FLINK-16168][build] Add build-debug profile URL: https://github.com/apache/flink/pull/11137 Adds a `debug-build` profile for convenience that prints a dependency tree and shows active profiles. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16168) Add build debugging profile
[ https://issues.apache.org/jira/browse/FLINK-16168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-16168: --- Labels: pull-request-available (was: ) > Add build debugging profile > --- > > Key: FLINK-16168 > URL: https://issues.apache.org/jira/browse/FLINK-16168 > Project: Flink > Issue Type: Improvement > Components: Build System >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Minor > Labels: pull-request-available > Fix For: 1.11.0 > > > We should add a profile for debugging build. > An initial version could print the dependency tree and list the currently > active profiles. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #11137: [FLINK-16168][build] Add build-debug profile
flinkbot commented on issue #11137: [FLINK-16168][build] Add build-debug profile URL: https://github.com/apache/flink/pull/11137#issuecomment-588191200 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 095d1c7d7b8aee7ea2e094ad17ef81c620dc8158 (Wed Feb 19 12:06:49 UTC 2020) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11131: [FLINK-16162][build] Bump flink-shaded to 10.0
flinkbot edited a comment on issue #11131: [FLINK-16162][build] Bump flink-shaded to 10.0 URL: https://github.com/apache/flink/pull/11131#issuecomment-588143394 ## CI report: * 3fc1d46420dfd3b316cb6639af7c861f5dac5653 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149602116) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5304) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11135: [FLINK-7727] [REST] Improve error handling in StaticFileServerHandlers
flinkbot edited a comment on issue #11135: [FLINK-7727] [REST] Improve error handling in StaticFileServerHandlers URL: https://github.com/apache/flink/pull/11135#issuecomment-588176266 ## CI report: * 8fd95233a26eb69eeb7947122335098ec718ba4b Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/149614204) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5312) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11133: [FLINK-15716] Update WindowOperatorMigrationTest to restore from 1.10 savepoint
flinkbot edited a comment on issue #11133: [FLINK-15716] Update WindowOperatorMigrationTest to restore from 1.10 savepoint URL: https://github.com/apache/flink/pull/11133#issuecomment-588160633 ## CI report: * 1f683ffa3d28b0d367dafe4ad3ebac4c3582e378 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149608426) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
