[GitHub] [flink] Tartarus0zm commented on pull request #12931: [FLINK-18636][rest] Asynchronous close via configurable timeout in seconds for CompletedOperationCache
Tartarus0zm commented on pull request #12931: URL: https://github.com/apache/flink/pull/12931#issuecomment-663808206 @tillrohrmann thank you for your reply! In our company, sometimes user cancel flink per-job long time(about 5 min),the jobmaster has no log,but not exit. after 5 min,we see a timeout message,so i find this code. ``` java.util.concurrent.TimeoutException at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:999) at org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211) at org.apache.flink.runtime.concurrent.FutureUtils.lambda$orTimeout$14(FutureUtils.java:427) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ``` Because we prohibit users from resubmitting. So let the user decide whether the timeout period is better This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18687) ProjectionCodeGenerator#generateProjectionExpression should remove for loop optimization
[
https://issues.apache.org/jira/browse/FLINK-18687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-18687:
Fix Version/s: (was: 1.12.0)
> ProjectionCodeGenerator#generateProjectionExpression should remove for loop
> optimization
>
>
> Key: FLINK-18687
> URL: https://issues.apache.org/jira/browse/FLINK-18687
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.0
>Reporter: Caizhi Weng
>Priority: Critical
>
> If too many fields of the same type are projected,
> {{ProjectionCodeGenerator#generateProjectionExpression}} currently performs a
> "for loop optimization" which, instead of generating code separately for each
> field, they'll be squashed into a for loop.
> However, if the indices of the fields with the same type are not continuous,
> this optimization will not write fields in index ascending order. This is not
> acceptable because {{BinaryWriter}}s expect the users to write fields in
> index ascending order (that is to say, we *have to* first write field 0, then
> field 1, then...), otherwise the variable length area of the two binary rows
> with same data might be different. Although we can use {{getXX}} methods of
> {{BinaryRow}} to get the fields correctly, states for streaming jobs compare
> state keys with binary bits, not with the contents of the keys. So we need to
> make sure the binary bits of the binary rows be the same if two rows contain
> the same data.
> What's worse, as the current implementation of
> {{ProjectionCodeGenerator#generateProjectionExpression}} uses a scala
> {{HashMap}}, the key order of the map might be different on different
> workers; Even if the projection does not meet the condition to be optimized,
> it will still be affected by this bug.
> What I suggest is to simply remove this optimization. Because if we still
> want this optimization, we have to make sure that the fields of the same type
> have continuous order, which is a very strict and rare condition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16947) ArtifactResolutionException: Could not transfer artifact. Entry [...] has not been leased from this pool
[
https://issues.apache.org/jira/browse/FLINK-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164741#comment-17164741
]
Dian Fu commented on FLINK-16947:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4900=logs=8fd975ef-f478-511d-4997-6f15fe8a1fd3=ac0fa443-5d45-5a6b-3597-0310ecc1d2ab]
> ArtifactResolutionException: Could not transfer artifact. Entry [...] has
> not been leased from this pool
> -
>
> Key: FLINK-16947
> URL: https://issues.apache.org/jira/browse/FLINK-16947
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Reporter: Piotr Nowojski
>Assignee: Robert Metzger
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6982=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> Build of flink-metrics-availability-test failed with:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (end-to-end-tests)
> on project flink-metrics-availability-test: Unable to generate classpath:
> org.apache.maven.artifact.resolver.ArtifactResolutionException: Could not
> transfer artifact org.apache.maven.surefire:surefire-grouper:jar:2.22.1
> from/to google-maven-central
> (https://maven-central-eu.storage-download.googleapis.com/maven2/): Entry
> [id:13][route:{s}->https://maven-central-eu.storage-download.googleapis.com:443][state:null]
> has not been leased from this pool
> [ERROR] org.apache.maven.surefire:surefire-grouper:jar:2.22.1
> [ERROR]
> [ERROR] from the specified remote repositories:
> [ERROR] google-maven-central
> (https://maven-central-eu.storage-download.googleapis.com/maven2/,
> releases=true, snapshots=false),
> [ERROR] apache.snapshots (https://repository.apache.org/snapshots,
> releases=false, snapshots=true)
> [ERROR] Path to dependency:
> [ERROR] 1) dummy:dummy:jar:1.0
> [ERROR] 2) org.apache.maven.surefire:surefire-junit47:jar:2.22.1
> [ERROR] 3) org.apache.maven.surefire:common-junit48:jar:2.22.1
> [ERROR] 4) org.apache.maven.surefire:surefire-grouper:jar:2.22.1
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :flink-metrics-availability-test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17274) Maven: Premature end of Content-Length delimited message body
[
https://issues.apache.org/jira/browse/FLINK-17274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164740#comment-17164740
]
Dian Fu commented on FLINK-17274:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4900=logs=6bfdaf55-0c08-5e3f-a2d2-2a0285fd41cf=fd9796c3-9ce8-5619-781c-42f873e126a6]
> Maven: Premature end of Content-Length delimited message body
> -
>
> Key: FLINK-17274
> URL: https://issues.apache.org/jira/browse/FLINK-17274
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Critical
> Fix For: 1.12.0
>
>
> CI:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=7786=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb
> {code}
> [ERROR] Failed to execute goal on project
> flink-connector-elasticsearch7_2.11: Could not resolve dependencies for
> project
> org.apache.flink:flink-connector-elasticsearch7_2.11:jar:1.11-SNAPSHOT: Could
> not transfer artifact org.apache.lucene:lucene-sandbox:jar:8.3.0 from/to
> alicloud-mvn-mirror
> (http://mavenmirror.alicloud.dak8s.net:/repository/maven-central/): GET
> request of: org/apache/lucene/lucene-sandbox/8.3.0/lucene-sandbox-8.3.0.jar
> from alicloud-mvn-mirror failed: Premature end of Content-Length delimited
> message body (expected: 289920; received: 239832 -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17949) KafkaShuffleITCase.testSerDeIngestionTime:156->testRecordSerDe:388 expected:<310> but was:<0>
[
https://issues.apache.org/jira/browse/FLINK-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164739#comment-17164739
]
Dian Fu commented on FLINK-17949:
-
Another instance on 1.11:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4901=logs=ce8f3cc3-c1ea-5281-f5eb-df9ebd24947f=d4549d78-6fab-5c0c-bdb9-abaafb66ea8b
{code}
2020-07-24T21:25:35.2060012Z java.lang.AssertionError: expected:<343> but
was:<0>
2020-07-24T21:25:35.2060323Zat org.junit.Assert.fail(Assert.java:88)
2020-07-24T21:25:35.2060638Zat
org.junit.Assert.failNotEquals(Assert.java:834)
2020-07-24T21:25:35.2060951Zat
org.junit.Assert.assertEquals(Assert.java:645)
2020-07-24T21:25:35.2061277Zat
org.junit.Assert.assertEquals(Assert.java:631)
2020-07-24T21:25:35.2061710Zat
org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleITCase.testRecordSerDe(KafkaShuffleITCase.java:388)
2020-07-24T21:25:35.2062309Zat
org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleITCase.testSerDeProcessingTime(KafkaShuffleITCase.java:146)
{code}
> KafkaShuffleITCase.testSerDeIngestionTime:156->testRecordSerDe:388
> expected:<310> but was:<0>
> -
>
> Key: FLINK-17949
> URL: https://issues.apache.org/jira/browse/FLINK-17949
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Attachments: logs-ci-kafkagelly-1590500380.zip,
> logs-ci-kafkagelly-1590524911.zip
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2209=logs=c5f0071e-1851-543e-9a45-9ac140befc32=684b1416-4c17-504e-d5ab-97ee44e08a20
> {code}
> 2020-05-26T13:35:19.4022562Z [ERROR]
> testSerDeIngestionTime(org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleITCase)
> Time elapsed: 5.786 s <<< FAILURE!
> 2020-05-26T13:35:19.4023185Z java.lang.AssertionError: expected:<310> but
> was:<0>
> 2020-05-26T13:35:19.4023498Z at org.junit.Assert.fail(Assert.java:88)
> 2020-05-26T13:35:19.4023825Z at
> org.junit.Assert.failNotEquals(Assert.java:834)
> 2020-05-26T13:35:19.4024461Z at
> org.junit.Assert.assertEquals(Assert.java:645)
> 2020-05-26T13:35:19.4024900Z at
> org.junit.Assert.assertEquals(Assert.java:631)
> 2020-05-26T13:35:19.4028546Z at
> org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleITCase.testRecordSerDe(KafkaShuffleITCase.java:388)
> 2020-05-26T13:35:19.4029629Z at
> org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleITCase.testSerDeIngestionTime(KafkaShuffleITCase.java:156)
> 2020-05-26T13:35:19.4030253Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-05-26T13:35:19.4030673Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-05-26T13:35:19.4031332Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-05-26T13:35:19.4031763Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-05-26T13:35:19.4032155Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-05-26T13:35:19.4032630Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-05-26T13:35:19.4033188Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-05-26T13:35:19.4033638Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-05-26T13:35:19.4034103Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2020-05-26T13:35:19.4034593Z at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> 2020-05-26T13:35:19.4035118Z at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> 2020-05-26T13:35:19.4035570Z at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2020-05-26T13:35:19.4035888Z at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-18122) Kubernetes test fails with "error: timed out waiting for the condition on jobs/flink-job-cluster"
[
https://issues.apache.org/jira/browse/FLINK-18122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164738#comment-17164738
]
Dian Fu edited comment on FLINK-18122 at 7/25/20, 4:12 AM:
---
Another instance on 1.11:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4901=logs=08866332-78f7-59e4-4f7e-49a56faa3179=3e8647c1-5a28-5917-dd93-bf78594ea994
was (Author: dian.fu):
Another instance on 1.11:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2697=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> Kubernetes test fails with "error: timed out waiting for the condition on
> jobs/flink-job-cluster"
> -
>
> Key: FLINK-18122
> URL: https://issues.apache.org/jira/browse/FLINK-18122
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2697=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-04T09:25:40.7205843Z service/flink-job-cluster created
> 2020-06-04T09:25:40.9661515Z job.batch/flink-job-cluster created
> 2020-06-04T09:25:41.2189123Z deployment.apps/flink-task-manager created
> 2020-06-04T10:32:32.6402983Z error: timed out waiting for the condition on
> jobs/flink-job-cluster
> 2020-06-04T10:32:33.8057757Z error: unable to upgrade connection: container
> not found ("flink-task-manager")
> 2020-06-04T10:32:33.8111302Z sort: cannot read:
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*':
> No such file or directory
> 2020-06-04T10:32:33.8124455Z FAIL WordCount: Output hash mismatch. Got
> d41d8cd98f00b204e9800998ecf8427e, expected e682ec6622b5e83f2eb614617d5ab2cf.
> 2020-06-04T10:32:33.8125379Z head hexdump of actual:
> 2020-06-04T10:32:33.8136133Z head: cannot open
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*'
> for reading: No such file or directory
> 2020-06-04T10:32:33.8344715Z Debugging failed Kubernetes test:
> 2020-06-04T10:32:33.8345469Z Currently existing Kubernetes resources
> 2020-06-04T10:32:36.4977853Z I0604 10:32:36.497383 13191 request.go:621]
> Throttling request took 1.198606989s, request:
> GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:32:46.6975735Z I0604 10:32:46.697234 13191 request.go:621]
> Throttling request took 4.398107353s, request:
> GET:https://10.1.0.4:8443/apis/authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:32:57.4978637Z I0604 10:32:57.497209 13191 request.go:621]
> Throttling request took 1.198449167s, request:
> GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
> 2020-06-04T10:33:07.4980104Z I0604 10:33:07.497320 13191 request.go:621]
> Throttling request took 4.198274438s, request:
> GET:https://10.1.0.4:8443/apis/apiextensions.k8s.io/v1?timeout=32s
> 2020-06-04T10:33:18.4976060Z I0604 10:33:18.497258 13191 request.go:621]
> Throttling request took 1.19871495s, request:
> GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
> 2020-06-04T10:33:28.4979129Z I0604 10:33:28.497276 13191 request.go:621]
> Throttling request took 4.198369672s, request:
> GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:33:30.9182069Z NAME READY
> STATUS RESTARTS AGE
> 2020-06-04T10:33:30.9184099Z pod/flink-job-cluster-dtb67 0/1
> ErrImageNeverPull 0 67m
> 2020-06-04T10:33:30.9184869Z pod/flink-task-manager-74ccc9bd9-psqwm 0/1
> ErrImageNeverPull 0 67m
> 2020-06-04T10:33:30.9185226Z
> 2020-06-04T10:33:30.9185926Z NAMETYPE
> CLUSTER-IP EXTERNAL-IP PORT(S)
> AGE
> 2020-06-04T10:33:30.9186832Z service/flink-job-cluster NodePort
> 10.111.92.199
> 6123:32501/TCP,6124:31360/TCP,6125:30025/TCP,8081:30081/TCP 67m
> 2020-06-04T10:33:30.9187545Z service/kubernetes ClusterIP
> 10.96.0.1 443/TCP
> 68m
> 2020-06-04T10:33:30.9187976Z
> 2020-06-04T10:33:30.9188472Z NAME READY
> UP-TO-DATE AVAILABLE AGE
> 2020-06-04T10:33:30.9189179Z deployment.apps/flink-task-manager 0/1 1
> 0 67m
> 2020-06-04T10:33:30.9189508Z
> 2020-06-04T10:33:30.9189815Z NAME
> DESIRED CURRENT
[jira] [Commented] (FLINK-18122) Kubernetes test fails with "error: timed out waiting for the condition on jobs/flink-job-cluster"
[
https://issues.apache.org/jira/browse/FLINK-18122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164738#comment-17164738
]
Dian Fu commented on FLINK-18122:
-
Another instance on 1.11:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2697=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> Kubernetes test fails with "error: timed out waiting for the condition on
> jobs/flink-job-cluster"
> -
>
> Key: FLINK-18122
> URL: https://issues.apache.org/jira/browse/FLINK-18122
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2697=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-04T09:25:40.7205843Z service/flink-job-cluster created
> 2020-06-04T09:25:40.9661515Z job.batch/flink-job-cluster created
> 2020-06-04T09:25:41.2189123Z deployment.apps/flink-task-manager created
> 2020-06-04T10:32:32.6402983Z error: timed out waiting for the condition on
> jobs/flink-job-cluster
> 2020-06-04T10:32:33.8057757Z error: unable to upgrade connection: container
> not found ("flink-task-manager")
> 2020-06-04T10:32:33.8111302Z sort: cannot read:
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*':
> No such file or directory
> 2020-06-04T10:32:33.8124455Z FAIL WordCount: Output hash mismatch. Got
> d41d8cd98f00b204e9800998ecf8427e, expected e682ec6622b5e83f2eb614617d5ab2cf.
> 2020-06-04T10:32:33.8125379Z head hexdump of actual:
> 2020-06-04T10:32:33.8136133Z head: cannot open
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*'
> for reading: No such file or directory
> 2020-06-04T10:32:33.8344715Z Debugging failed Kubernetes test:
> 2020-06-04T10:32:33.8345469Z Currently existing Kubernetes resources
> 2020-06-04T10:32:36.4977853Z I0604 10:32:36.497383 13191 request.go:621]
> Throttling request took 1.198606989s, request:
> GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:32:46.6975735Z I0604 10:32:46.697234 13191 request.go:621]
> Throttling request took 4.398107353s, request:
> GET:https://10.1.0.4:8443/apis/authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:32:57.4978637Z I0604 10:32:57.497209 13191 request.go:621]
> Throttling request took 1.198449167s, request:
> GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
> 2020-06-04T10:33:07.4980104Z I0604 10:33:07.497320 13191 request.go:621]
> Throttling request took 4.198274438s, request:
> GET:https://10.1.0.4:8443/apis/apiextensions.k8s.io/v1?timeout=32s
> 2020-06-04T10:33:18.4976060Z I0604 10:33:18.497258 13191 request.go:621]
> Throttling request took 1.19871495s, request:
> GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
> 2020-06-04T10:33:28.4979129Z I0604 10:33:28.497276 13191 request.go:621]
> Throttling request took 4.198369672s, request:
> GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:33:30.9182069Z NAME READY
> STATUS RESTARTS AGE
> 2020-06-04T10:33:30.9184099Z pod/flink-job-cluster-dtb67 0/1
> ErrImageNeverPull 0 67m
> 2020-06-04T10:33:30.9184869Z pod/flink-task-manager-74ccc9bd9-psqwm 0/1
> ErrImageNeverPull 0 67m
> 2020-06-04T10:33:30.9185226Z
> 2020-06-04T10:33:30.9185926Z NAMETYPE
> CLUSTER-IP EXTERNAL-IP PORT(S)
> AGE
> 2020-06-04T10:33:30.9186832Z service/flink-job-cluster NodePort
> 10.111.92.199
> 6123:32501/TCP,6124:31360/TCP,6125:30025/TCP,8081:30081/TCP 67m
> 2020-06-04T10:33:30.9187545Z service/kubernetes ClusterIP
> 10.96.0.1 443/TCP
> 68m
> 2020-06-04T10:33:30.9187976Z
> 2020-06-04T10:33:30.9188472Z NAME READY
> UP-TO-DATE AVAILABLE AGE
> 2020-06-04T10:33:30.9189179Z deployment.apps/flink-task-manager 0/1 1
> 0 67m
> 2020-06-04T10:33:30.9189508Z
> 2020-06-04T10:33:30.9189815Z NAME
> DESIRED CURRENT READY AGE
> 2020-06-04T10:33:30.9190418Z replicaset.apps/flink-task-manager-74ccc9bd9 1
> 1 0 67m
> 2020-06-04T10:33:30.9190662Z
> 2020-06-04T10:33:30.9190891Z NAME COMPLETIONS
> DURATION AGE
>
[GitHub] [flink] flinkbot edited a comment on pull request #12919: [FLINK-16048][avro] Support read/write confluent schema registry avro…
flinkbot edited a comment on pull request #12919: URL: https://github.com/apache/flink/pull/12919#issuecomment-659928275 ## CI report: * d1e4ba7690be134e21d193fbc1cb01aa51aaeb9b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4703) * f0da3cee91d22ec20cbba1b6c5be45da1440cf05 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4713) * 9104e12b0394cd6d578d2380ca4554b75e6e00f9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4730) * 9b557c718fe731e8d5c58e7c5d9c3452a245ee5a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4769) * 71218ee49095663a641e56889831536a2a2e69ea Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4818) * 52518eecfce65f5adceda689fa720f15c85413b6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4902) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17510) StreamingKafkaITCase. testKafka timeouts on downloading Kafka
[
https://issues.apache.org/jira/browse/FLINK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164735#comment-17164735
]
Dian Fu commented on FLINK-17510:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4897=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529]
SQLClientHBaseITCase.testHBase failed with the similar reason:
{code}
[ERROR] testHBase(org.apache.flink.tests.util.hbase.SQLClientHBaseITCase) Time
elapsed: 120.022 s <<< ERROR!
java.io.IOException: Process ([wget, -q, -P,
/home/vsts/work/1/e2e_cache/downloads/1598516010,
https://archive.apache.org/dist/hbase/1.4.3/hbase-1.4.3-bin.tar.gz]) exceeded
timeout (12) or number of retries (3).
{code}
> StreamingKafkaITCase. testKafka timeouts on downloading Kafka
> -
>
> Key: FLINK-17510
> URL: https://issues.apache.org/jira/browse/FLINK-17510
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Connectors / Kafka, Tests
>Reporter: Robert Metzger
>Priority: Major
>
> CI:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=585=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-05-05T00:06:49.7268716Z [INFO]
> ---
> 2020-05-05T00:06:49.7268938Z [INFO] T E S T S
> 2020-05-05T00:06:49.7269282Z [INFO]
> ---
> 2020-05-05T00:06:50.5336315Z [INFO] Running
> org.apache.flink.tests.util.kafka.StreamingKafkaITCase
> 2020-05-05T00:11:26.8603439Z [ERROR] Tests run: 3, Failures: 0, Errors: 2,
> Skipped: 0, Time elapsed: 276.323 s <<< FAILURE! - in
> org.apache.flink.tests.util.kafka.StreamingKafkaITCase
> 2020-05-05T00:11:26.8604882Z [ERROR] testKafka[1:
> kafka-version:0.11.0.2](org.apache.flink.tests.util.kafka.StreamingKafkaITCase)
> Time elapsed: 120.024 s <<< ERROR!
> 2020-05-05T00:11:26.8605942Z java.io.IOException: Process ([wget, -q, -P,
> /tmp/junit2815750531595874769/downloads/1290570732,
> https://archive.apache.org/dist/kafka/0.11.0.2/kafka_2.11-0.11.0.2.tgz])
> exceeded timeout (12) or number of retries (3).
> 2020-05-05T00:11:26.8606732Z at
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlockingWithRetry(AutoClosableProcess.java:132)
> 2020-05-05T00:11:26.8607321Z at
> org.apache.flink.tests.util.cache.AbstractDownloadCache.getOrDownload(AbstractDownloadCache.java:127)
> 2020-05-05T00:11:26.8607826Z at
> org.apache.flink.tests.util.cache.LolCache.getOrDownload(LolCache.java:31)
> 2020-05-05T00:11:26.8608343Z at
> org.apache.flink.tests.util.kafka.LocalStandaloneKafkaResource.setupKafkaDist(LocalStandaloneKafkaResource.java:98)
> 2020-05-05T00:11:26.8608892Z at
> org.apache.flink.tests.util.kafka.LocalStandaloneKafkaResource.before(LocalStandaloneKafkaResource.java:92)
> 2020-05-05T00:11:26.8609602Z at
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:46)
> 2020-05-05T00:11:26.8610026Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-05-05T00:11:26.8610553Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-05-05T00:11:26.8610958Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-05-05T00:11:26.8611388Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-05-05T00:11:26.8612214Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-05-05T00:11:26.8612706Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-05-05T00:11:26.8613109Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-05-05T00:11:26.8613551Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-05-05T00:11:26.8614019Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-05-05T00:11:26.8614442Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-05-05T00:11:26.8614869Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-05-05T00:11:26.8615251Z at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2020-05-05T00:11:26.8615654Z at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2020-05-05T00:11:26.8616060Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-05-05T00:11:26.8616465Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-05-05T00:11:26.8616893Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-05-05T00:11:26.8617893Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-05-05T00:11:26.8618490Z at
>
[jira] [Created] (FLINK-18714) Support consuming multiple topics for KafkaTableSource
hailong wang created FLINK-18714:
Summary: Support consuming multiple topics for KafkaTableSource
Key: FLINK-18714
URL: https://issues.apache.org/jira/browse/FLINK-18714
Project: Flink
Issue Type: Improvement
Components: Table SQL / Planner
Affects Versions: 1.11.0
Reporter: hailong wang
Fix For: 1.12.0
SQL one kafka table only supports one topic by `{{'topic' = 'topic1'}}` or
`{{'connector.topic' = 'topic1'}}`.
We can support consuming multiple topics in one kafka table.
My idea is that We can concatenate multi topic with comma just like,
`{{'topic' = 'topic1,topic2,topic3'}}`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18713) Allow default ms unit for table.exec.mini-batch.allow-latency etc.
hailong wang created FLINK-18713:
Summary: Allow default ms unit for
table.exec.mini-batch.allow-latency etc.
Key: FLINK-18713
URL: https://issues.apache.org/jira/browse/FLINK-18713
Project: Flink
Issue Type: Improvement
Components: Table SQL / Planner
Affects Versions: 1.11.0
Reporter: hailong wang
Fix For: 1.12.0
We use `scala.concurrent.duration.Duration.create` to parse timeStr in
`TableConfigUtils#
getMillisecondFromConfigDuration` for the following properties,
{code:java}
table.exec.async-lookup.timeout
table.exec.source.idle-timeout
table.exec.mini-batch.allow-latency
table.exec.emit.early-fire.delay
table.exec.emit.late-fire.delay{code}
And it must has the unit.
I think we can replace it with `TimeUtils.parseDuration(timeStr)` to parse
timeStr just like `DescriptorProperties#getOptionalDuration` to has default ms
unit and be consistent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18712) Flink RocksDB statebackend memory leak issue
Farnight created FLINK-18712: Summary: Flink RocksDB statebackend memory leak issue Key: FLINK-18712 URL: https://issues.apache.org/jira/browse/FLINK-18712 Project: Flink Issue Type: Bug Components: Runtime / State Backends Affects Versions: 1.10.0 Reporter: Farnight When using RocksDB as our statebackend, we found it will lead to memory leak when restarting job (manually or in recovery case). How to reproduce: # increase RocksDB blockcache size(e.g. 1G), it is easier to monitor and reproduce. # start a job using RocksDB statebackend. # when the RocksDB blockcache reachs maximum size, restart the job. and monitor the memory usage (k8s pod working set) of the TM. # go through step 2-3 few more times. and memory will keep raising. Any solution or suggestion for this? Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18711) Trim the blank space for TableOptions
hailong wang created FLINK-18711:
Summary: Trim the blank space for TableOptions
Key: FLINK-18711
URL: https://issues.apache.org/jira/browse/FLINK-18711
Project: Flink
Issue Type: Improvement
Components: Table SQL / Planner
Affects Versions: 1.11.0
Reporter: hailong wang
Fix For: 1.12.0
If ddl table options have space before or after key or value, it will fail to
create table source.
For example,
{code:java}
"CREATE TABLE " + INPUT_TABLE + "(" +
"id BIGINT," +
"timestamp6_col TIMESTAMP(6)," +
"timestamp9_col TIMESTAMP(9)," +
"time_col TIME," +
"real_col FLOAT," +
"double_col DOUBLE," +
"decimal_col DECIMAL(10, 4)" +
") WITH (" +
" 'connector'=' jdbc'," +
" 'url '='" + DB_URL + "'," +
" 'table-name'='" + INPUT_TABLE + "'" +
")"
{code}
If there have blank space before jdbc or blank space after url, it will fail.
I think If we can trim it, will be better for users.
My idea is we can just trim keys and values on MergeTableLikeUtil#mergeOptions
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-18686) Getting the emit time when `table.exec.emit.early-fire.enabled` is true
[ https://issues.apache.org/jira/browse/FLINK-18686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hailong wang closed FLINK-18686. Resolution: Not A Problem > Getting the emit time when `table.exec.emit.early-fire.enabled` is true > --- > > Key: FLINK-18686 > URL: https://issues.apache.org/jira/browse/FLINK-18686 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.11.0 >Reporter: hailong wang >Priority: Major > Fix For: 1.12.0 > > > We can turn `table.exec.emit.early-fire.enabled` on to let window > early-fire. But users always want to get the emit time. > So can we support auxiliary Function to support this, may be like > TUMBLE_EMIT, HOP_EMIT? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12919: [FLINK-16048][avro] Support read/write confluent schema registry avro…
flinkbot edited a comment on pull request #12919: URL: https://github.com/apache/flink/pull/12919#issuecomment-659928275 ## CI report: * d1e4ba7690be134e21d193fbc1cb01aa51aaeb9b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4703) * f0da3cee91d22ec20cbba1b6c5be45da1440cf05 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4713) * 9104e12b0394cd6d578d2380ca4554b75e6e00f9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4730) * 9b557c718fe731e8d5c58e7c5d9c3452a245ee5a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4769) * 71218ee49095663a641e56889831536a2a2e69ea Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4818) * 6bd8c02de778a8ff2f34a19d5beee414beac3f69 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4875) * 52518eecfce65f5adceda689fa720f15c85413b6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4902) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12919: [FLINK-16048][avro] Support read/write confluent schema registry avro…
flinkbot edited a comment on pull request #12919: URL: https://github.com/apache/flink/pull/12919#issuecomment-659928275 ## CI report: * d1e4ba7690be134e21d193fbc1cb01aa51aaeb9b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4703) * f0da3cee91d22ec20cbba1b6c5be45da1440cf05 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4713) * 9104e12b0394cd6d578d2380ca4554b75e6e00f9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4730) * 9b557c718fe731e8d5c58e7c5d9c3452a245ee5a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4769) * 71218ee49095663a641e56889831536a2a2e69ea Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4818) * 6bd8c02de778a8ff2f34a19d5beee414beac3f69 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4875) * 52518eecfce65f5adceda689fa720f15c85413b6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12991: [FLINK-18710] Make ResourceProfileInfo serializable
flinkbot edited a comment on pull request #12991: URL: https://github.com/apache/flink/pull/12991#issuecomment-663683249 ## CI report: * 76995e11bfa399f4f2f51b76561927c8dd959256 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4899) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman closed pull request #12973: [BP-1.11][FLINK-18341][walkthroughs] Drop remaining table walkthrough archetypes
sjwiesman closed pull request #12973: URL: https://github.com/apache/flink/pull/12973 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17073) Slow checkpoint cleanup causing OOMs
[
https://issues.apache.org/jira/browse/FLINK-17073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164623#comment-17164623
]
Etienne Chauchot commented on FLINK-17073:
--
Hi [~roman_khachatryan],
thanks for the suggestions! Overall this is what I intended to do modulo these
minor things:
In 1. I meant to use the _CompletedCheckpoints_ dequeue to keep track of the
checkpoints to clean and avoid adding a new queue
In 2. Yes indeed, it needs to call _executeQueuedRequest,_ but it needs also
not to call it when the previous checkpoint cleaning is not done (still need to
figure out how to sync cleaning with work in CheckpointCoordinator) so that
checkpoint cleaning becomes part of the checkpoint process and not a side
fire-and-forget process. This behavior will be configurable to avoid lowering
checkpoint rate when CP cleaning rate is not a problem. Once cleaning is part
of the standard checkpointing process, checking in flight checkpoints will tell
how many potential cleaning checkpoints there are and if there are too much,
drop any new CP trigger request.
In 3. yes that is what I meant in "drop any new CP trigger request" above.
In 4. I'm not clear yet about concurrency in checkpointing.
> Slow checkpoint cleanup causing OOMs
>
>
> Key: FLINK-17073
> URL: https://issues.apache.org/jira/browse/FLINK-17073
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.7.3, 1.8.0, 1.9.0, 1.10.0, 1.11.0
>Reporter: Till Rohrmann
>Assignee: Etienne Chauchot
>Priority: Major
> Fix For: 1.12.0
>
>
> A user reported that he sees a decline in checkpoint cleanup speed when
> upgrading from Flink 1.7.2 to 1.10.0. The result is that a lot of cleanup
> tasks are waiting in the execution queue occupying memory. Ultimately, the JM
> process dies with an OOM.
> Compared to Flink 1.7.2, we introduced a dedicated {{ioExecutor}} which is
> used by the {{HighAvailabilityServices}} (FLINK-11851). Before, we use the
> {{AkkaRpcService}} thread pool which was a {{ForkJoinPool}} with a max
> parallelism of 64. Now it is a {{FixedThreadPool}} with as many threads as
> CPU cores. This change might have caused the decline in completed checkpoint
> discard throughput. This suspicion needs to be validated before trying to fix
> it!
> [1]
> https://lists.apache.org/thread.html/r390e5d775878918edca0b6c9f18de96f828c266a888e34ed30ce8494%40%3Cuser.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12989: [FLINK-18656][network,metrics] Fix startDelay metric for unaligned checkpoints
flinkbot edited a comment on pull request #12989: URL: https://github.com/apache/flink/pull/12989#issuecomment-663583428 ## CI report: * 0841311adec0be29af94dad85f0e1615bfa42045 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4886) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12991: [FLINK-18710] Make ResourceProfileInfo serializable
flinkbot edited a comment on pull request #12991: URL: https://github.com/apache/flink/pull/12991#issuecomment-663683249 ## CI report: * 76995e11bfa399f4f2f51b76561927c8dd959256 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4899) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12991: [FLINK-18710] Make ResourceProfileInfo serializable
flinkbot commented on pull request #12991: URL: https://github.com/apache/flink/pull/12991#issuecomment-663683249 ## CI report: * 76995e11bfa399f4f2f51b76561927c8dd959256 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12991: [FLINK-18710] Make ResourceProfileInfo serializable
flinkbot commented on pull request #12991: URL: https://github.com/apache/flink/pull/12991#issuecomment-663679953 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 76995e11bfa399f4f2f51b76561927c8dd959256 (Fri Jul 24 18:44:12 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18710) ResourceProfileInfo is not serializable
[
https://issues.apache.org/jira/browse/FLINK-18710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-18710:
---
Labels: pull-request-available (was: )
> ResourceProfileInfo is not serializable
> ---
>
> Key: FLINK-18710
> URL: https://issues.apache.org/jira/browse/FLINK-18710
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2
>
>
> {{ResourceProfileInfo}} should be {{Serializable}} because it is sent as an
> RPC payload.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tillrohrmann opened a new pull request #12991: [FLINK-18710] Make ResourceProfileInfo serializable
tillrohrmann opened a new pull request #12991: URL: https://github.com/apache/flink/pull/12991 Make `ResourceProfileInfo` `Serializable`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18710) ResourceProfileInfo is not serializable
Till Rohrmann created FLINK-18710:
-
Summary: ResourceProfileInfo is not serializable
Key: FLINK-18710
URL: https://issues.apache.org/jira/browse/FLINK-18710
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.11.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 1.12.0, 1.11.2
{{ResourceProfileInfo}} should be {{Serializable}} because it is sent as an RPC
payload.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18620) Unify behaviors of active resource managers
[
https://issues.apache.org/jira/browse/FLINK-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164562#comment-17164562
]
Till Rohrmann commented on FLINK-18620:
---
Thanks a lot for creating the design and POC [~xintongsong]. I've given both a
pass and I think they look very good. They make the whole {{ResourceManager}}
abstraction more maintainable and evolvable. +1 for creating the subtasks
according to the implementation plan and then to move forward.
> Unify behaviors of active resource managers
> ---
>
> Key: FLINK-18620
> URL: https://issues.apache.org/jira/browse/FLINK-18620
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Xintong Song
>Assignee: Xintong Song
>Priority: Major
>
> Flink supports various deployment modes: standalone, Kubernetes, Yarn &
> Mesos. For each deployment mode, a resource manager is implemented for
> managing the resources.
> While StandaloneResourceManager is quite different from the others by not
> being able to dynamically request and release resources, the other three
> (KubernetesResourceManager, YarnResourceManager and MesosResourceManager)
> share many logics in common. These common logics are currently duplicately
> implemented by each of the active resource managers. Such duplication leads
> to extra maintaining overhead and amplifies stability risks.
> This ticket proposes a refactor design for the resource managers, with better
> abstraction deduplicating common logics implementations and minimizing the
> deployment specific behaviors.
> This proposal is a pure refactor effort. It does not intend to change any of
> the current resource management behaviors.
> A detailed design doc and a simplified proof-of-concept implementation for
> the Kubernetes deployment are linked to this ticket.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rmetzger commented on pull request #12926: [hotfix][docs] Fix the typo when describing DagConnection
rmetzger commented on pull request #12926: URL: https://github.com/apache/flink/pull/12926#issuecomment-663649627 Thanks a lot for your contribution! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger closed pull request #12926: [hotfix][docs] Fix the typo when describing DagConnection
rmetzger closed pull request #12926: URL: https://github.com/apache/flink/pull/12926 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #12776: [FLINK-Streanming] [Streaming API] fixed punctuation
rmetzger commented on pull request #12776: URL: https://github.com/apache/flink/pull/12776#issuecomment-663646899 Thanks a lot for this contribution. I'm not sure if this comma is grammatically correct there. I propose to close this PR without merging it, because it doesn't really improve Flink, and it's more overhead for us to review and merge this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18341) Building Flink Walkthrough Table Java 0.1 COMPILATION ERROR
[
https://issues.apache.org/jira/browse/FLINK-18341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seth Wiesman closed FLINK-18341.
Resolution: Fixed
> Building Flink Walkthrough Table Java 0.1 COMPILATION ERROR
> ---
>
> Key: FLINK-18341
> URL: https://issues.apache.org/jira/browse/FLINK-18341
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API, Tests
>Affects Versions: 1.12.0, 1.11.1
>Reporter: Piotr Nowojski
>Assignee: Seth Wiesman
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0, 1.11.2
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3652=logs=08866332-78f7-59e4-4f7e-49a56faa3179=931b3127-d6ee-5f94-e204-48d51cd1c334
> {noformat}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-22294375765/flink-walkthrough-table-java/src/main/java/org/apache/flink/walkthrough/SpendReport.java:[23,46]
> cannot access org.apache.flink.table.api.bridge.java.BatchTableEnvironment
> bad class file:
> /home/vsts/work/1/.m2/repository/org/apache/flink/flink-table-api-java-bridge_2.11/1.12-SNAPSHOT/flink-table-api-java-bridge_2.11-1.12-SNAPSHOT.jar(org/apache/flink/table/api/bridge/java/BatchTableEnvironment.class)
> class file has wrong version 55.0, should be 52.0
> Please remove or make sure it appears in the correct subdirectory of the
> classpath.
> (...)
> [FAIL] 'Walkthrough Table Java nightly end-to-end test' failed after 0
> minutes and 4 seconds! Test exited with exit code 1
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18341) Building Flink Walkthrough Table Java 0.1 COMPILATION ERROR
[
https://issues.apache.org/jira/browse/FLINK-18341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seth Wiesman updated FLINK-18341:
-
Fix Version/s: 1.12.0
> Building Flink Walkthrough Table Java 0.1 COMPILATION ERROR
> ---
>
> Key: FLINK-18341
> URL: https://issues.apache.org/jira/browse/FLINK-18341
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API, Tests
>Affects Versions: 1.12.0, 1.11.1
>Reporter: Piotr Nowojski
>Assignee: Seth Wiesman
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0, 1.11.2
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3652=logs=08866332-78f7-59e4-4f7e-49a56faa3179=931b3127-d6ee-5f94-e204-48d51cd1c334
> {noformat}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-22294375765/flink-walkthrough-table-java/src/main/java/org/apache/flink/walkthrough/SpendReport.java:[23,46]
> cannot access org.apache.flink.table.api.bridge.java.BatchTableEnvironment
> bad class file:
> /home/vsts/work/1/.m2/repository/org/apache/flink/flink-table-api-java-bridge_2.11/1.12-SNAPSHOT/flink-table-api-java-bridge_2.11-1.12-SNAPSHOT.jar(org/apache/flink/table/api/bridge/java/BatchTableEnvironment.class)
> class file has wrong version 55.0, should be 52.0
> Please remove or make sure it appears in the correct subdirectory of the
> classpath.
> (...)
> [FAIL] 'Walkthrough Table Java nightly end-to-end test' failed after 0
> minutes and 4 seconds! Test exited with exit code 1
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18341) Building Flink Walkthrough Table Java 0.1 COMPILATION ERROR
[
https://issues.apache.org/jira/browse/FLINK-18341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164552#comment-17164552
]
Seth Wiesman commented on FLINK-18341:
--
Fixed in master: d88f9b1f9577291b32b472387f6981059522d9e2
release-1.11: 2777ecd6b7adc67cb0f01523a2f55688aaaf21d5
> Building Flink Walkthrough Table Java 0.1 COMPILATION ERROR
> ---
>
> Key: FLINK-18341
> URL: https://issues.apache.org/jira/browse/FLINK-18341
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API, Tests
>Affects Versions: 1.12.0, 1.11.1
>Reporter: Piotr Nowojski
>Assignee: Seth Wiesman
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.11.2
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3652=logs=08866332-78f7-59e4-4f7e-49a56faa3179=931b3127-d6ee-5f94-e204-48d51cd1c334
> {noformat}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-22294375765/flink-walkthrough-table-java/src/main/java/org/apache/flink/walkthrough/SpendReport.java:[23,46]
> cannot access org.apache.flink.table.api.bridge.java.BatchTableEnvironment
> bad class file:
> /home/vsts/work/1/.m2/repository/org/apache/flink/flink-table-api-java-bridge_2.11/1.12-SNAPSHOT/flink-table-api-java-bridge_2.11-1.12-SNAPSHOT.jar(org/apache/flink/table/api/bridge/java/BatchTableEnvironment.class)
> class file has wrong version 55.0, should be 52.0
> Please remove or make sure it appears in the correct subdirectory of the
> classpath.
> (...)
> [FAIL] 'Walkthrough Table Java nightly end-to-end test' failed after 0
> minutes and 4 seconds! Test exited with exit code 1
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] sjwiesman closed pull request #12970: [FLINK-18341][walkthroughs] Drop remaining table walkthrough archetypes
sjwiesman closed pull request #12970: URL: https://github.com/apache/flink/pull/12970 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #12980: [FLINK-18646] Verify memory manager empty in a separate thread with l…
tillrohrmann commented on a change in pull request #12980:
URL: https://github.com/apache/flink/pull/12980#discussion_r460156112
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlot.java
##
@@ -84,15 +85,22 @@
/** The closing future is completed when the slot is freed and closed.
*/
private final CompletableFuture closingFuture;
+ /**
+* {@link ExecutorService} for background actions, e.g. verify all
managed memory released.
+*/
+ private final ExecutorService asyncExecutor;
Review comment:
```suggestion
private final Executor asyncExecutor;
```
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java
##
@@ -37,9 +37,10 @@
class UnsafeMemoryBudget {
// max. number of sleeps during try-reserving with exponentially
// increasing delay before throwing OutOfMemoryError:
- // 1, 2, 4, 8, 16, 32, 64, 128, 256, 512 (total 1023 ms ~ 1 s)
+ // 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, ... (total (2 x 1024) - 1 ms
~ 2 s)
// which means that MemoryReservationException will be thrown after 1 s
of trying
Review comment:
```suggestion
// which means that MemoryReservationException will be thrown after 2 s
of trying
```
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlot.java
##
@@ -84,15 +85,22 @@
/** The closing future is completed when the slot is freed and closed.
*/
private final CompletableFuture closingFuture;
+ /**
+* {@link ExecutorService} for background actions, e.g. verify all
managed memory released.
Review comment:
```suggestion
* {@link Executor} for background actions, e.g. verify all managed
memory released.
```
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTableImpl.java
##
@@ -134,6 +143,12 @@ public TaskSlotTableImpl(
slotActions = null;
state = State.CREATED;
closingFuture = new CompletableFuture<>();
+
+ asyncExecutor = new ThreadPoolExecutor(
+ 0,
+ numberSlots,
+ 60L, TimeUnit.SECONDS,
+ new SynchronousQueue<>());
Review comment:
```suggestion
new SynchronousQueue<>(),
new ExecutorThreadFactory("task-slot-memory-verifier"));
```
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlot.java
##
@@ -295,22 +303,25 @@ public String toString() {
// and set the slot state to releasing so that
it gets eventually freed
tasks.values().forEach(task ->
task.failExternally(cause));
}
+
final CompletableFuture cleanupFuture =
FutureUtils
.waitForAll(tasks.values().stream().map(TaskSlotPayload::getTerminationFuture).collect(Collectors.toList()))
- .thenRun(() -> {
- verifyMemoryFreed();
- this.memoryManager.shutdown();
- });
-
+ .thenRun(memoryManager::shutdown);
+ verifyAllManagedMemoryIsReleasedAfter(cleanupFuture);
Review comment:
Maybe add a test which ensures that we can call
`MemoryManager.verifyEmpty()` after it has been shut down. That way we
explicitly state that this an explicit contract we want to guard.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTableImpl.java
##
@@ -134,6 +143,12 @@ public TaskSlotTableImpl(
slotActions = null;
state = State.CREATED;
closingFuture = new CompletableFuture<>();
+
+ asyncExecutor = new ThreadPoolExecutor(
Review comment:
What spoke against using the `ioExecutor` in `TaskManagerSharedServices`
when creating the `TaskSlotTableImpl`?
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlot.java
##
@@ -295,22 +303,25 @@ public String toString() {
// and set the slot state to releasing so that
it gets eventually freed
tasks.values().forEach(task ->
task.failExternally(cause));
}
+
final CompletableFuture cleanupFuture =
FutureUtils
.waitForAll(tasks.values().stream().map(TaskSlotPayload::getTerminationFuture).collect(Collectors.toList()))
- .thenRun(() -> {
-
[GitHub] [flink] flinkbot edited a comment on pull request #12986: [FLINK-18706] fix sync-savepoint exactly-once semantic
flinkbot edited a comment on pull request #12986: URL: https://github.com/apache/flink/pull/12986#issuecomment-663517112 ## CI report: * 9834eb95c35dd119d42d147597979acc02c3839a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4882) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18655) Set failOnUnableToExtractRepoInfo to false for git-commit-id-plugin in module flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-18655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hequn Cheng updated FLINK-18655:
Fix Version/s: 1.11.2
1.12.0
> Set failOnUnableToExtractRepoInfo to false for git-commit-id-plugin in module
> flink-runtime
> ---
>
> Key: FLINK-18655
> URL: https://issues.apache.org/jira/browse/FLINK-18655
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.11.1
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2
>
>
> Exception may be thrown when building source distribution without the .git
> folder(for the flink-runtime module):
> {code:java}
> [ERROR] Failed to execute goal
> pl.project13.maven:git-commit-id-plugin:4.0.0:revision (get-the-git-infos) on
> project flink-runtime_2.11: Could not complete Mojo execution... Error: Could
> not get HEAD Ref, are you sure you have set the dotGitDirectory property of
> this plugin to a valid path? -> [Help 1]
> {code}
>
> To solve the problem, we need to add the
> {{false}}
> configuration in addition to
> {{false}} in the pom of
> flink-runtime.
>
> The reason is the plugin:git-commit-id-plugin would search up all the maven
> parent project hierarchy until a .git directory is found. For example, if we
> config dotGitDirectory as /a/b/c/.git and if /a/b/c/.git is invalid, the
> plugin would search /a/b/.git and /a/.git. However, once the plugin found a
> /a/.git folder, it would fail on extracting repo info which leads to the
> failure above. The search logic of the plugin can be found
> [here|https://github.com/git-commit-id/git-commit-id-maven-plugin/blob/v4.0.0/maven/src/main/java/pl/project13/maven/git/GitDirLocator.java#L74].
> You can reproduce the exception with:
> - download the 1.11.0 source distribution.
> - put a .git folder under the path (or parent path) of
> ${project.basedir}/../.git. For example, my dotGitDirectory is:
> `/Users/hequn.chq/Downloads/flink-1.11.0/flink-runtime/../.git` and there is
> a .git folder under `/Users/hequn.chq/.git`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18655) Set failOnUnableToExtractRepoInfo to false for git-commit-id-plugin in module flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-18655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hequn Cheng updated FLINK-18655:
Affects Version/s: (was: 1.11.0)
1.12.0
1.11.1
> Set failOnUnableToExtractRepoInfo to false for git-commit-id-plugin in module
> flink-runtime
> ---
>
> Key: FLINK-18655
> URL: https://issues.apache.org/jira/browse/FLINK-18655
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.12.0, 1.11.1
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
> Labels: pull-request-available
>
> Exception may be thrown when building source distribution without the .git
> folder(for the flink-runtime module):
> {code:java}
> [ERROR] Failed to execute goal
> pl.project13.maven:git-commit-id-plugin:4.0.0:revision (get-the-git-infos) on
> project flink-runtime_2.11: Could not complete Mojo execution... Error: Could
> not get HEAD Ref, are you sure you have set the dotGitDirectory property of
> this plugin to a valid path? -> [Help 1]
> {code}
>
> To solve the problem, we need to add the
> {{false}}
> configuration in addition to
> {{false}} in the pom of
> flink-runtime.
>
> The reason is the plugin:git-commit-id-plugin would search up all the maven
> parent project hierarchy until a .git directory is found. For example, if we
> config dotGitDirectory as /a/b/c/.git and if /a/b/c/.git is invalid, the
> plugin would search /a/b/.git and /a/.git. However, once the plugin found a
> /a/.git folder, it would fail on extracting repo info which leads to the
> failure above. The search logic of the plugin can be found
> [here|https://github.com/git-commit-id/git-commit-id-maven-plugin/blob/v4.0.0/maven/src/main/java/pl/project13/maven/git/GitDirLocator.java#L74].
> You can reproduce the exception with:
> - download the 1.11.0 source distribution.
> - put a .git folder under the path (or parent path) of
> ${project.basedir}/../.git. For example, my dotGitDirectory is:
> `/Users/hequn.chq/Downloads/flink-1.11.0/flink-runtime/../.git` and there is
> a .git folder under `/Users/hequn.chq/.git`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-18655) Set failOnUnableToExtractRepoInfo to false for git-commit-id-plugin in module flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-18655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hequn Cheng closed FLINK-18655.
---
Resolution: Fixed
> Set failOnUnableToExtractRepoInfo to false for git-commit-id-plugin in module
> flink-runtime
> ---
>
> Key: FLINK-18655
> URL: https://issues.apache.org/jira/browse/FLINK-18655
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.11.1
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2
>
>
> Exception may be thrown when building source distribution without the .git
> folder(for the flink-runtime module):
> {code:java}
> [ERROR] Failed to execute goal
> pl.project13.maven:git-commit-id-plugin:4.0.0:revision (get-the-git-infos) on
> project flink-runtime_2.11: Could not complete Mojo execution... Error: Could
> not get HEAD Ref, are you sure you have set the dotGitDirectory property of
> this plugin to a valid path? -> [Help 1]
> {code}
>
> To solve the problem, we need to add the
> {{false}}
> configuration in addition to
> {{false}} in the pom of
> flink-runtime.
>
> The reason is the plugin:git-commit-id-plugin would search up all the maven
> parent project hierarchy until a .git directory is found. For example, if we
> config dotGitDirectory as /a/b/c/.git and if /a/b/c/.git is invalid, the
> plugin would search /a/b/.git and /a/.git. However, once the plugin found a
> /a/.git folder, it would fail on extracting repo info which leads to the
> failure above. The search logic of the plugin can be found
> [here|https://github.com/git-commit-id/git-commit-id-maven-plugin/blob/v4.0.0/maven/src/main/java/pl/project13/maven/git/GitDirLocator.java#L74].
> You can reproduce the exception with:
> - download the 1.11.0 source distribution.
> - put a .git folder under the path (or parent path) of
> ${project.basedir}/../.git. For example, my dotGitDirectory is:
> `/Users/hequn.chq/Downloads/flink-1.11.0/flink-runtime/../.git` and there is
> a .git folder under `/Users/hequn.chq/.git`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18655) Set failOnUnableToExtractRepoInfo to false for git-commit-id-plugin in module flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-18655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hequn Cheng updated FLINK-18655:
Affects Version/s: (was: 1.12.0)
> Set failOnUnableToExtractRepoInfo to false for git-commit-id-plugin in module
> flink-runtime
> ---
>
> Key: FLINK-18655
> URL: https://issues.apache.org/jira/browse/FLINK-18655
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.11.1
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
> Labels: pull-request-available
>
> Exception may be thrown when building source distribution without the .git
> folder(for the flink-runtime module):
> {code:java}
> [ERROR] Failed to execute goal
> pl.project13.maven:git-commit-id-plugin:4.0.0:revision (get-the-git-infos) on
> project flink-runtime_2.11: Could not complete Mojo execution... Error: Could
> not get HEAD Ref, are you sure you have set the dotGitDirectory property of
> this plugin to a valid path? -> [Help 1]
> {code}
>
> To solve the problem, we need to add the
> {{false}}
> configuration in addition to
> {{false}} in the pom of
> flink-runtime.
>
> The reason is the plugin:git-commit-id-plugin would search up all the maven
> parent project hierarchy until a .git directory is found. For example, if we
> config dotGitDirectory as /a/b/c/.git and if /a/b/c/.git is invalid, the
> plugin would search /a/b/.git and /a/.git. However, once the plugin found a
> /a/.git folder, it would fail on extracting repo info which leads to the
> failure above. The search logic of the plugin can be found
> [here|https://github.com/git-commit-id/git-commit-id-maven-plugin/blob/v4.0.0/maven/src/main/java/pl/project13/maven/git/GitDirLocator.java#L74].
> You can reproduce the exception with:
> - download the 1.11.0 source distribution.
> - put a .git folder under the path (or parent path) of
> ${project.basedir}/../.git. For example, my dotGitDirectory is:
> `/Users/hequn.chq/Downloads/flink-1.11.0/flink-runtime/../.git` and there is
> a .git folder under `/Users/hequn.chq/.git`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-11547) JsonMappingException in DynamoDBStreamsSchema
[
https://issues.apache.org/jira/browse/FLINK-11547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise resolved FLINK-11547.
--
Fix Version/s: 1.12.0
Resolution: Fixed
> JsonMappingException in DynamoDBStreamsSchema
> -
>
> Key: FLINK-11547
> URL: https://issues.apache.org/jira/browse/FLINK-11547
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.8.0
>Reporter: Devin Thomson
>Assignee: Devin Thomson
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
> Attachments: full_flink_trace.txt
>
>
> If DynamoDBStreamsSchema is used as the deserializer for
> FlinkDynamoDBStreamsConsumer, an exception occurs during deserialization of a
> record. The stack trace is attached.
>
> This is a blocker for using DynamoDBStreamsSchema, but can be worked around
> by implementing a custom deserializer. The issue appears to be due to the
> usage of vanilla ObjectMapper:
>
> {code:java}
> private static final ObjectMapper MAPPER = new ObjectMapper();
> {code}
> When it should be using the appropriate mix-ins offered by the dynamodb
> stream adapter library:
> {code:java}
> private static final ObjectMapper MAPPER = new RecordObjectMapper();
> {code}
> This appears to resolve the issue, I tested by using my own deserializer
> implementation.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tweise merged pull request #12916: [FLINK-11547][flink-connector-kinesis] Fix JsonMappingException in DynamoDBStreamsSchema
tweise merged pull request #12916: URL: https://github.com/apache/flink/pull/12916 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16867) Simplify default timeout configuration
[
https://issues.apache.org/jira/browse/FLINK-16867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164519#comment-17164519
]
Steven Zhen Wu commented on FLINK-16867:
I should clarify that web.timeout becomes irrelevant to us for job submission
via REST API. It seems that only client.timeout matters for that particular
code path of submitting job.
> Simplify default timeout configuration
> --
>
> Key: FLINK-16867
> URL: https://issues.apache.org/jira/browse/FLINK-16867
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration, Runtime / Coordination
>Affects Versions: 1.9.2, 1.10.0, 1.11.0
>Reporter: Till Rohrmann
>Priority: Minor
> Fix For: 1.12.0
>
>
> At the moment, Flink has several timeout options:
> * {{akka.ask.timeout}}: Timeout for intra cluster RPCs (JM <\-> RM <\-> TE)
> * {{web.timeout}}: Timeout for RPCs between REST handlers and RM, JM, TE
> At the moment, these values are separately configured. This requires the user
> to know about both configuration options and that Flink has multiple timeout
> values.
> In order to simplify setups I would suggest that {{web.timeout}} defaults to
> {{akka.ask.timeout}}, if {{web.timeout}} has not been explicitly configured.
> This has the benefits that the user only need to know about a single timeout
> value which is applied cluster wide.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18655) Set failOnUnableToExtractRepoInfo to false for git-commit-id-plugin in module flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-18655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164521#comment-17164521
]
Hequn Cheng commented on FLINK-18655:
-
Resolved
in 1.12.0 via 09f1674cb3e6cb536a67f060cb882aae7e7a63bb
in 1.11.2 via 0c8f8d7ca645863024474477dea5995e990fd37a
> Set failOnUnableToExtractRepoInfo to false for git-commit-id-plugin in module
> flink-runtime
> ---
>
> Key: FLINK-18655
> URL: https://issues.apache.org/jira/browse/FLINK-18655
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.11.0
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
> Labels: pull-request-available
>
> Exception may be thrown when building source distribution without the .git
> folder(for the flink-runtime module):
> {code:java}
> [ERROR] Failed to execute goal
> pl.project13.maven:git-commit-id-plugin:4.0.0:revision (get-the-git-infos) on
> project flink-runtime_2.11: Could not complete Mojo execution... Error: Could
> not get HEAD Ref, are you sure you have set the dotGitDirectory property of
> this plugin to a valid path? -> [Help 1]
> {code}
>
> To solve the problem, we need to add the
> {{false}}
> configuration in addition to
> {{false}} in the pom of
> flink-runtime.
>
> The reason is the plugin:git-commit-id-plugin would search up all the maven
> parent project hierarchy until a .git directory is found. For example, if we
> config dotGitDirectory as /a/b/c/.git and if /a/b/c/.git is invalid, the
> plugin would search /a/b/.git and /a/.git. However, once the plugin found a
> /a/.git folder, it would fail on extracting repo info which leads to the
> failure above. The search logic of the plugin can be found
> [here|https://github.com/git-commit-id/git-commit-id-maven-plugin/blob/v4.0.0/maven/src/main/java/pl/project13/maven/git/GitDirLocator.java#L74].
> You can reproduce the exception with:
> - download the 1.11.0 source distribution.
> - put a .git folder under the path (or parent path) of
> ${project.basedir}/../.git. For example, my dotGitDirectory is:
> `/Users/hequn.chq/Downloads/flink-1.11.0/flink-runtime/../.git` and there is
> a .git folder under `/Users/hequn.chq/.git`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16510) Task manager safeguard shutdown may not be reliable
[
https://issues.apache.org/jira/browse/FLINK-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164511#comment-17164511
]
Thomas Weise commented on FLINK-16510:
--
I think there are 2 interests involved here:
1. How to ensure that the task manager JVM gets safely terminated on a fatal
error, regardless of the behavior of user code and other dependencies
2. Why execution of shutdown hooks can cause the JVM to get stuck
We at Lyft need the option to safely terminate the VMs so that pods are
replaced immediately. We are not interested in executing of shutdown hooks when
the JVM is already in a problematic state.
> Task manager safeguard shutdown may not be reliable
> ---
>
> Key: FLINK-16510
> URL: https://issues.apache.org/jira/browse/FLINK-16510
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Attachments: stack2-1.txt
>
>
> The {{JvmShutdownSafeguard}} does not always succeed but can hang when
> multiple threads attempt to shutdown the JVM. Apparently mixing
> {{System.exit()}} with ShutdownHooks and forcefully terminating the JVM via
> {{Runtime.halt()}} does not play together well:
> {noformat}
> "Jvm Terminator" #22 daemon prio=5 os_prio=0 tid=0x7fb8e82f2800
> nid=0x5a96 runnable [0x7fb35cffb000]
>java.lang.Thread.State: RUNNABLE
> at java.lang.Shutdown.$$YJP$$halt0(Native Method)
> at java.lang.Shutdown.halt0(Shutdown.java)
> at java.lang.Shutdown.halt(Shutdown.java:139)
> - locked <0x00047ed67638> (a java.lang.Shutdown$Lock)
> at java.lang.Runtime.halt(Runtime.java:276)
> at
> org.apache.flink.runtime.util.JvmShutdownSafeguard$DelayedTerminator.run(JvmShutdownSafeguard.java:86)
> at java.lang.Thread.run(Thread.java:748)
>Locked ownable synchronizers:
> - None
> "FlinkCompletableFutureDelayScheduler-thread-1" #18154 daemon prio=5
> os_prio=0 tid=0x7fb708a7d000 nid=0x5a8a waiting for monitor entry
> [0x7fb289d49000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Shutdown.halt(Shutdown.java:139)
> - waiting to lock <0x00047ed67638> (a java.lang.Shutdown$Lock)
> at java.lang.Shutdown.exit(Shutdown.java:213)
> - locked <0x00047edb7348> (a java.lang.Class for java.lang.Shutdown)
> at java.lang.Runtime.exit(Runtime.java:110)
> at java.lang.System.exit(System.java:973)
> at
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner.terminateJVM(TaskManagerRunner.java:266)
> at
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$onFatalError$1(TaskManagerRunner.java:260)
> at
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner$$Lambda$27464/1464672548.accept(Unknown
> Source)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:943)
> at
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$orTimeout$11(FutureUtils.java:361)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$$Lambda$27435/159015392.run(Unknown
> Source)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>Locked ownable synchronizers:
> - <0x0006d5e56bd0> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
> {noformat}
> Note that under this condition the JVM should terminate but it still hangs.
> Sometimes it quits after several minutes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] hequn8128 merged pull request #12941: [FLINK-18655][flink-runtime] Set failOnUnableToExtractRepoInfo to false for git-commit-id-plugin
hequn8128 merged pull request #12941: URL: https://github.com/apache/flink/pull/12941 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] hequn8128 commented on pull request #12941: [FLINK-18655][flink-runtime] Set failOnUnableToExtractRepoInfo to false for git-commit-id-plugin
hequn8128 commented on pull request #12941: URL: https://github.com/apache/flink/pull/12941#issuecomment-663617091 @tillrohrmann @zentol Thanks a lot for the review. Merging this, and will address the checking problem during the release in another jira. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16510) Task manager safeguard shutdown may not be reliable
[
https://issues.apache.org/jira/browse/FLINK-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164508#comment-17164508
]
Till Rohrmann commented on FLINK-16510:
---
Jup, if it is a GC bug, then this is a JVM problem. Moreover, this problem
should then also occur when only using {{Runtime.halt()}}. But maybe it is more
likely to be triggered when running additional shut down hooks which create
some lambdas. But let's first try to figure out whether it is really this
problem.
> Task manager safeguard shutdown may not be reliable
> ---
>
> Key: FLINK-16510
> URL: https://issues.apache.org/jira/browse/FLINK-16510
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Attachments: stack2-1.txt
>
>
> The {{JvmShutdownSafeguard}} does not always succeed but can hang when
> multiple threads attempt to shutdown the JVM. Apparently mixing
> {{System.exit()}} with ShutdownHooks and forcefully terminating the JVM via
> {{Runtime.halt()}} does not play together well:
> {noformat}
> "Jvm Terminator" #22 daemon prio=5 os_prio=0 tid=0x7fb8e82f2800
> nid=0x5a96 runnable [0x7fb35cffb000]
>java.lang.Thread.State: RUNNABLE
> at java.lang.Shutdown.$$YJP$$halt0(Native Method)
> at java.lang.Shutdown.halt0(Shutdown.java)
> at java.lang.Shutdown.halt(Shutdown.java:139)
> - locked <0x00047ed67638> (a java.lang.Shutdown$Lock)
> at java.lang.Runtime.halt(Runtime.java:276)
> at
> org.apache.flink.runtime.util.JvmShutdownSafeguard$DelayedTerminator.run(JvmShutdownSafeguard.java:86)
> at java.lang.Thread.run(Thread.java:748)
>Locked ownable synchronizers:
> - None
> "FlinkCompletableFutureDelayScheduler-thread-1" #18154 daemon prio=5
> os_prio=0 tid=0x7fb708a7d000 nid=0x5a8a waiting for monitor entry
> [0x7fb289d49000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Shutdown.halt(Shutdown.java:139)
> - waiting to lock <0x00047ed67638> (a java.lang.Shutdown$Lock)
> at java.lang.Shutdown.exit(Shutdown.java:213)
> - locked <0x00047edb7348> (a java.lang.Class for java.lang.Shutdown)
> at java.lang.Runtime.exit(Runtime.java:110)
> at java.lang.System.exit(System.java:973)
> at
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner.terminateJVM(TaskManagerRunner.java:266)
> at
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$onFatalError$1(TaskManagerRunner.java:260)
> at
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner$$Lambda$27464/1464672548.accept(Unknown
> Source)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:943)
> at
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$orTimeout$11(FutureUtils.java:361)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$$Lambda$27435/159015392.run(Unknown
> Source)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>Locked ownable synchronizers:
> - <0x0006d5e56bd0> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
> {noformat}
> Note that under this condition the JVM should terminate but it still hangs.
> Sometimes it quits after several minutes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tillrohrmann commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
tillrohrmann commented on a change in pull request #12981: URL: https://github.com/apache/flink/pull/12981#discussion_r460148559 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTableImpl.java ## @@ -134,6 +143,12 @@ public TaskSlotTableImpl( slotActions = null; state = State.CREATED; closingFuture = new CompletableFuture<>(); + + asyncExecutor = new ThreadPoolExecutor( Review comment: No. Mainly for having an uncaught exception handler configured. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
tillrohrmann commented on a change in pull request #12981:
URL: https://github.com/apache/flink/pull/12981#discussion_r460147299
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java
##
@@ -147,8 +147,10 @@ void reserveMemory(long size, int maxSleeps) throws
MemoryReservationException {
}
// no luck
- throw new MemoryReservationException(
- String.format("Could not allocate %d bytes,
only %d bytes are remaining", size, availableOrReserved));
+ throw new MemoryReservationException(String.format(
+ "Could not allocate %d bytes, only %d bytes are
remaining, try to upgrade to Java 8u72 or higher",
Review comment:
Yes adding a bit more context for why to upgrade would be helpful. E.g.
"This usually indicates that you are requesting more memory than you have
reserved. However, when running an old JVM version it can also be caused by
slow garbage collection. Try to upgrade to Java 8u72 or higher if running on an
old Java version.".
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12985: [FLINK-18682][orc][hive] Vector orc reader cannot read Hive 2.0.0 table
flinkbot edited a comment on pull request #12985: URL: https://github.com/apache/flink/pull/12985#issuecomment-663499515 ## CI report: * d5db2be542e04d592c05711ebd4a8b4c971ceed5 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4880) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18708) The links of the connector sql jar of Kafka 0.10 and 0.11 are extinct
[ https://issues.apache.org/jira/browse/FLINK-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-18708. --- Fix Version/s: 1.11.2 1.12.0 Resolution: Fixed master: ce24bf79a4e67133edc350d35015f1f1fb79963c release-1.11: 699ffb4bcf604354d2c8b7429099bf759ae95053 > The links of the connector sql jar of Kafka 0.10 and 0.11 are extinct > - > > Key: FLINK-18708 > URL: https://issues.apache.org/jira/browse/FLINK-18708 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka, Documentation >Affects Versions: 1.12.0, 1.11.1 >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.11.2 > > > The links of the connector sql jar of Kafka 0.10 and 0.11 are extinct. I will > fix it as soon as possible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dianfu closed pull request #12990: [FLINK-18708][docs] Update the sql client jar url of kafka 0.10 and 0.11
dianfu closed pull request #12990: URL: https://github.com/apache/flink/pull/12990 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12990: [FLINK-18708][docs] Update the sql client jar url of kafka 0.10 and 0.11
flinkbot edited a comment on pull request #12990: URL: https://github.com/apache/flink/pull/12990#issuecomment-663591721 ## CI report: * d446315c9a0e7870b387f8db118d9896aeb3bcdf Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4890) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18709) Implement PhysicalSlotProvider
Andrey Zagrebin created FLINK-18709: --- Summary: Implement PhysicalSlotProvider Key: FLINK-18709 URL: https://issues.apache.org/jira/browse/FLINK-18709 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Reporter: Andrey Zagrebin Assignee: Andrey Zagrebin PhysicalSlotProviderImpl tries to allocate a physical slot from the available idle cached slots in SlotPool. If it is not possible, it requests a new slot from the SlotPool. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-18662) Provide more detailed metrics why unaligned checkpoint is taking long time
[
https://issues.apache.org/jira/browse/FLINK-18662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164495#comment-17164495
]
Piotr Nowojski edited comment on FLINK-18662 at 7/24/20, 3:34 PM:
--
I'm not sure which option is better:
# Add new metric {{checkpointUnalignmentTime}}, which would be mutually
exclusive with {{checkpointAlignmentTime}} - if one is non zero, the other must
be zero and vice versa. This might be a bit annoying to for someone setting up
some metric reporting system - he would have to setup and monitor both things.
# Overload the existing {{checkpointAlignmentTime}} metric, and adapt it to
repot unalignment time for unaligned checkpoints. Note that in the webUI user
would need to understand with what type of checkpoint he is dealing with and
what's the difference. Also it might be a bit confusing if web UI presents an
aligned savepoint and unaligned checkpoint at the same time. On the other hand
the delay between the first and last checkpoint barrier for unaligned
checkpoints could arguably be called "alignment time".
Regarding the size of unaligned data, we need to add a new metric, as
{{lastCheckpointUnalignmentSize}}/"Alignment Buffered" was removed in
FLINK-16404.
[~zjwang], [~AHeise], [~roman_khachatryan] what do you think?
was (Author: pnowojski):
I'm not sure which option is better:
# Add new metric {{checkpointUnalignmentTime}}, which would be mutually
exclusive with {{checkpointAlignmentTime}} - if one is non zero, the other must
be zero and vice versa. This might be a bit annoying to for someone setting up
some metric reporting system - he would have to setup both things.
# Overload the existing {{checkpointAlignmentTime}} metric, and adapt it to
repot unalignment time for unaligned checkpoints. Note that in the webUI user
would need to understand with what type of checkpoint he is dealing with and
what's the difference. Also it might be a bit confusing if web UI presents an
aligned savepoint and unaligned checkpoint at the same time. On the other hand
the delay between the first and last checkpoint barrier for unaligned
checkpoints could arguably be called "alignment time".
Regarding the size of unaligned data, we need to add a new metric, as
{{lastCheckpointUnalignmentSize}}/"Alignment Buffered" was removed in
FLINK-16404.
[~zjwang], [~AHeise], [~roman_khachatryan] what do you think?
> Provide more detailed metrics why unaligned checkpoint is taking long time
> --
>
> Key: FLINK-18662
> URL: https://issues.apache.org/jira/browse/FLINK-18662
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics, Runtime / Network
>Affects Versions: 1.11.1
>Reporter: Piotr Nowojski
>Priority: Critical
> Fix For: 1.12.0
>
> Attachments: Screenshot 2020-07-21 at 11.50.02.png
>
>
> With unaligned checkpoint there can happen situation as in the attached
> screenshot.
> Task reports long end to end checkpoint time (~2h50min), ~0s sync time,
> ~2h50min async time, ~0s start delay. It means that task received first
> checkpoint barrier from one of the channels very quickly (~0s), sync part was
> quick, but we do not know why async part was taking so long. It could be
> because of three things:
> # long operator state IO writes
> # long spilling of in-flight data
> # long time to receive the final checkpoint barrier from the last lagging
> channel
> First and second are probably indistinguishable and the difference between
> them doesn't matter much for analyzing. However the last one is quite
> different. It might be independent of the IO, and we are missing this
> information.
> Maybe we could report it as "alignment duration" and while we are at it, we
> could also report amount of spilled in-flight data for unaligned checkpoints
> as "alignment buffered"?
> Ideally we should report it as new metrics, but that leaves a question how to
> display it in the UI, with limited space available. Maybe it could be
> reported as:
> ||Alignment Buffered||Alignment Duration||
> |0 B (632 MB)|0ms (2h 49m 32s)|
> Where the values in the parenthesis would come from unaligned checkpoints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18662) Provide more detailed metrics why unaligned checkpoint is taking long time
[
https://issues.apache.org/jira/browse/FLINK-18662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164495#comment-17164495
]
Piotr Nowojski commented on FLINK-18662:
I'm not sure which option is better:
# Add new metric {{checkpointUnalignmentTime}}, which would be mutually
exclusive with {{checkpointAlignmentTime}} - if one is non zero, the other must
be zero and vice versa. This might be a bit annoying to for someone setting up
some metric reporting system - he would have to setup both things.
# Overload the existing {{checkpointAlignmentTime}} metric, and adapt it to
repot unalignment time for unaligned checkpoints. Note that in the webUI user
would need to understand with what type of checkpoint he is dealing with and
what's the difference. Also it might be a bit confusing if web UI presents an
aligned savepoint and unaligned checkpoint at the same time. On the other hand
the delay between the first and last checkpoint barrier for unaligned
checkpoints could arguably be called "alignment time".
Regarding the size of unaligned data, we need to add a new metric, as
{{lastCheckpointUnalignmentSize}}/"Alignment Buffered" was removed in
FLINK-16404.
[~zjwang], [~AHeise], [~roman_khachatryan] what do you think?
> Provide more detailed metrics why unaligned checkpoint is taking long time
> --
>
> Key: FLINK-18662
> URL: https://issues.apache.org/jira/browse/FLINK-18662
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics, Runtime / Network
>Affects Versions: 1.11.1
>Reporter: Piotr Nowojski
>Priority: Critical
> Fix For: 1.12.0
>
> Attachments: Screenshot 2020-07-21 at 11.50.02.png
>
>
> With unaligned checkpoint there can happen situation as in the attached
> screenshot.
> Task reports long end to end checkpoint time (~2h50min), ~0s sync time,
> ~2h50min async time, ~0s start delay. It means that task received first
> checkpoint barrier from one of the channels very quickly (~0s), sync part was
> quick, but we do not know why async part was taking so long. It could be
> because of three things:
> # long operator state IO writes
> # long spilling of in-flight data
> # long time to receive the final checkpoint barrier from the last lagging
> channel
> First and second are probably indistinguishable and the difference between
> them doesn't matter much for analyzing. However the last one is quite
> different. It might be independent of the IO, and we are missing this
> information.
> Maybe we could report it as "alignment duration" and while we are at it, we
> could also report amount of spilled in-flight data for unaligned checkpoints
> as "alignment buffered"?
> Ideally we should report it as new metrics, but that leaves a question how to
> display it in the UI, with limited space available. Maybe it could be
> reported as:
> ||Alignment Buffered||Alignment Duration||
> |0 B (632 MB)|0ms (2h 49m 32s)|
> Where the values in the parenthesis would come from unaligned checkpoints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16404) Avoid caching buffers for blocked input channels before barrier alignment
[
https://issues.apache.org/jira/browse/FLINK-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164492#comment-17164492
]
Piotr Nowojski commented on FLINK-16404:
Hey [~zjwang] [~kevin.cyj], I think you have forgotten to remove this metric in
at least one remaining place [in the web
UI|https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/subtask/job-checkpoints-subtask.component.html#L105].
Potentially we should also drop it from
{{TaskCheckpointStatisticsWithSubtaskDetails.CheckpointAlignment#CheckpointAlignment}}
- but this might affect our REST API and I'm not sure if the API is
{{@Public}} and stable?
> Avoid caching buffers for blocked input channels before barrier alignment
> -
>
> Key: FLINK-16404
> URL: https://issues.apache.org/jira/browse/FLINK-16404
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
>Reporter: Zhijiang
>Assignee: Yingjie Cao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> One motivation of this issue is for reducing the in-flight data in the case
> of back pressure to speed up checkpoint. The current default exclusive
> buffers per channel is 2. If we reduce it to 0 and increase somewhat floating
> buffers for compensation, it might cause deadlock problem because all the
> floating buffers might be requested away by some blocked input channels and
> never recycled until barrier alignment.
> In order to solve above deadlock concern, we can make some logic changes on
> both sender and receiver sides.
> * Sender side: It should revoke previous received credit after sending
> checkpoint barrier, that means it would not send any following buffers until
> receiving new credits.
> * Receiver side: The respective channel releases the requested floating
> buffers if barrier is received from the network. After barrier alignment, it
> would request floating buffers for the channels with positive backlog, and
> notify the sender side of available credits. Then the sender can continue
> transporting the buffers.
> Based on above changes, we can also remove the `BufferStorage` component
> completely, because the receiver would never reading buffers for blocked
> channels. Another possible benefit is that the floating buffers might be more
> properly made use of before barrier alignment.
> The only side effect would bring somehow cold setup after barrier alignment.
> That means the sender side has to wait for credit feedback to transport data
> just after alignment, which would impact on delay and network throughput. But
> considering the checkpoint interval not too short in general, so the above
> side effect can be ignored in practice. We can further verify it via existing
> micro-benchmark.
> After this ticket done, we still can not set exclusive buffers to zero ATM,
> there exists another deadlock issue which would be solved separately in
> another ticket.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-16404) Avoid caching buffers for blocked input channels before barrier alignment
[
https://issues.apache.org/jira/browse/FLINK-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164492#comment-17164492
]
Piotr Nowojski edited comment on FLINK-16404 at 7/24/20, 3:25 PM:
--
Hey [~zjwang] [~kevin.cyj], I think you have forgotten to remove this metric in
at least one remaining place [in the web
UI|https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/subtask/job-checkpoints-subtask.component.html#L105].
As it is now, this value is always zero.
Potentially we should also drop it from
{{TaskCheckpointStatisticsWithSubtaskDetails.CheckpointAlignment#CheckpointAlignment}}
- but this might affect our REST API and I'm not sure if the API is
{{@Public}} and stable?
was (Author: pnowojski):
Hey [~zjwang] [~kevin.cyj], I think you have forgotten to remove this metric in
at least one remaining place [in the web
UI|https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/subtask/job-checkpoints-subtask.component.html#L105].
Potentially we should also drop it from
{{TaskCheckpointStatisticsWithSubtaskDetails.CheckpointAlignment#CheckpointAlignment}}
- but this might affect our REST API and I'm not sure if the API is
{{@Public}} and stable?
> Avoid caching buffers for blocked input channels before barrier alignment
> -
>
> Key: FLINK-16404
> URL: https://issues.apache.org/jira/browse/FLINK-16404
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
>Reporter: Zhijiang
>Assignee: Yingjie Cao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> One motivation of this issue is for reducing the in-flight data in the case
> of back pressure to speed up checkpoint. The current default exclusive
> buffers per channel is 2. If we reduce it to 0 and increase somewhat floating
> buffers for compensation, it might cause deadlock problem because all the
> floating buffers might be requested away by some blocked input channels and
> never recycled until barrier alignment.
> In order to solve above deadlock concern, we can make some logic changes on
> both sender and receiver sides.
> * Sender side: It should revoke previous received credit after sending
> checkpoint barrier, that means it would not send any following buffers until
> receiving new credits.
> * Receiver side: The respective channel releases the requested floating
> buffers if barrier is received from the network. After barrier alignment, it
> would request floating buffers for the channels with positive backlog, and
> notify the sender side of available credits. Then the sender can continue
> transporting the buffers.
> Based on above changes, we can also remove the `BufferStorage` component
> completely, because the receiver would never reading buffers for blocked
> channels. Another possible benefit is that the floating buffers might be more
> properly made use of before barrier alignment.
> The only side effect would bring somehow cold setup after barrier alignment.
> That means the sender side has to wait for credit feedback to transport data
> just after alignment, which would impact on delay and network throughput. But
> considering the checkpoint interval not too short in general, so the above
> side effect can be ignored in practice. We can further verify it via existing
> micro-benchmark.
> After this ticket done, we still can not set exclusive buffers to zero ATM,
> there exists another deadlock issue which would be solved separately in
> another ticket.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12990: [FLINK-18708][docs] Update the sql client jar url of kafka 0.10 and 0.11
flinkbot commented on pull request #12990: URL: https://github.com/apache/flink/pull/12990#issuecomment-663591721 ## CI report: * d446315c9a0e7870b387f8db118d9896aeb3bcdf UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12989: [FLINK-18656][network,metrics] Fix startDelay metric for unaligned checkpoints
flinkbot edited a comment on pull request #12989: URL: https://github.com/apache/flink/pull/12989#issuecomment-663583428 ## CI report: * 0841311adec0be29af94dad85f0e1615bfa42045 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4886) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #12561: [hotfix][docs]Fix typos in types_serialization.md and types_serializa…
rmetzger commented on pull request #12561: URL: https://github.com/apache/flink/pull/12561#issuecomment-663591009 Nice catch! Thanks for the fix! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger merged pull request #12561: [hotfix][docs]Fix typos in types_serialization.md and types_serializa…
rmetzger merged pull request #12561: URL: https://github.com/apache/flink/pull/12561 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #12532: [FLINK-18182] [flink-connector-kinesis] Upgrade to latest AWS SDK (to include af-south-1 region)
rmetzger commented on pull request #12532: URL: https://github.com/apache/flink/pull/12532#issuecomment-663590036 Most likely the version upgrade also implies an update to the NOTICE file for the license stuff. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18552) Update migration tests in master to cover migration from release-1.11
[
https://issues.apache.org/jira/browse/FLINK-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski closed FLINK-18552.
--
Fix Version/s: 1.11.2
Resolution: Fixed
Added missing tests for 1.11 to master branch as a49a350d85..740ffb5bfa and to
release-1.11 as e55516a040..7cfd72d489
> Update migration tests in master to cover migration from release-1.11
> -
>
> Key: FLINK-18552
> URL: https://issues.apache.org/jira/browse/FLINK-18552
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Reporter: Zhijiang
>Assignee: Yun Gao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2
>
>
> We should update the following tests to cover migration from release-1.11:
> * {{CEPMigrationTest}}
> * {{BucketingSinkMigrationTest}}
> * {{FlinkKafkaConsumerBaseMigrationTest}}
> * {{ContinuousFileProcessingMigrationTest}}
> * {{WindowOperatorMigrationTest}}
> * {{StatefulJobSavepointMigrationITCase.scala}}
> * {{StatefulJobWBroadcastStateMigrationITCase.scala}}
>
> Refers to https://issues.apache.org/jira/browse/FLINK-13613, there are more
> migration tests requires to update:
>
> * FlinkKafkaProducer011MigrationTest
> * FlinkKafkaProducerMigrationOperatorTest
> * FlinkKafkaProducerMigrationTest
> * StatefulJobSavepointMigrationITCase
> * StatefulJobWBroadcastStateMigrationITCase
> * TypeSerializerSnapshotMigrationITCase
> * AbstractKeyedOperatorRestoreTestBase
> * AbstractNonKeyedOperatorRestoreTestBase
> * FlinkKinesisConsumerMigrationTest
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] azagrebin commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
azagrebin commented on a change in pull request #12981: URL: https://github.com/apache/flink/pull/12981#discussion_r460101119 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTableImpl.java ## @@ -134,6 +143,12 @@ public TaskSlotTableImpl( slotActions = null; state = State.CREATED; closingFuture = new CompletableFuture<>(); + + asyncExecutor = new ThreadPoolExecutor( + 0, + numberSlots, + 60L, TimeUnit.SECONDS, + new SynchronousQueue<>()); Review comment: What kind of test do you mean? That the closing verification is off-loaded? I want the executor to allocate a new thread if there is no idle one for each slot closing but I do not want to keep idle threads too long. Is it not how the `ThreadPoolExecutor` is specified? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski merged pull request #12984: [FLINK-18552][tests] Update migration tests in release-1.11 to cover migration for 1.11
pnowojski merged pull request #12984: URL: https://github.com/apache/flink/pull/12984 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski merged pull request #12972: [FLINK-18552][tests] Update migration tests in master to cover migration for 1.11
pnowojski merged pull request #12972: URL: https://github.com/apache/flink/pull/12972 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
azagrebin commented on a change in pull request #12981: URL: https://github.com/apache/flink/pull/12981#discussion_r460101119 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTableImpl.java ## @@ -134,6 +143,12 @@ public TaskSlotTableImpl( slotActions = null; state = State.CREATED; closingFuture = new CompletableFuture<>(); + + asyncExecutor = new ThreadPoolExecutor( + 0, + numberSlots, + 60L, TimeUnit.SECONDS, + new SynchronousQueue<>()); Review comment: What kind of test do you mean? I just want to use it as a cached executor. I want it to allocate a new thread if there is no idle one for each slot closing but I do not want to keep idle threads too long. Is it not how the `ThreadPoolExecutor` is specified? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
azagrebin commented on a change in pull request #12981: URL: https://github.com/apache/flink/pull/12981#discussion_r460101119 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTableImpl.java ## @@ -134,6 +143,12 @@ public TaskSlotTableImpl( slotActions = null; state = State.CREATED; closingFuture = new CompletableFuture<>(); + + asyncExecutor = new ThreadPoolExecutor( + 0, + numberSlots, + 60L, TimeUnit.SECONDS, + new SynchronousQueue<>()); Review comment: What kind of test do you mean? I just want to use it as a cached executor. I want it to allocate a new thread if there is no idle one for each slot closing but I do not want to keep idle threads too long. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
azagrebin commented on a change in pull request #12981:
URL: https://github.com/apache/flink/pull/12981#discussion_r460112209
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTableImpl.java
##
@@ -134,6 +143,12 @@ public TaskSlotTableImpl(
slotActions = null;
state = State.CREATED;
closingFuture = new CompletableFuture<>();
+
+ asyncExecutor = new ThreadPoolExecutor(
Review comment:
I suppose mostly for naming purposes?
`Util.threadFactory("tm-slot-async-call", false)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12990: [FLINK-18708][docs] Update the sql client jar url of kafka 0.10 and 0.11
flinkbot commented on pull request #12990: URL: https://github.com/apache/flink/pull/12990#issuecomment-663583841 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit d446315c9a0e7870b387f8db118d9896aeb3bcdf (Fri Jul 24 15:04:05 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18708) The links of the connector sql jar of Kafka 0.10 and 0.11 are extinct
[ https://issues.apache.org/jira/browse/FLINK-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18708: --- Labels: pull-request-available (was: ) > The links of the connector sql jar of Kafka 0.10 and 0.11 are extinct > - > > Key: FLINK-18708 > URL: https://issues.apache.org/jira/browse/FLINK-18708 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka, Documentation >Affects Versions: 1.12.0, 1.11.1 >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > > The links of the connector sql jar of Kafka 0.10 and 0.11 are extinct. I will > fix it as soon as possible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12989: [FLINK-18656][network,metrics] Fix startDelay metric for unaligned checkpoints
flinkbot commented on pull request #12989: URL: https://github.com/apache/flink/pull/12989#issuecomment-663583428 ## CI report: * 0841311adec0be29af94dad85f0e1615bfa42045 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-11127) Make metrics query service establish connection to JobManager
[
https://issues.apache.org/jira/browse/FLINK-11127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164484#comment-17164484
]
Robert Metzger commented on FLINK-11127:
I don't know ECS well enough to really comment on this.
But since no other user has complained about the Flink on ECS, and you've also
moved away from it, I would consider this problem resolved.
> Make metrics query service establish connection to JobManager
> -
>
> Key: FLINK-11127
> URL: https://issues.apache.org/jira/browse/FLINK-11127
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes, Runtime / Coordination, Runtime
> / Metrics
>Affects Versions: 1.7.0, 1.9.2, 1.10.0
>Reporter: Ufuk Celebi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As part of FLINK-10247, the internal metrics query service has been separated
> into its own actor system. Before this change, the JobManager (JM) queried
> TaskManager (TM) metrics via the TM actor. Now, the JM needs to establish a
> separate connection to the TM metrics query service actor.
> In the context of Kubernetes, this is problematic as the JM will typically
> *not* be able to resolve the TMs by name, resulting in warnings as follows:
> {code}
> 2018-12-11 08:32:33,962 WARN akka.remote.ReliableDeliverySupervisor
> - Association with remote system
> [akka.tcp://flink-metrics@flink-task-manager-64b868487c-x9l4b:39183] has
> failed, address is now gated for [50] ms. Reason: [Association failed with
> [akka.tcp://flink-metrics@flink-task-manager-64b868487c-x9l4b:39183]] Caused
> by: [flink-task-manager-64b868487c-x9l4b: Name does not resolve]
> {code}
> In order to expose the TMs by name in Kubernetes, users require a service
> *for each* TM instance which is not practical.
> This currently results in the web UI not being to display some basic metrics
> about number of sent records. You can reproduce this by following the READMEs
> in {{flink-container/kubernetes}}.
> This worked before, because the JM is typically exposed via a service with a
> known name and the TMs establish the connection to it which the metrics query
> service piggybacked on.
> A potential solution to this might be to let the query service connect to the
> JM similar to how the TMs register.
> I tagged this ticket as an improvement, but in the context of Kubernetes I
> would consider this to be a bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12987: [FLINK-18703][table] Use new data structure converters when legacy types are not present
flinkbot edited a comment on pull request #12987: URL: https://github.com/apache/flink/pull/12987#issuecomment-663517173 ## CI report: * deab377fd5b947ff0f8dd5ed8e06274a4aaa7cdd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4885) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangXingBo opened a new pull request #12990: [FLINK-18708][docs] Update the sql client jar url of kafka 0.10 and 0.11
HuangXingBo opened a new pull request #12990: URL: https://github.com/apache/flink/pull/12990 ## What is the purpose of the change *This pull request update the sql client jar url of kafka 0.10 and 0.11* ## Brief change log - *update the sql client jar url of kafka 0.10 and 0.11* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
azagrebin commented on a change in pull request #12981:
URL: https://github.com/apache/flink/pull/12981#discussion_r460109160
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java
##
@@ -122,15 +128,15 @@ void reserveMemory(long size) throws
MemoryReservationException {
if (availableOrReserved >= size) {
return;
}
- if (sleeps >= MAX_SLEEPS) {
+ if (sleeps >= maxSleeps) {
break;
}
- if (sleeps >= RETRIGGER_GC_AFTER_SLEEPS) {
- // trigger again VM's Reference
processing if we have to wait longer
- System.gc();
- }
try {
if
(!JavaGcCleanerWrapper.tryRunPendingCleaners()) {
+ if (sleeps >=
RETRIGGER_GC_AFTER_SLEEPS) {
Review comment:
no, it is even a fix, see #12980. the commit is from #12980
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
azagrebin commented on a change in pull request #12981:
URL: https://github.com/apache/flink/pull/12981#discussion_r460109361
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlot.java
##
@@ -295,22 +303,25 @@ public String toString() {
// and set the slot state to releasing so that
it gets eventually freed
tasks.values().forEach(task ->
task.failExternally(cause));
}
+
final CompletableFuture cleanupFuture =
FutureUtils
Review comment:
true
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
azagrebin commented on a change in pull request #12981:
URL: https://github.com/apache/flink/pull/12981#discussion_r460108755
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java
##
@@ -61,7 +62,8 @@ long getAvailableMemorySize() {
boolean verifyEmpty() {
try {
- reserveMemory(totalMemorySize);
+ // we wait longer as we have to GC all memory,
allocated by task, to perform the verification
Review comment:
indeed details would be good
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
azagrebin commented on a change in pull request #12981:
URL: https://github.com/apache/flink/pull/12981#discussion_r460108171
##
File path:
flink-core/src/main/java/org/apache/flink/util/JavaGcCleanerWrapper.java
##
@@ -303,32 +302,38 @@ private Runnable create(Object owner, Runnable
cleanupOperation) {
private static class PendingCleanersRunnerProvider {
private static final String REFERENCE_CLASS =
"java.lang.ref.Reference";
private final String cleanerName;
- private final ReflectionUtils reflectionUtils;
private final String waitForReferenceProcessingName;
private final Object[] waitForReferenceProcessingArgs;
private final Class[] waitForReferenceProcessingArgTypes;
private PendingCleanersRunnerProvider(
String cleanerName,
- ReflectionUtils reflectionUtils,
String waitForReferenceProcessingName,
Object[] waitForReferenceProcessingArgs,
Class[] waitForReferenceProcessingArgTypes) {
this.cleanerName = cleanerName;
- this.reflectionUtils = reflectionUtils;
this.waitForReferenceProcessingName =
waitForReferenceProcessingName;
this.waitForReferenceProcessingArgs =
waitForReferenceProcessingArgs;
this.waitForReferenceProcessingArgTypes =
waitForReferenceProcessingArgTypes;
}
+ @Nullable
private PendingCleanersRunner createPendingCleanersRunner() {
Review comment:
This is used for one private purpose to consume the result atm, I can do
it but not sure about benefits of `Optional` here, maybe by later refactoring.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java
##
@@ -37,9 +37,10 @@
class UnsafeMemoryBudget {
// max. number of sleeps during try-reserving with exponentially
// increasing delay before throwing OutOfMemoryError:
- // 1, 2, 4, 8, 16, 32, 64, 128, 256, 512 (total 1023 ms ~ 1 s)
+ // 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, ... (total (2 x 1024) - 1 ms
~ 2 s)
Review comment:
true
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
azagrebin commented on a change in pull request #12981:
URL: https://github.com/apache/flink/pull/12981#discussion_r460106701
##
File path:
flink-core/src/main/java/org/apache/flink/util/JavaGcCleanerWrapper.java
##
@@ -303,32 +302,38 @@ private Runnable create(Object owner, Runnable
cleanupOperation) {
private static class PendingCleanersRunnerProvider {
private static final String REFERENCE_CLASS =
"java.lang.ref.Reference";
private final String cleanerName;
- private final ReflectionUtils reflectionUtils;
private final String waitForReferenceProcessingName;
private final Object[] waitForReferenceProcessingArgs;
private final Class[] waitForReferenceProcessingArgTypes;
private PendingCleanersRunnerProvider(
String cleanerName,
- ReflectionUtils reflectionUtils,
String waitForReferenceProcessingName,
Object[] waitForReferenceProcessingArgs,
Class[] waitForReferenceProcessingArgTypes) {
this.cleanerName = cleanerName;
- this.reflectionUtils = reflectionUtils;
this.waitForReferenceProcessingName =
waitForReferenceProcessingName;
this.waitForReferenceProcessingArgs =
waitForReferenceProcessingArgs;
this.waitForReferenceProcessingArgTypes =
waitForReferenceProcessingArgTypes;
}
+ @Nullable
private PendingCleanersRunner createPendingCleanersRunner() {
- Class referenceClass =
reflectionUtils.findClass(REFERENCE_CLASS);
- Method waitForReferenceProcessingMethod =
reflectionUtils.findMethod(
- referenceClass,
- waitForReferenceProcessingName,
- waitForReferenceProcessingArgTypes);
- waitForReferenceProcessingMethod.setAccessible(true);
- return new
PendingCleanersRunner(waitForReferenceProcessingMethod,
waitForReferenceProcessingArgs);
+ try {
+ Class referenceClass =
Class.forName(REFERENCE_CLASS);
+ Method waitForReferenceProcessingMethod =
referenceClass.getDeclaredMethod(
+ waitForReferenceProcessingName,
+ waitForReferenceProcessingArgTypes);
+
waitForReferenceProcessingMethod.setAccessible(true);
+ return new
PendingCleanersRunner(waitForReferenceProcessingMethod,
waitForReferenceProcessingArgs);
+ } catch (ClassNotFoundException | NoSuchMethodException
e) {
Review comment:
`findClass(String clazz, FunctionWithException,
FlinkRuntimeException>)` seems to be too much for one use atm. I do not see big
difference between checking `FlinkRuntimeException` and catching the exceptions
explicitly, except it is explicit which is mostly the asked reason. There is no
much code duplication or reuse. `ReflectionUtils` just packs the checked
exceptions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
azagrebin commented on a change in pull request #12981:
URL: https://github.com/apache/flink/pull/12981#discussion_r460106701
##
File path:
flink-core/src/main/java/org/apache/flink/util/JavaGcCleanerWrapper.java
##
@@ -303,32 +302,38 @@ private Runnable create(Object owner, Runnable
cleanupOperation) {
private static class PendingCleanersRunnerProvider {
private static final String REFERENCE_CLASS =
"java.lang.ref.Reference";
private final String cleanerName;
- private final ReflectionUtils reflectionUtils;
private final String waitForReferenceProcessingName;
private final Object[] waitForReferenceProcessingArgs;
private final Class[] waitForReferenceProcessingArgTypes;
private PendingCleanersRunnerProvider(
String cleanerName,
- ReflectionUtils reflectionUtils,
String waitForReferenceProcessingName,
Object[] waitForReferenceProcessingArgs,
Class[] waitForReferenceProcessingArgTypes) {
this.cleanerName = cleanerName;
- this.reflectionUtils = reflectionUtils;
this.waitForReferenceProcessingName =
waitForReferenceProcessingName;
this.waitForReferenceProcessingArgs =
waitForReferenceProcessingArgs;
this.waitForReferenceProcessingArgTypes =
waitForReferenceProcessingArgTypes;
}
+ @Nullable
private PendingCleanersRunner createPendingCleanersRunner() {
- Class referenceClass =
reflectionUtils.findClass(REFERENCE_CLASS);
- Method waitForReferenceProcessingMethod =
reflectionUtils.findMethod(
- referenceClass,
- waitForReferenceProcessingName,
- waitForReferenceProcessingArgTypes);
- waitForReferenceProcessingMethod.setAccessible(true);
- return new
PendingCleanersRunner(waitForReferenceProcessingMethod,
waitForReferenceProcessingArgs);
+ try {
+ Class referenceClass =
Class.forName(REFERENCE_CLASS);
+ Method waitForReferenceProcessingMethod =
referenceClass.getDeclaredMethod(
+ waitForReferenceProcessingName,
+ waitForReferenceProcessingArgTypes);
+
waitForReferenceProcessingMethod.setAccessible(true);
+ return new
PendingCleanersRunner(waitForReferenceProcessingMethod,
waitForReferenceProcessingArgs);
+ } catch (ClassNotFoundException | NoSuchMethodException
e) {
Review comment:
`findClass(String clazz, FunctionWithException,
FlinkRuntimeException>)` seems to be too much for one use atm. I do not see big
difference between checking `FlinkRuntimeException` and catching the exceptions
explicitly, except it is explicit which is mostly the asked reason. There is no
much code duplication or reuse.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
azagrebin commented on a change in pull request #12981:
URL: https://github.com/apache/flink/pull/12981#discussion_r460103524
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java
##
@@ -147,8 +147,10 @@ void reserveMemory(long size, int maxSleeps) throws
MemoryReservationException {
}
// no luck
- throw new MemoryReservationException(
- String.format("Could not allocate %d bytes,
only %d bytes are remaining", size, availableOrReserved));
+ throw new MemoryReservationException(String.format(
+ "Could not allocate %d bytes, only %d bytes are
remaining, try to upgrade to Java 8u72 or higher",
Review comment:
The warning can be overlooked. We agreed with @tillrohrmann that we want
to duplicate it here if the warning has been overlooked. Maybe, we could add a
bit details why to upgrade or ask `JavaGcCleanerWrapper` whether running
cleaners is supported by the JVM in use to make the hint conditional.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
azagrebin commented on a change in pull request #12981:
URL: https://github.com/apache/flink/pull/12981#discussion_r460103524
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java
##
@@ -147,8 +147,10 @@ void reserveMemory(long size, int maxSleeps) throws
MemoryReservationException {
}
// no luck
- throw new MemoryReservationException(
- String.format("Could not allocate %d bytes,
only %d bytes are remaining", size, availableOrReserved));
+ throw new MemoryReservationException(String.format(
+ "Could not allocate %d bytes, only %d bytes are
remaining, try to upgrade to Java 8u72 or higher",
Review comment:
The warning can be overlooked. We agreed with @tillrohrmann that we want
to duplicate it here if the warning has been overlooked.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] azagrebin commented on pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
azagrebin commented on pull request #12981: URL: https://github.com/apache/flink/pull/12981#issuecomment-663577682 @zentol thanks for the review sorry, I did not mention that the PR was based on #12980 so only the last commit belongs to this PR. I will address comments in #12980. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12981: [FLINK-18581] Do not run GC phantom cleaners for Java < 8u72
azagrebin commented on a change in pull request #12981: URL: https://github.com/apache/flink/pull/12981#discussion_r460101119 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/slot/TaskSlotTableImpl.java ## @@ -134,6 +143,12 @@ public TaskSlotTableImpl( slotActions = null; state = State.CREATED; closingFuture = new CompletableFuture<>(); + + asyncExecutor = new ThreadPoolExecutor( + 0, + numberSlots, + 60L, TimeUnit.SECONDS, + new SynchronousQueue<>()); Review comment: What kind of test do you mean? I just want to use it as a cached executor. I want it to allocate a new thread if there is no idle one for each slot closing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12989: [FLINK-18656][network,metrics] Fix startDelay metric for unaligned checkpoints
flinkbot commented on pull request #12989: URL: https://github.com/apache/flink/pull/12989#issuecomment-663576679 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 77a6fb4965d6896483b3753d1b06f2241e50b90d (Fri Jul 24 14:49:28 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18656) Start Delay metric is always zero for unaligned checkpoints
[
https://issues.apache.org/jira/browse/FLINK-18656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-18656:
---
Labels: pull-request-available (was: )
> Start Delay metric is always zero for unaligned checkpoints
> ---
>
> Key: FLINK-18656
> URL: https://issues.apache.org/jira/browse/FLINK-18656
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Metrics, Runtime / Network
>Affects Versions: 1.11.1
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2
>
> Attachments: Screenshot 2020-07-21 at 09.53.21.png, Screenshot
> 2020-07-21 at 09.58.29.png
>
>
> As visible in the attached screenshot, start delay is 0, despite very high
> end to end time and very quick async/sync times. This is impossible, as
> "start delay" should be equal to "end_to_end_time - sync_time - async_time".
> The bug is probably in {{CheckpointBarrierUnaligner}} class, which seems to
> never invoke {{CheckpointBarrierHandler#markCheckpointStart}}.
> *It's important to make sure this metric is also set for source tasks* (time
> to acquire checkpoint lock).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12987: [FLINK-18703][table] Use new data structure converters when legacy types are not present
flinkbot edited a comment on pull request #12987: URL: https://github.com/apache/flink/pull/12987#issuecomment-663517173 ## CI report: * 03db38a2348a080c970f5f068e24caad704d5b61 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4883) * deab377fd5b947ff0f8dd5ed8e06274a4aaa7cdd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4885) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-18708) The links of the connector sql jar of Kafka 0.10 and 0.11 are extinct
[ https://issues.apache.org/jira/browse/FLINK-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu reassigned FLINK-18708: --- Assignee: Huang Xingbo > The links of the connector sql jar of Kafka 0.10 and 0.11 are extinct > - > > Key: FLINK-18708 > URL: https://issues.apache.org/jira/browse/FLINK-18708 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka, Documentation >Affects Versions: 1.12.0, 1.11.1 >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > > The links of the connector sql jar of Kafka 0.10 and 0.11 are extinct. I will > fix it as soon as possible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] pnowojski opened a new pull request #12989: [FLINK-18656][network,metrics] Fix startDelay metric for unaligned checkpoints
pnowojski opened a new pull request #12989: URL: https://github.com/apache/flink/pull/12989 Before this fix, `startDelay` metric was always set to zero for unaligned checkpoints. ## Verifying this change This change adds a unit test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12984: [FLINK-18552][tests] Update migration tests in release-1.11 to cover migration for 1.11
flinkbot edited a comment on pull request #12984: URL: https://github.com/apache/flink/pull/12984#issuecomment-663488263 ## CI report: * ce9bcade4e123fea37f054be3d8229ecea5d8646 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4877) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12919: [FLINK-16048][avro] Support read/write confluent schema registry avro…
flinkbot edited a comment on pull request #12919: URL: https://github.com/apache/flink/pull/12919#issuecomment-659928275 ## CI report: * d1e4ba7690be134e21d193fbc1cb01aa51aaeb9b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4703) * f0da3cee91d22ec20cbba1b6c5be45da1440cf05 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4713) * 9104e12b0394cd6d578d2380ca4554b75e6e00f9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4730) * 9b557c718fe731e8d5c58e7c5d9c3452a245ee5a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4769) * 71218ee49095663a641e56889831536a2a2e69ea Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4818) * 6bd8c02de778a8ff2f34a19d5beee414beac3f69 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4875) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #12987: [FLINK-18703][table] Use new data structure converters when legacy types are not present
aljoscha commented on a change in pull request #12987:
URL: https://github.com/apache/flink/pull/12987#discussion_r460091224
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/codegen/SinkCodeGenerator.scala
##
@@ -102,7 +102,8 @@ object SinkCodeGenerator {
}
val consumedDataType = sink.getConsumedDataType
-val outTerm = genToExternal(ctx, physicalOutputType, afterIndexModify)
+// still uses the old conversion stack due to FLINK-18701
Review comment:
We will merge this as is without first adressing FLINK-18701?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12987: [FLINK-18703][table] Use new data structure converters when legacy types are not present
flinkbot edited a comment on pull request #12987: URL: https://github.com/apache/flink/pull/12987#issuecomment-663517173 ## CI report: * 03db38a2348a080c970f5f068e24caad704d5b61 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4883) * deab377fd5b947ff0f8dd5ed8e06274a4aaa7cdd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12972: [FLINK-18552][tests] Update migration tests in master to cover migration for 1.11
flinkbot edited a comment on pull request #12972: URL: https://github.com/apache/flink/pull/12972#issuecomment-663126426 ## CI report: * a68d9141d36092e9bcdcb4b98954514d5053e71a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4876) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18708) The links of the connector sql jar of Kafka 0.10 and 0.11 are extinct
Huang Xingbo created FLINK-18708: Summary: The links of the connector sql jar of Kafka 0.10 and 0.11 are extinct Key: FLINK-18708 URL: https://issues.apache.org/jira/browse/FLINK-18708 Project: Flink Issue Type: Bug Components: Connectors / Kafka, Documentation Affects Versions: 1.11.1, 1.12.0 Reporter: Huang Xingbo The links of the connector sql jar of Kafka 0.10 and 0.11 are extinct. I will fix it as soon as possible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-7129) Support dynamically changing CEP patterns
[ https://issues.apache.org/jira/browse/FLINK-7129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164441#comment-17164441 ] Lucas Senn edited comment on FLINK-7129 at 7/24/20, 2:25 PM: - [~knaufk] you're sure this approach will work? Based on this comment and [this|[https://flink.apache.org/news/2020/03/24/demo-fraud-detection-2.html]] article we then will try to implement such a case. Just wanted to ensure, that your comment wasn't just posted without thinking it through. :) But as I see it, this is a common functionality, that will be needed by many consumers. Can we expect a the deliver of the proposed Co Operator at anytime or will it be abondened? Is there a possibility to get an example as you think how a multi pattern job would have to look like? was (Author: lukefilwalker): [~knaufk] you're sure this approach will work? Based on this comment and [this|[https://flink.apache.org/news/2020/03/24/demo-fraud-detection-2.html]] article we then will try to implement such a case. Just wanted to ensure, that your comment wasn't just posted without thinking it through. :) But as I see it, this is a common functionality, that will be needed by many consumers. Can we expect a the deliver of the proposed Co Operator at anytime or will it be abondened? > Support dynamically changing CEP patterns > - > > Key: FLINK-7129 > URL: https://issues.apache.org/jira/browse/FLINK-7129 > Project: Flink > Issue Type: New Feature > Components: Library / CEP >Reporter: Dawid Wysakowicz >Priority: Major > > An umbrella task for introducing mechanism for injecting patterns through > coStream -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-7129) Support dynamically changing CEP patterns
[ https://issues.apache.org/jira/browse/FLINK-7129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164441#comment-17164441 ] Lucas Senn edited comment on FLINK-7129 at 7/24/20, 2:24 PM: - [~knaufk] you're sure this approach will work? Based on this comment and [this|[https://flink.apache.org/news/2020/03/24/demo-fraud-detection-2.html]] article we then will try to implement such a case. Just wanted to ensure, that your comment wasn't just posted without thinking it through. :) But as I see it, this is a common functionality, that will be needed by many consumers. Can we expect a the deliver of the proposed Co Operator at anytime or will it be abondened? was (Author: lukefilwalker): [~knaufk] you're sure this approach will work? Based on this comment and [this|[https://flink.apache.org/news/2020/03/24/demo-fraud-detection-2.html]] article we then will try to implement such a case. Just wanted to ensure, that your comment wasn't just posted without thinking it through. :) But as I see it, this is a common functionality, that will be needed. As I see it - Stream Processing will be a successor for Business Rule Engines (Drools etc.) in the future, if they develop the needed high level API's so that everyone is able to build simple BRE's without a bigger effort. > Support dynamically changing CEP patterns > - > > Key: FLINK-7129 > URL: https://issues.apache.org/jira/browse/FLINK-7129 > Project: Flink > Issue Type: New Feature > Components: Library / CEP >Reporter: Dawid Wysakowicz >Priority: Major > > An umbrella task for introducing mechanism for injecting patterns through > coStream -- This message was sent by Atlassian Jira (v8.3.4#803005)
