[jira] [Updated] (FLINK-21600) Resuming Savepoint (rocks, no parallelism change, heap timers) end-to-end test failed
[
https://issues.apache.org/jira/browse/FLINK-21600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guowei Ma updated FLINK-21600:
--
Description:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14081=logs=6caf31d6-847a-526e-9624-468e053467d6=0b23652f-b18b-5b6e-6eb6-a11070364610
{code:java}
2021-03-03T23:23:22.9286204Z The program finished with the following exception:
2021-03-03T23:23:22.9286551Z
2021-03-03T23:23:22.9287394Z org.apache.flink.util.FlinkException: Could not
stop with a savepoint job "6c88aa43b703192e88987473e722c22c".
2021-03-03T23:23:22.9288135Zat
org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:581)
2021-03-03T23:23:22.9288846Zat
org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002)
2021-03-03T23:23:22.9289787Zat
org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:569)
2021-03-03T23:23:22.9290721Zat
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1069)
2021-03-03T23:23:22.9291800Zat
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
2021-03-03T23:23:22.9292565Zat
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
2021-03-03T23:23:22.9293368Zat
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
2021-03-03T23:23:22.9294111Z Caused by: java.util.concurrent.TimeoutException
2021-03-03T23:23:22.9294842Zat
java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
2021-03-03T23:23:22.9295735Zat
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
2021-03-03T23:23:22.9296418Zat
org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:579)
2021-03-03T23:23:22.9297527Z... 6 more
2021-03-03T23:23:22.9622887Z Mar 03 23:23:22 Waiting for job
(6c88aa43b703192e88987473e722c22c) to reach terminal state FINISHED ...
2021-03-03T23:36:57.3094695Z Mar 03 23:36:57 Test (pid: 3517) did not finish
after 900 seconds.
{code}
killing logs
{code:java}
2021-03-03T23:36:57.3232128Z Mar 03 23:36:57
java.util.concurrent.RejectedExecutionException: event executor terminated
2021-03-03T23:36:57.3233348Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3234766Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3236170Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3237567Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3265526Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3267472Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:471)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3268806Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:87)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3275491Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3277092Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3279030Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:323)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3280277Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:155)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3281591Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:139)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3282748Z Mar 03 23:36:57at
[jira] [Updated] (FLINK-21594) Flink - {"errors":["Service temporarily unavailable due to an ongoing leader election. Please refresh."]}
[
https://issues.apache.org/jira/browse/FLINK-21594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bhagi updated FLINK-21594:
--
Attachment: image-2021-03-04-13-27-00-728.png
> Flink - {"errors":["Service temporarily unavailable due to an ongoing leader
> election. Please refresh."]}
> -
>
> Key: FLINK-21594
> URL: https://issues.apache.org/jira/browse/FLINK-21594
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Affects Versions: 1.12.0
> Environment: debian kubernetes
>Reporter: Bhagi
>Priority: Critical
> Fix For: 1.12.0
>
> Attachments: image-2021-03-04-11-59-24-095.png,
> image-2021-03-04-12-06-12-504.png, image-2021-03-04-13-27-00-728.png
>
>
> Hi Team,
> I deployed standalone Flink session cluster on kubernetes.. i have created
> the HA with 3 job managers. But Flink Web UI is throwing an error saying
> "\{"errors":["Service temporarily unavailable due to an ongoing leader
> election. Please refresh."]}".
> Please help to fix the issue with UI and HA.
> Logs are showing with the Leader election and updated to the all configmaps..
> below configmaps are created in the same namespace where flink cluster is
> running.
> 111-dispatcher-leader
> 111-resourcemanager-leader
> 111-restserver-leader
> but i have one doubt on configmap created, when i described above configmaps
> showing type as flink-native-kubernetes.. But i created cluster with flink
> standalone kubernetes.
> configmap-type: high-availability
> type: flink-native-kubernetes
>
> 2) Let me know how to confirm the whether flink is running on native
> kubernetes or standalone kubernetes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21600) Resuming Savepoint (rocks, no parallelism change, heap timers) end-to-end test failed
Guowei Ma created FLINK-21600:
-
Summary: Resuming Savepoint (rocks, no parallelism change, heap
timers) end-to-end test failed
Key: FLINK-21600
URL: https://issues.apache.org/jira/browse/FLINK-21600
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 1.12.2
Reporter: Guowei Ma
{code:java}
2021-03-03T23:23:22.9286204Z The program finished with the following exception:
2021-03-03T23:23:22.9286551Z
2021-03-03T23:23:22.9287394Z org.apache.flink.util.FlinkException: Could not
stop with a savepoint job "6c88aa43b703192e88987473e722c22c".
2021-03-03T23:23:22.9288135Zat
org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:581)
2021-03-03T23:23:22.9288846Zat
org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002)
2021-03-03T23:23:22.9289787Zat
org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:569)
2021-03-03T23:23:22.9290721Zat
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1069)
2021-03-03T23:23:22.9291800Zat
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
2021-03-03T23:23:22.9292565Zat
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
2021-03-03T23:23:22.9293368Zat
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
2021-03-03T23:23:22.9294111Z Caused by: java.util.concurrent.TimeoutException
2021-03-03T23:23:22.9294842Zat
java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
2021-03-03T23:23:22.9295735Zat
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
2021-03-03T23:23:22.9296418Zat
org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:579)
2021-03-03T23:23:22.9297527Z... 6 more
2021-03-03T23:23:22.9622887Z Mar 03 23:23:22 Waiting for job
(6c88aa43b703192e88987473e722c22c) to reach terminal state FINISHED ...
2021-03-03T23:36:57.3094695Z Mar 03 23:36:57 Test (pid: 3517) did not finish
after 900 seconds.
{code}
killing logs
{code:java}
2021-03-03T23:36:57.3232128Z Mar 03 23:36:57
java.util.concurrent.RejectedExecutionException: event executor terminated
2021-03-03T23:36:57.3233348Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3234766Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3236170Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3237567Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3265526Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3267472Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:471)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3268806Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:87)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3275491Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3277092Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3279030Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:323)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3280277Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:155)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3281591Z Mar 03 23:36:57at
org.apache.flink.shaded.netty4.io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:139)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
[jira] [Commented] (FLINK-21591) Flink using OSS Backend on Native K8S found some errors
[
https://issues.apache.org/jira/browse/FLINK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295069#comment-17295069
]
Yang Wang commented on FLINK-21591:
---
It seems to be a INFO log of aliyun oss. Could you please check whether the
checkpoint has been stored successfully?
> Flink using OSS Backend on Native K8S found some errors
> ---
>
> Key: FLINK-21591
> URL: https://issues.apache.org/jira/browse/FLINK-21591
> Project: Flink
> Issue Type: Bug
>Reporter: yufan wang
>Priority: Blocker
>
> version: flink 1.12.0
> environment: Native Kubernetes
> mode: Application Mode
> backend: filesystem OSS
>
> Using OSS as the backend. When the job starts, I get some error logs with
> Jobmanager and Taskmanager.
> {code:java}
> 2021-03-04 02:55:02,634 WARN
> org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - Unable to
> find 'mime.types' file in classpath
> 2021-03-04 02:55:02,768 INFO
> org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
> [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C38387B9F21
> [HostId]: null
> {code}
> start command:
> {code:java}
> ./bin/flink run-application \
> --target kubernetes-application \
> -Dkubernetes.cluster-id=demo \
> -Dkubernetes.container.image=xx/xx/job:2.0.20 \
> -Dstate.backend=filesystem \
> -Dstate.checkpoints.dir=oss://xx/backend/ \
> -Dfs.oss.endpoint=oss-cn-beijing-internal.aliyuncs.com \
> -Dfs.oss.accessKeyId=xx \
> -Dfs.oss.accessKeySecret=xx \
> -Dhigh-availability.storageDir=oss://xx/flink-ha/ \
>
> -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
> \
> local:///opt/flink/usrlib/my-flink-job.jar
> {code}
> morelog:
> {code:java}
> Start command: $JAVA_HOME/bin/java -classpath $FLINK_CLASSPATH -Xmx1073741824
> -Xms1073741824 -XX:MaxMetaspaceSize=268435456
> -Dlog.file=/opt/flink/log/jobmanager.log
> -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
> -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
> -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
> org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
> -D jobmanager.memory.off-heap.size=134217728b -D
> jobmanager.memory.jvm-overhead.min=201326592b -D
> jobmanager.memory.jvm-metaspace.size=268435456b -D
> jobmanager.memory.heap.size=1073741824b -D
> jobmanager.memory.jvm-overhead.max=201326592b
> 2021-03-04 02:54:50,721 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] -
>
> 2021-03-04 02:54:50,723 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Starting
> KubernetesApplicationClusterEntrypoint (Version: 1.12.0, Scala: 2.12,
> Rev:fc00492, Date:2020-12-02T08:49:16+01:00)
> 2021-03-04 02:54:50,723 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - OS current
> user: flink
> 2021-03-04 02:54:50,724 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Current
> Hadoop/Kerberos user:
> 2021-03-04 02:54:50,724 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - JVM:
> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.275-b01
> 2021-03-04 02:54:50,724 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Maximum
> heap size: 989 MiBytes
> 2021-03-04 02:54:50,724 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - JAVA_HOME:
> /usr/local/openjdk-8
> 2021-03-04 02:54:50,773 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - No Hadoop
> Dependency available
> 2021-03-04 02:54:50,773 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - JVM
> Options:
> 2021-03-04 02:54:50,774 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] -
> -Xmx1073741824
> 2021-03-04 02:54:50,774 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] -
> -Xms1073741824
> 2021-03-04 02:54:50,774 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] -
> -XX:MaxMetaspaceSize=268435456
> 2021-03-04 02:54:50,774 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] -
> -Dlog.file=/opt/flink/log/jobmanager.log
> 2021-03-04 02:54:50,774 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] -
> -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
> 2021-03-04 02:54:50,774 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] -
> -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
> 2021-03-04 02:54:50,774 INFO
>
[GitHub] [flink-docker] rmetzger commented on pull request #67: Dev 1.12: Add GPG and update testing script for 1.12.2
rmetzger commented on pull request #67: URL: https://github.com/apache/flink-docker/pull/67#issuecomment-790400272 It seems there's something wrong with the `wget -nv -O flink.tgz https://www.apache.org/dyn/closer.cgi?action=download=flink/flink-1.12.2/flink-1.12.2-bin-scala_2.11.tgz` command. Also locally, I'm facing issues with the wget call (downloading from the browser works). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-21599) execute[Parallel union, p = 10](org.apache.flink.test.checkpointing.UnalignedCheckpointITCase) failed because of oom.
[
https://issues.apache.org/jira/browse/FLINK-21599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guowei Ma closed FLINK-21599.
-
Resolution: Duplicate
> execute[Parallel union, p =
> 10](org.apache.flink.test.checkpointing.UnalignedCheckpointITCase) failed
> because of oom.
> -
>
> Key: FLINK-21599
> URL: https://issues.apache.org/jira/browse/FLINK-21599
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.12.2
>Reporter: Guowei Ma
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14081=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56
> {code:java}
> 2021-03-04T01:33:19.3512653Z [ERROR] execute[Parallel union, p =
> 10](org.apache.flink.test.checkpointing.UnalignedCheckpointITCase) Time
> elapsed: 885.628 s <<< ERROR!
> 2021-03-04T01:33:19.3513465Z
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2021-03-04T01:33:19.3514306Z at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> 2021-03-04T01:33:19.3515086Z at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$2(MiniClusterJobClient.java:117)
> 2021-03-04T01:33:19.3515844Z at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> 2021-03-04T01:33:19.3516518Z at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> 2021-03-04T01:33:19.3517201Z at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2021-03-04T01:33:19.3517871Z at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> 2021-03-04T01:33:19.3518603Z at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
> 2021-03-04T01:33:19.3519346Z at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> 2021-03-04T01:33:19.3520057Z at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> 2021-03-04T01:33:19.3520747Z at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2021-03-04T01:33:19.3521409Z at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> 2021-03-04T01:33:19.3522083Z at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1046)
> 2021-03-04T01:33:19.3522700Z at
> akka.dispatch.OnComplete.internal(Future.scala:264)
> 2021-03-04T01:33:19.3523236Z at
> akka.dispatch.OnComplete.internal(Future.scala:261)
> 2021-03-04T01:33:19.3523787Z at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
> 2021-03-04T01:33:19.3524356Z at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
> 2021-03-04T01:33:19.3524914Z at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> 2021-03-04T01:33:19.3525581Z at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
> 2021-03-04T01:33:19.3526379Z at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> 2021-03-04T01:33:19.3527022Z at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> 2021-03-04T01:33:19.3527630Z at
> akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
> 2021-03-04T01:33:19.3528284Z at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
> 2021-03-04T01:33:19.3529023Z at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
> 2021-03-04T01:33:19.3529695Z at
> scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
> 2021-03-04T01:33:19.3530308Z at
> scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
> 2021-03-04T01:33:19.3530905Z at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> 2021-03-04T01:33:19.3531545Z at
> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
> 2021-03-04T01:33:19.3532292Z at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
> 2021-03-04T01:33:19.3533047Z at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> 2021-03-04T01:33:19.3533789Z at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> 2021-03-04T01:33:19.3534469Z at
> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> 2021-03-04T01:33:19.3535120Z at
>
[jira] [Commented] (FLINK-21535) UnalignedCheckpointITCase.execute failed with "OutOfMemoryError: Java heap space"
[
https://issues.apache.org/jira/browse/FLINK-21535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295064#comment-17295064
]
Guowei Ma commented on FLINK-21535:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14081=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56
> UnalignedCheckpointITCase.execute failed with "OutOfMemoryError: Java heap
> space"
> -
>
> Key: FLINK-21535
> URL: https://issues.apache.org/jira/browse/FLINK-21535
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13866=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56
> {code}
> 2021-02-27T02:11:41.5659201Z
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2021-02-27T02:11:41.5659947Z at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> 2021-02-27T02:11:41.5660794Z at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
> 2021-02-27T02:11:41.5661618Z at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> 2021-02-27T02:11:41.5662356Z at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> 2021-02-27T02:11:41.5663104Z at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2021-02-27T02:11:41.5664016Z at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> 2021-02-27T02:11:41.5664817Z at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
> 2021-02-27T02:11:41.5665638Z at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> 2021-02-27T02:11:41.5666405Z at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> 2021-02-27T02:11:41.5667609Z at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2021-02-27T02:11:41.5668358Z at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> 2021-02-27T02:11:41.5669218Z at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1066)
> 2021-02-27T02:11:41.5669928Z at
> akka.dispatch.OnComplete.internal(Future.scala:264)
> 2021-02-27T02:11:41.5670540Z at
> akka.dispatch.OnComplete.internal(Future.scala:261)
> 2021-02-27T02:11:41.5671268Z at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
> 2021-02-27T02:11:41.5671881Z at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
> 2021-02-27T02:11:41.5672512Z at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> 2021-02-27T02:11:41.5673219Z at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
> 2021-02-27T02:11:41.5674085Z at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> 2021-02-27T02:11:41.5674794Z at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> 2021-02-27T02:11:41.5675466Z at
> akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
> 2021-02-27T02:11:41.5676181Z at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
> 2021-02-27T02:11:41.5676977Z at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
> 2021-02-27T02:11:41.5677717Z at
> scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
> 2021-02-27T02:11:41.5678409Z at
> scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
> 2021-02-27T02:11:41.5679071Z at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> 2021-02-27T02:11:41.5679776Z at
> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
> 2021-02-27T02:11:41.5680576Z at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
> 2021-02-27T02:11:41.5681383Z at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> 2021-02-27T02:11:41.5682167Z at
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
> 2021-02-27T02:11:41.5683040Z at
> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> 2021-02-27T02:11:41.5683759Z at
> akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
> 2021-02-27T02:11:41.5684493Z at
>
[jira] [Created] (FLINK-21599) execute[Parallel union, p = 10](org.apache.flink.test.checkpointing.UnalignedCheckpointITCase) failed because of oom.
Guowei Ma created FLINK-21599:
-
Summary: execute[Parallel union, p =
10](org.apache.flink.test.checkpointing.UnalignedCheckpointITCase) failed
because of oom.
Key: FLINK-21599
URL: https://issues.apache.org/jira/browse/FLINK-21599
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 1.12.2
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14081=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56
{code:java}
2021-03-04T01:33:19.3512653Z [ERROR] execute[Parallel union, p =
10](org.apache.flink.test.checkpointing.UnalignedCheckpointITCase) Time
elapsed: 885.628 s <<< ERROR!
2021-03-04T01:33:19.3513465Z
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-03-04T01:33:19.3514306Zat
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2021-03-04T01:33:19.3515086Zat
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$2(MiniClusterJobClient.java:117)
2021-03-04T01:33:19.3515844Zat
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
2021-03-04T01:33:19.3516518Zat
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
2021-03-04T01:33:19.3517201Zat
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-03-04T01:33:19.3517871Zat
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-03-04T01:33:19.3518603Zat
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
2021-03-04T01:33:19.3519346Zat
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-03-04T01:33:19.3520057Zat
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-03-04T01:33:19.3520747Zat
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-03-04T01:33:19.3521409Zat
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-03-04T01:33:19.3522083Zat
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1046)
2021-03-04T01:33:19.3522700Zat
akka.dispatch.OnComplete.internal(Future.scala:264)
2021-03-04T01:33:19.3523236Zat
akka.dispatch.OnComplete.internal(Future.scala:261)
2021-03-04T01:33:19.3523787Zat
akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
2021-03-04T01:33:19.3524356Zat
akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
2021-03-04T01:33:19.3524914Zat
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-03-04T01:33:19.3525581Zat
org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
2021-03-04T01:33:19.3526379Zat
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
2021-03-04T01:33:19.3527022Zat
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
2021-03-04T01:33:19.3527630Zat
akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
2021-03-04T01:33:19.3528284Zat
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
2021-03-04T01:33:19.3529023Zat
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
2021-03-04T01:33:19.3529695Zat
scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
2021-03-04T01:33:19.3530308Zat
scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
2021-03-04T01:33:19.3530905Zat
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-03-04T01:33:19.3531545Zat
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
2021-03-04T01:33:19.3532292Zat
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
2021-03-04T01:33:19.3533047Zat
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
2021-03-04T01:33:19.3533789Zat
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
2021-03-04T01:33:19.3534469Zat
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
2021-03-04T01:33:19.3535120Zat
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
2021-03-04T01:33:19.3535747Zat
akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
2021-03-04T01:33:19.3536430Zat
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
2021-03-04T01:33:19.3537124Zat
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
2021-03-04T01:33:19.3537758Zat
[GitHub] [flink-docker] rmetzger commented on pull request #67: Dev 1.12: Add GPG and update testing script for 1.12.2
rmetzger commented on pull request #67: URL: https://github.com/apache/flink-docker/pull/67#issuecomment-790393873 I manually restarted the travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15083: [FLINK-21242][python] Support state access API for the map/flat_map/filter/reduce operations of Python KeyedStream
flinkbot commented on pull request #15083: URL: https://github.com/apache/flink/pull/15083#issuecomment-790393119 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit edf533eccd54ccd16fca238397b1417fd98828c5 (Thu Mar 04 07:38:57 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on a change in pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on a change in pull request #14737:
URL: https://github.com/apache/flink/pull/14737#discussion_r587221640
##
File path:
flink-formats/flink-avro-glue-schema-registry/src/test/java/org/apache/flink/formats/avro/glue/schema/registry/User.java
##
@@ -0,0 +1,434 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.avro.glue.schema.registry;
+
+import org.apache.avro.message.BinaryMessageDecoder;
+import org.apache.avro.message.BinaryMessageEncoder;
+import org.apache.avro.message.SchemaStore;
+import org.apache.avro.specific.SpecificData;
+
+@SuppressWarnings("all")
+@org.apache.avro.specific.AvroGenerated
+public class User extends org.apache.avro.specific.SpecificRecordBase
Review comment:
Okay I misunderstood what the dependency does. Will address this in
follow up PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15082: [FLINK-21592][table-planner-blink] RemoveSingleAggregateRule fails du…
flinkbot edited a comment on pull request #15082: URL: https://github.com/apache/flink/pull/15082#issuecomment-790381856 ## CI report: * 02251f131e8c7d3c398a6a59617b1e7899150fec Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14101) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15058: [FLINK-21419][coordination] Remove GC cleaner mechanism for unsafe memory segments
flinkbot edited a comment on pull request #15058: URL: https://github.com/apache/flink/pull/15058#issuecomment-788795717 ## CI report: * 5693fa8be61c954d3662cc8ffe27689e5f6d097e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13991) * b2f53518552258f786ef4bf4031d78f24ba51440 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14100) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21242) Support state access API for the map/flat_map/filter/reduce operations of Python KeyedStream
[ https://issues.apache.org/jira/browse/FLINK-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-21242: --- Labels: pull-request-available (was: ) > Support state access API for the map/flat_map/filter/reduce operations of > Python KeyedStream > > > Key: FLINK-21242 > URL: https://issues.apache.org/jira/browse/FLINK-21242 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Wei Zhong >Assignee: Wei Zhong >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] WeiZhong94 opened a new pull request #15083: [FLINK-21242][python] Support state access API for the map/flat_map/filter/reduce operations of Python KeyedStream
WeiZhong94 opened a new pull request #15083: URL: https://github.com/apache/flink/pull/15083 ## What is the purpose of the change *This pull request supports state access API for the map/flat_map/filter/reduce operations of Python KeyedStream.* ## Verifying this change This change is already covered by existing tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21482) Fix bug when use grouping set/rollup/cube syntax in Window TVF
[
https://issues.apache.org/jira/browse/FLINK-21482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy updated FLINK-21482:
-
Summary: Fix bug when use grouping set/rollup/cube syntax in Window TVF
(was: Support grouping set syntax for Window TVF based aggregation)
> Fix bug when use grouping set/rollup/cube syntax in Window TVF
> ---
>
> Key: FLINK-21482
> URL: https://issues.apache.org/jira/browse/FLINK-21482
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API, Table SQL / Planner
>Reporter: Andy
>Assignee: Andy
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Now window tvf could work well like the following sql.
> {code:java}
> SELECT
> GROUPING_ID(`name`),
> `name`,
> window_start,
> window_end,
> COUNT(*),
> SUM(`bigdec`),
> MAX(`double`),
> MIN(`float`),
> COUNT(DISTINCT `string`),
> concat_distinct_agg(`string`)
> FROM TABLE(
>CUMULATE(
> TABLE T1,
> DESCRIPTOR(rowtime),
> INTERVAL '5' SECOND,
> INTERVAL '15' SECOND))
> GROUP BY GROUPING SETS((`name`),()), window_start, window_end{code}
>
> However, an exception would be thrown out when generate plan for the
> following sql. (We could not support this case, but we need a friendly tip
> for user.)
> {code:java}
> SELECT
>a,
>count(distinct c) AS uv
> FROM TABLE(TUMBLE(TABLE MyTable, DESCRIPTOR(rowtime), INTERVAL '15' MINUTE))
> GROUP BY GROUPING SETS ((a), (window_start), (window_end)){code}
> The exception like the following:
> {code:java}
> java.lang.AssertionError: Sql optimization: Assertion error: project filed
> count is not equal to output field count.java.lang.AssertionError: Sql
> optimization: Assertion error: project filed count is not equal to output
> field count.
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:84)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:62)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:58)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
> scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157) at
> scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104) at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:57)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.optimizeTree(StreamCommonSubGraphBasedOptimizer.scala:163)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.doOptimize(StreamCommonSubGraphBasedOptimizer.scala:79)
> at
> org.apache.flink.table.planner.plan.optimize.CommonSubGraphBasedOptimizer.optimize(CommonSubGraphBasedOptimizer.scala:77)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.optimize(PlannerBase.scala:288)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.assertPlanEquals(TableTestBase.scala:860)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.doVerifyPlan(TableTestBase.scala:751)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.verifyRelPlan(TableTestBase.scala:398)
> at
> org.apache.flink.table.planner.plan.stream.sql.agg.WindowAggregateTest.testCantMergeWindowTVF_GroupingSetsWithoutWindowStartEnd(WindowAggregateTest.scala:696)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at
>
[GitHub] [flink] beyond1920 commented on pull request #15081: [FLINK-21553][table-runtime-blink] Copy record if needed when flush window buffer records to state
beyond1920 commented on pull request #15081: URL: https://github.com/apache/flink/pull/15081#issuecomment-790389209 @wuchong Thanks for your review, I've address all your comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] leonardBang commented on a change in pull request #14684: [FLINK-20460][Connector-HBase] Support async lookup for HBase connector
leonardBang commented on a change in pull request #14684:
URL: https://github.com/apache/flink/pull/14684#discussion_r587208036
##
File path:
flink-connectors/flink-connector-hbase-base/src/main/java/org/apache/flink/connector/hbase/util/HBaseSerde.java
##
@@ -225,6 +217,13 @@ public Get createGet(Object rowKey) {
/** Converts HBase {@link Result} into {@link RowData}. */
public RowData convertToRow(Result result) {
+// The output rows needs to be initialized each time
Review comment:
Could we add a `reuse` flag for this function so that we can improve the
performance when user didn't enable the cache or used in the
HBaseRowDataInputFormat
##
File path:
flink-connectors/flink-connector-hbase-2.2/src/test/java/org/apache/flink/connector/hbase2/source/HBaseRowDataAsyncLookupFunctionTest.java
##
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.hbase2.source;
+
+import org.apache.flink.connector.hbase.options.HBaseLookupOptions;
+import org.apache.flink.connector.hbase.util.HBaseTableSchema;
+import org.apache.flink.connector.hbase.util.PlannerType;
+import org.apache.flink.connector.hbase2.util.HBaseTestBase;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.data.RowData;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.Lists;
+
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.stream.Collectors;
+
+import static org.junit.Assert.assertEquals;
+
+/** Test suite for {@link HBaseRowDataAsyncLookupFunction}. */
+public class HBaseRowDataAsyncLookupFunctionTest extends HBaseTestBase {
Review comment:
Could we add an ITCase for `AsyncLookupFunction` like
`HBaseConnectorITCase.testHBaseLookupTableSource` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-21414) javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
[
https://issues.apache.org/jira/browse/FLINK-21414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295056#comment-17295056
]
Matthias commented on FLINK-21414:
--
Hello Bhagi,
this could be a configuration issue. Have you been able to resolve it already?
As a side note: Things like that are more likely to be picked up on the [User
Mailing
List|https://flink.apache.org/community.html#how-to-subscribe-to-a-mailing-list].
> javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
> --
>
> Key: FLINK-21414
> URL: https://issues.apache.org/jira/browse/FLINK-21414
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Affects Versions: 1.12.0
> Environment: flink@flink-jobmanager-5d9594b76c-t6cc8:~$ cat
> /etc/os-release
> PRETTY_NAME="Debian GNU/Linux 10 (buster)"
> NAME="Debian GNU/Linux"
> VERSION_ID="10"
> VERSION="10 (buster)"
> VERSION_CODENAME=buster
> ID=debian
>Reporter: Bhagi
>Priority: Critical
> Fix For: 1.12.0
>
> Attachments: image-2021-02-19-12-26-27-296.png
>
>
> Enabled High availability for FLink with kubernetes API with Leader election
> and retirieval mechanism,After configuring HA parameters with Leader election
> and Leader retrival ..I started getting this error in job manager logs.
> +error logs in job manager logs:+
> at
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> ... 16 more
> 2021-02-19 06:11:04,358 WARN
> org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Unhandled
> exception
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException:
> javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at org
>
> +UI is throwing error:+
> {"errors":["Service temporarily unavailable due to an ongoing leader
> election. Please refresh."]}[link
> title|[http://example.com]|http://example.com]/]
> !image-2021-02-19-12-26-27-296.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] ruanhang1993 edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
ruanhang1993 edited a comment on pull request #14376:
URL: https://github.com/apache/flink/pull/14376#issuecomment-790247456
> @ruanhang1993 Many thanks for finding issues of this PR.
> I tried to solve two issues you raised.
>
> 1. Regarding the first issue, I relocate protobuf package in pom.xml
> 2. I'm not sure that open method will be called or not during checkpoint
recovery process. @libenchao Could you help answer this question?
@maosuhan I am honored to receive your reply. I use the fixed version for
problem 1 in flink 1.12.1 recently.
In flink 1.12.1, the problem 2 is gone. The `open` method will be called in
flink 1.12.1 during checkpoint recovery process.
With the fixed code for problem 1 in flink 1.12.1, I got the error like this:
```
Caused by: java.lang.ClassCastException:
com.google.protobuf.Descriptors$Descriptor cannot be cast to
org.apache.flink.formats.protobuf.shaded.com.google.protobuf.Descriptors$Descriptor
at
org.apache.flink.formats.protobuf.PbFormatUtils.getDescriptor(PbFormatUtils.java:81)
~[?:?]
at
org.apache.flink.formats.protobuf.deserialize.PbRowDataDeserializationSchema.(PbRowDataDeserializationSchema.java:67)
~[?:?]
at
org.apache.flink.formats.protobuf.PbDecodingFormat.createRuntimeDecoder(PbDecodingFormat.java:49)
~[?:?]
at
org.apache.flink.formats.protobuf.PbDecodingFormat.createRuntimeDecoder(PbDecodingFormat.java:31)
~[?:?]
at
org.apache.flink.streaming.connectors.kafka.table.KafkaDynamicSource.createDeserialization(KafkaDynamicSource.java:427)
~[?:?]
..
```
I generate the protobuf message class and package like this, and use it when
submitting the flink job.
```bash
protoc -I=$PROTO_FILE_PATH --java_out=$JAVA_OUTPUT/src $PROTO_FILE
javac -cp $JAVA_OUTPUT/libs/protobuf-java.jar -d $JAVA_OUTPUT/target
$JAVA_OUTPUT/src/$PACKAGE_PATH/*.java
cd $JAVA_OUTPUT
jar -cvf $JAR_NAME -C $JAVA_OUTPUT/target/ .
```
The problem is that the `getDescriptor` in my class return the
`com.google.protobuf.Descriptors$Descriptor` class, which can not cast to the
relocated class.
```java
public static Descriptors.Descriptor getDescriptor(String className) {
try {
Class pbClass = Class.forName(className);
return (Descriptors.Descriptor)
pbClass.getMethod(PbConstant.PB_METHOD_GET_DESCRIPTOR).invoke(null);
} catch (Exception y) {
throw new IllegalArgumentException(
String.format("get %s
descriptors error!", className), y);
}
}
```
Do I need to relocate the class like you when generating the protobuf
message class?
Or is there some other way to fix it?
ps: The setting in
`META-INF/services/org.apache.flink.table.factories.Factory` needs to be
changed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-21594) Flink - {"errors":["Service temporarily unavailable due to an ongoing leader election. Please refresh."]}
[
https://issues.apache.org/jira/browse/FLINK-21594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias closed FLINK-21594.
Resolution: Duplicate
Hi Bhagi,
it sounds like this is still the issue that is already discussed in
FLINK-21432. Could you move the discussion back into FLINK-21432? Please add
your findings to the older issue if you have followed [~fly_in_gis] proposal of
aligning the Kubernetes namespace.
I'm closing the issue for now.
> Flink - {"errors":["Service temporarily unavailable due to an ongoing leader
> election. Please refresh."]}
> -
>
> Key: FLINK-21594
> URL: https://issues.apache.org/jira/browse/FLINK-21594
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Affects Versions: 1.12.0
> Environment: debian kubernetes
>Reporter: Bhagi
>Priority: Critical
> Fix For: 1.12.0
>
> Attachments: image-2021-03-04-11-59-24-095.png,
> image-2021-03-04-12-06-12-504.png
>
>
> Hi Team,
> I deployed standalone Flink session cluster on kubernetes.. i have created
> the HA with 3 job managers. But Flink Web UI is throwing an error saying
> "\{"errors":["Service temporarily unavailable due to an ongoing leader
> election. Please refresh."]}".
> Please help to fix the issue with UI and HA.
> Logs are showing with the Leader election and updated to the all configmaps..
> below configmaps are created in the same namespace where flink cluster is
> running.
> 111-dispatcher-leader
> 111-resourcemanager-leader
> 111-restserver-leader
> but i have one doubt on configmap created, when i described above configmaps
> showing type as flink-native-kubernetes.. But i created cluster with flink
> standalone kubernetes.
> configmap-type: high-availability
> type: flink-native-kubernetes
>
> 2) Let me know how to confirm the whether flink is running on native
> kubernetes or standalone kubernetes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong commented on a change in pull request #15081: [FLINK-21553][table-runtime-blink] Copy record if needed when flush window buffer records to state
wuchong commented on a change in pull request #15081:
URL: https://github.com/apache/flink/pull/15081#discussion_r587210465
##
File path:
flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/WindowDistinctAggregateITCase.scala
##
@@ -733,8 +733,9 @@ object WindowDistinctAggregateITCase {
def parameters(): util.Collection[Array[java.lang.Object]] = {
Seq[Array[AnyRef]](
Array(Boolean.box(true), HEAP_BACKEND),
+ // add SplitDistinct for HEAP back
Review comment:
remove the comment
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #15081: [FLINK-21553][table-runtime-blink] Copy record if needed when flush window buffer records to state
wuchong commented on a change in pull request #15081:
URL: https://github.com/apache/flink/pull/15081#discussion_r587207786
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/aggregate/window/combines/CombineRecordsFunction.java
##
@@ -65,21 +70,31 @@ public CombineRecordsFunction(
StateKeyContext keyContext,
WindowValueState accState,
NamespaceAggsHandleFunction aggregator,
-boolean requiresCopyKey,
+boolean requiresCopy,
+RowType recordType,
boolean isEventTime) {
this.timerService = timerService;
this.keyContext = keyContext;
this.accState = accState;
this.aggregator = aggregator;
-this.requiresCopyKey = requiresCopyKey;
+this.requiresCopy = requiresCopy;
+if (requiresCopy) {
+LogicalType[] recordFieldTypes =
+recordType.getFields().stream()
+.map(RowType.RowField::getType)
+.toArray(LogicalType[]::new);
+this.recordSerializer = new RowDataSerializer(recordFieldTypes);
+} else {
+this.recordSerializer = null;
Review comment:
I think it's fine to always initialize `recordSerializer`.
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/aggregate/window/combines/CombineRecordsFunction.java
##
@@ -152,13 +174,14 @@ public WindowCombineFunction create(
aggregator.open(
new PerWindowStateDataViewStore(
stateBackend, LongSerializer.INSTANCE,
runtimeContext));
-boolean requiresCopyKey =
!isStateImmutableInStateBackend(stateBackend);
+boolean requiresCopyRecord =
!isStateImmutableInStateBackend(stateBackend);
Review comment:
`requiresCopyRecord` ==> `requiresCopy`.
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/aggregate/window/combines/CombineRecordsFunction.java
##
@@ -65,21 +70,31 @@ public CombineRecordsFunction(
StateKeyContext keyContext,
WindowValueState accState,
NamespaceAggsHandleFunction aggregator,
-boolean requiresCopyKey,
+boolean requiresCopy,
+RowType recordType,
boolean isEventTime) {
this.timerService = timerService;
this.keyContext = keyContext;
this.accState = accState;
this.aggregator = aggregator;
-this.requiresCopyKey = requiresCopyKey;
+this.requiresCopy = requiresCopy;
+if (requiresCopy) {
+LogicalType[] recordFieldTypes =
+recordType.getFields().stream()
+.map(RowType.RowField::getType)
+.toArray(LogicalType[]::new);
+this.recordSerializer = new RowDataSerializer(recordFieldTypes);
Review comment:
Why not `new RowDataSerializer(recordType)`?
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/aggregate/window/combines/CombineRecordsFunction.java
##
@@ -54,8 +57,10 @@
/** Function used to handle all aggregates. */
private final NamespaceAggsHandleFunction aggregator;
-/** Whether to copy input key, because key is reused. */
-private final boolean requiresCopyKey;
+/** Whether to copy input record, because record is reused. */
Review comment:
```suggestion
/** Whether to copy key and input record, because key and record are
reused. */
```
We should also mention key, because key and record are separate.
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/aggregate/window/combines/CombineRecordsFunction.java
##
@@ -54,8 +57,10 @@
/** Function used to handle all aggregates. */
private final NamespaceAggsHandleFunction aggregator;
-/** Whether to copy input key, because key is reused. */
-private final boolean requiresCopyKey;
+/** Whether to copy input record, because record is reused. */
+private final boolean requiresCopy;
+
+private final RowDataSerializer recordSerializer;
Review comment:
Add a simple Javadoc to describe it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15082: [FLINK-21592][table-planner-blink] RemoveSingleAggregateRule fails du…
flinkbot commented on pull request #15082: URL: https://github.com/apache/flink/pull/15082#issuecomment-790381856 ## CI report: * 02251f131e8c7d3c398a6a59617b1e7899150fec UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15058: [FLINK-21419][coordination] Remove GC cleaner mechanism for unsafe memory segments
flinkbot edited a comment on pull request #15058: URL: https://github.com/apache/flink/pull/15058#issuecomment-788795717 ## CI report: * 5693fa8be61c954d3662cc8ffe27689e5f6d097e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13991) * b2f53518552258f786ef4bf4031d78f24ba51440 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-790381549 Thanks for your review @dannycranmer! Before we can merge this PR, we need to make sure that CI is passing without credentials (this is the case in the PR CI validation), and with credentials (my personal CI). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15049: [FLINK-21190][runtime-web] Expose exception history
flinkbot edited a comment on pull request #15049: URL: https://github.com/apache/flink/pull/15049#issuecomment-787719691 ## CI report: * 2cbffce55c35f7e163739d07f88e480870a0fc37 UNKNOWN * ecdc2416f67f8cf580a65b30374c1a0bc08b1c0e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14078) * 54de76d97aea0dacc6515abc6c74be4bffea5994 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14099) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21537) SavepointITCase fails on azure
[
https://issues.apache.org/jira/browse/FLINK-21537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295046#comment-17295046
]
Guowei Ma commented on FLINK-21537:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14079=logs=8fd9202e-fd17-5b26-353c-ac1ff76c8f28=a0a633b8-47ef-5c5a-2806-3c13b9e48228
> SavepointITCase fails on azure
> --
>
> Key: FLINK-21537
> URL: https://issues.apache.org/jira/browse/FLINK-21537
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13866=logs=8fd9202e-fd17-5b26-353c-ac1ff76c8f28=a0a633b8-47ef-5c5a-2806-3c13b9e48228
> {code}
> 2021-02-26T23:25:17.5041521Z [ERROR]
> testStopWithSavepointFailingInSnapshotCreation(org.apache.flink.test.checkpointing.SavepointITCase)
> Time elapsed: 0.359 s <<< FAILURE!
> 2021-02-26T23:25:17.5042304Z java.lang.AssertionError
> 2021-02-26T23:25:17.5042938Z at org.junit.Assert.fail(Assert.java:86)
> 2021-02-26T23:25:17.5043637Z at org.junit.Assert.assertTrue(Assert.java:41)
> 2021-02-26T23:25:17.5046831Z at org.junit.Assert.assertTrue(Assert.java:52)
> 2021-02-26T23:25:17.5047567Z at
> org.apache.flink.test.checkpointing.SavepointITCase.lambda$assertInSnapshotCreationFailure$4(SavepointITCase.java:604)
> 2021-02-26T23:25:17.5048194Z at
> org.apache.flink.test.checkpointing.SavepointITCase.testStopWithFailingSourceInOnePipeline(SavepointITCase.java:684)
> 2021-02-26T23:25:17.5049245Z at
> org.apache.flink.test.checkpointing.SavepointITCase.testStopWithSavepointFailingInSnapshotCreation(SavepointITCase.java:564)
> 2021-02-26T23:25:17.5049751Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-02-26T23:25:17.5050168Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-02-26T23:25:17.5051317Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-02-26T23:25:17.5052136Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-02-26T23:25:17.5052931Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-02-26T23:25:17.5053700Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-02-26T23:25:17.5054466Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-02-26T23:25:17.5055163Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-02-26T23:25:17.5055865Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-02-26T23:25:17.5056560Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-02-26T23:25:17.5057240Z at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-02-26T23:25:17.5057906Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-02-26T23:25:17.5058488Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-02-26T23:25:17.5059193Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-02-26T23:25:17.5059935Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-02-26T23:25:17.5060685Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2021-02-26T23:25:17.5061305Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-02-26T23:25:17.5061940Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-02-26T23:25:17.5062717Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2021-02-26T23:25:17.5063355Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2021-02-26T23:25:17.5064011Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2021-02-26T23:25:17.5064648Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2021-02-26T23:25:17.5065227Z at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2021-02-26T23:25:17.5065750Z at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2021-02-26T23:25:17.5066719Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-02-26T23:25:17.5067330Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-02-26T23:25:17.5067988Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2021-02-26T23:25:17.5068659Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2021-02-26T23:25:17.5069424Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>
[GitHub] [flink] flinkbot edited a comment on pull request #13845: [FLINK-19801] Adding virtual channels for rescaling unaligned checkpoints.
flinkbot edited a comment on pull request #13845: URL: https://github.com/apache/flink/pull/13845#issuecomment-718853990 ## CI report: * f63fa7264bda0bafc3e97ac79b920eb34cae3e95 UNKNOWN * 3d418c42a5ffe60a99d17a22a6aaba9955e347f9 UNKNOWN * 6cc4a796b26b66b761f50730f7534c36afad5afa UNKNOWN * dff9f25ac4086acf4b2dbe650a0ed80dd0385ddb UNKNOWN * 6ff570d417423ac84ad5d906900758fbce2b8f43 UNKNOWN * 894e952378dd3ae2c7e92f65b90689ec6c989c8b UNKNOWN * f3aff0c1259abd05d4c18d404874e687431a157c UNKNOWN * e07c51a3c4cf17ad447312889227e33e4b13d4f3 UNKNOWN * 23323a7b983fac19fdd620b0cc82eace73cc587f UNKNOWN * e4bdf60dbf7a9939a9a6ad36f99feb7b4408952a UNKNOWN * d42e17ff93fab9f5e276fe257e687ac254bcd032 UNKNOWN * 8cc2e0b15c68e22dff3805266d15eac38bfea9f0 UNKNOWN * fd2565595a6ca1d8a19678aee6d3050b8ccd UNKNOWN * 8adff526ff72d0a6a08c826293c96fe13e17945d UNKNOWN * 90873ee40bb726e584a2711e076ca26ad171d83f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14084) * 3248e7c8f3c01e1ac5ba69a463d4272b733b10d2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-21598) testDetachedModeSecureWithPreInstallKeytab fail
Guowei Ma created FLINK-21598:
-
Summary: testDetachedModeSecureWithPreInstallKeytab fail
Key: FLINK-21598
URL: https://issues.apache.org/jira/browse/FLINK-21598
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 1.13.0
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14079=logs=8fd975ef-f478-511d-4997-6f15fe8a1fd3=ac0fa443-5d45-5a6b-3597-0310ecc1d2ab
{code:java}
2021-03-03T23:53:32.1060336Z 2021-03-03 23:52:30,669 ERROR
org.apache.flink.runtime.blob.BlobClient [] - Failed to
fetch BLOB
cb48df78410d3d7319d32477aa68c5f5/p-496c3ccd82a70ee27f7f85532569e6871337a1ec-2a5bd98b6e2e355e86797bd2b1c5e1f5
from c08db7158bcf/172.27.0.2:42041 and store it under
/__w/1/s/flink-yarn-tests/target/flink-yarn-tests-fifo-secured/flink-yarn-tests-fifo-secured-localDir-nm-0_0/usercache/hadoop/appcache/application_1614815513529_0001/blobStore-8a708ff7-4c43-4373-aede-0e9f511adddf/incoming/temp-
No retries left.
2021-03-03T23:53:32.1061435Z java.io.IOException: Could not connect to
BlobServer at address c08db7158bcf/172.27.0.2:42041
2021-03-03T23:53:32.1061906Z java.io.IOException: Could not connect to
BlobServer at address c08db7158bcf/172.27.0.2:42041
2021-03-03T23:53:32.1062642Zat
org.apache.flink.runtime.blob.BlobClient.(BlobClient.java:102)
~[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1063501Zat
org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:137)
[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1064585Zat
org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:166)
[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1065487Zat
org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:187)
[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1066542Zat
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.createUserCodeClassLoader(BlobLibraryCacheManager.java:251)
[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1067717Zat
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:228)
[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1068930Zat
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1100(BlobLibraryCacheManager.java:199)
[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1070084Zat
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:333)
[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1071083Zat
org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:991)
[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1071907Zat
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:624)
[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1072818Zat
org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1073289Zat java.lang.Thread.run(Thread.java:748)
[?:1.8.0_282]
2021-03-03T23:53:32.1073662Z Caused by: java.net.ConnectException: Connection
refused (Connection refused)
2021-03-03T23:53:32.1074137Zat
java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282]
2021-03-03T23:53:32.1074650Zat
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
~[?:1.8.0_282]
2021-03-03T23:53:32.1075276Zat
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
~[?:1.8.0_282]
2021-03-03T23:53:32.1075796Zat
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
~[?:1.8.0_282]
2021-03-03T23:53:32.1076271Zat
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282]
2021-03-03T23:53:32.1076663Zat java.net.Socket.connect(Socket.java:607)
~[?:1.8.0_282]
2021-03-03T23:53:32.1077389Zat
org.apache.flink.runtime.blob.BlobClient.(BlobClient.java:96)
~[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
2021-03-03T23:53:32.1077794Z... 11 more
2021-03-03T23:53:32.1079748Z 2021-03-03 23:52:30,680 WARN
org.apache.flink.runtime.taskmanager.Task[] - Source:
Custom File Source (1/1)#0 (07ae013da887de04146706553020c491) switched from
DEPLOYING to FAILED with failure cause: java.io.IOException: Failed to fetch
BLOB
cb48df78410d3d7319d32477aa68c5f5/p-496c3ccd82a70ee27f7f85532569e6871337a1ec-2a5bd98b6e2e355e86797bd2b1c5e1f5
from
[jira] [Updated] (FLINK-21598) testDetachedModeSecureWithPreInstallKeytab fail
[
https://issues.apache.org/jira/browse/FLINK-21598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guowei Ma updated FLINK-21598:
--
Component/s: (was: Runtime / Checkpointing)
Runtime / Coordination
> testDetachedModeSecureWithPreInstallKeytab fail
>
>
> Key: FLINK-21598
> URL: https://issues.apache.org/jira/browse/FLINK-21598
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Guowei Ma
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14079=logs=8fd975ef-f478-511d-4997-6f15fe8a1fd3=ac0fa443-5d45-5a6b-3597-0310ecc1d2ab
> {code:java}
> 2021-03-03T23:53:32.1060336Z 2021-03-03 23:52:30,669 ERROR
> org.apache.flink.runtime.blob.BlobClient [] - Failed to
> fetch BLOB
> cb48df78410d3d7319d32477aa68c5f5/p-496c3ccd82a70ee27f7f85532569e6871337a1ec-2a5bd98b6e2e355e86797bd2b1c5e1f5
> from c08db7158bcf/172.27.0.2:42041 and store it under
> /__w/1/s/flink-yarn-tests/target/flink-yarn-tests-fifo-secured/flink-yarn-tests-fifo-secured-localDir-nm-0_0/usercache/hadoop/appcache/application_1614815513529_0001/blobStore-8a708ff7-4c43-4373-aede-0e9f511adddf/incoming/temp-
> No retries left.
> 2021-03-03T23:53:32.1061435Z java.io.IOException: Could not connect to
> BlobServer at address c08db7158bcf/172.27.0.2:42041
> 2021-03-03T23:53:32.1061906Z java.io.IOException: Could not connect to
> BlobServer at address c08db7158bcf/172.27.0.2:42041
> 2021-03-03T23:53:32.1062642Z at
> org.apache.flink.runtime.blob.BlobClient.(BlobClient.java:102)
> ~[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> 2021-03-03T23:53:32.1063501Z at
> org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:137)
> [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> 2021-03-03T23:53:32.1064585Z at
> org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:166)
> [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> 2021-03-03T23:53:32.1065487Z at
> org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:187)
> [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> 2021-03-03T23:53:32.1066542Z at
> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.createUserCodeClassLoader(BlobLibraryCacheManager.java:251)
> [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> 2021-03-03T23:53:32.1067717Z at
> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:228)
> [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> 2021-03-03T23:53:32.1068930Z at
> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1100(BlobLibraryCacheManager.java:199)
> [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> 2021-03-03T23:53:32.1070084Z at
> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:333)
> [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> 2021-03-03T23:53:32.1071083Z at
> org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:991)
> [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> 2021-03-03T23:53:32.1071907Z at
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:624)
> [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> 2021-03-03T23:53:32.1072818Z at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
> [flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> 2021-03-03T23:53:32.1073289Z at java.lang.Thread.run(Thread.java:748)
> [?:1.8.0_282]
> 2021-03-03T23:53:32.1073662Z Caused by: java.net.ConnectException: Connection
> refused (Connection refused)
> 2021-03-03T23:53:32.1074137Z at
> java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282]
> 2021-03-03T23:53:32.1074650Z at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> ~[?:1.8.0_282]
> 2021-03-03T23:53:32.1075276Z at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> ~[?:1.8.0_282]
> 2021-03-03T23:53:32.1075796Z at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> ~[?:1.8.0_282]
> 2021-03-03T23:53:32.1076271Z at
> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282]
> 2021-03-03T23:53:32.1076663Z at java.net.Socket.connect(Socket.java:607)
> ~[?:1.8.0_282]
> 2021-03-03T23:53:32.1077389Z at
> org.apache.flink.runtime.blob.BlobClient.(BlobClient.java:96)
> ~[flink-dist_2.12-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> 2021-03-03T23:53:32.1077794Z ... 11 more
>
[GitHub] [flink] rmetzger commented on a change in pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on a change in pull request #14737:
URL: https://github.com/apache/flink/pull/14737#discussion_r587207858
##
File path:
flink-formats/flink-avro-glue-schema-registry/src/test/java/org/apache/flink/formats/avro/glue/schema/registry/User.java
##
@@ -0,0 +1,434 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.avro.glue.schema.registry;
+
+import org.apache.avro.message.BinaryMessageDecoder;
+import org.apache.avro.message.BinaryMessageEncoder;
+import org.apache.avro.message.SchemaStore;
+import org.apache.avro.specific.SpecificData;
+
+@SuppressWarnings("all")
+@org.apache.avro.specific.AvroGenerated
+public class User extends org.apache.avro.specific.SpecificRecordBase
Review comment:
I'm okay with addressing this in a follow up PR if you prefer.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on a change in pull request #14737:
URL: https://github.com/apache/flink/pull/14737#discussion_r587207722
##
File path:
flink-formats/flink-avro-glue-schema-registry/src/test/java/org/apache/flink/formats/avro/glue/schema/registry/User.java
##
@@ -0,0 +1,434 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.avro.glue.schema.registry;
+
+import org.apache.avro.message.BinaryMessageDecoder;
+import org.apache.avro.message.BinaryMessageEncoder;
+import org.apache.avro.message.SchemaStore;
+import org.apache.avro.specific.SpecificData;
+
+@SuppressWarnings("all")
+@org.apache.avro.specific.AvroGenerated
+public class User extends org.apache.avro.specific.SpecificRecordBase
Review comment:
But isn't this file generated from the schema defined in
`flink-formats/flink-avro-glue-schema-registry/src/test/java/resources/avro/user.avsc`?
The `avro-maven-plugin` can generated this `User.java` file based on
`user.avsc`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (FLINK-21242) Support state access API for the map/flat_map/filter/reduce operations of Python KeyedStream
[ https://issues.apache.org/jira/browse/FLINK-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zhong reassigned FLINK-21242: - Assignee: Wei Zhong > Support state access API for the map/flat_map/filter/reduce operations of > Python KeyedStream > > > Key: FLINK-21242 > URL: https://issues.apache.org/jira/browse/FLINK-21242 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Wei Zhong >Assignee: Wei Zhong >Priority: Major > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21242) Support state access API for the map/flat_map/filter/reduce operations of Python KeyedStream
[ https://issues.apache.org/jira/browse/FLINK-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zhong updated FLINK-21242: -- Summary: Support state access API for the map/flat_map/filter/reduce operations of Python KeyedStream (was: Support state access API for the map operation of Python KeyedStream) > Support state access API for the map/flat_map/filter/reduce operations of > Python KeyedStream > > > Key: FLINK-21242 > URL: https://issues.apache.org/jira/browse/FLINK-21242 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Wei Zhong >Priority: Major > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #15082: [FLINK-21592][table-planner-blink] RemoveSingleAggregateRule fails du…
flinkbot commented on pull request #15082: URL: https://github.com/apache/flink/pull/15082#issuecomment-790377746 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 02251f131e8c7d3c398a6a59617b1e7899150fec (Thu Mar 04 07:11:01 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-21592).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21592) RemoveSingleAggregateRule fails due to nullability mismatch
[
https://issues.apache.org/jira/browse/FLINK-21592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-21592:
---
Labels: pull-request-available (was: )
> RemoveSingleAggregateRule fails due to nullability mismatch
> ---
>
> Key: FLINK-21592
> URL: https://issues.apache.org/jira/browse/FLINK-21592
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Rui Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> The test case to reproduce the issue:
> {code}
> @Test
> public void test() throws Exception {
> tableEnv.executeSql("create table foo(x int,y int)");
> tableEnv.executeSql("create table bar(i int,s string)");
> System.out.println(tableEnv.explainSql("select (select count(x)-1
> from foo where foo.y=bar.i) from bar"));
> }
> {code}
> Error stack trace is:
> {noformat}
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(BIGINT NOT NULL $f0) NOT NULL
> expression type is RecordType(BIGINT $f0) NOT NULL
> set is
> rel#94:LogicalAggregate.NONE.any.[](input=HepRelVertex#93,group={},agg#0=SINGLE_VALUE($0))
> expression is LogicalProject($f0=[CAST(-($0, 1)):BIGINT])
> LogicalAggregate(group=[{}], agg#0=[COUNT($0)])
> LogicalProject(x=[$0])
> LogicalFilter(condition=[=($1, $cor0.i)])
> LogicalTableScan(table=[[test-catalog, default, foo]])
> at
> org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:381)
> at
> org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:58)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:268)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:283)
> at
> org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1881)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:243)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:346)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:192)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:169)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkDecorrelateProgram.optimize(FlinkDecorrelateProgram.scala:41)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] lirui-apache opened a new pull request #15082: [FLINK-21592][table-planner-blink] RemoveSingleAggregateRule fails du…
lirui-apache opened a new pull request #15082: URL: https://github.com/apache/flink/pull/15082 …e to nullability mismatch ## What is the purpose of the change Fix query plan failure due to RemoveSingleAggregateRule. ## Brief change log - Don't assume nullable value in `RemoveSingleAggregateRule::onMatch` - Add test case ## Verifying this change Added test case ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? NA This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-790375946 I pushed this PR to my personal azure, where the CI credentials are provided: https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8929=results This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-790375151 Looks like the e2e test is failing. Can you write the e2e test in a way that it only executes if the credentials are available? (personal Azure accounts won't have credentials) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21528) Rest Api Support Wildcard
[ https://issues.apache.org/jira/browse/FLINK-21528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias updated FLINK-21528: - Priority: Major (was: Blocker) > Rest Api Support Wildcard > - > > Key: FLINK-21528 > URL: https://issues.apache.org/jira/browse/FLINK-21528 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Affects Versions: 1.11.1 >Reporter: wxmimperio >Priority: Major > Attachments: image-2021-03-01-12-00-36-269.png, > image-2021-03-01-12-05-05-881.png, image-2021-03-01-12-06-15-340.png > > > !image-2021-03-01-12-00-36-269.png! > To obtain detailed metrics information, you need to specify the complete > metrics name. > But these names are automatically generated, only by obtaining a list of all > metrics names, and then filtering by keywords. > For example, the content of the red box is automatically generated without > knowing in advance. > !image-2021-03-01-12-06-15-340.png! > I can only get all the metrics names, and then filter TableSourceScan and > join-time-max. > If I can http://xxx/metrics/get=*join-time-max, I can easily filter what i > want. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21588) unable to create catalogs for postgresql
[ https://issues.apache.org/jira/browse/FLINK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-21588: Fix Version/s: (was: 1.12.0) > unable to create catalogs for postgresql > > > Key: FLINK-21588 > URL: https://issues.apache.org/jira/browse/FLINK-21588 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC >Affects Versions: 1.12.0 > Environment: debian,kubernetes environment >Reporter: Bhagi >Priority: Minor > Attachments: psql catalog error.PNG > > > i want to read the data from postgreSQL database. hence I am trying to > catalogs for postgres as mentioned in Flink document. but getting this error. > i have copied jar files into this path /opt/flink/lib/ > -rw-r- 1 flink flink 1005078 Mar 3 16:53 postgresql-42.2.19.jar > -rw-r- 1 flink flink 196416 Mar 3 16:53 > flink-connector-jdbc_2.11-1.12.0.jar > Please can you suggest how to fix this issue > !psql catalog error.PNG! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21597) testMapAfterRepartitionHasCorrectParallelism2 Fail because of "NoResourceAvailableException"
Guowei Ma created FLINK-21597:
-
Summary: testMapAfterRepartitionHasCorrectParallelism2 Fail
because of "NoResourceAvailableException"
Key: FLINK-21597
URL: https://issues.apache.org/jira/browse/FLINK-21597
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Guowei Ma
{code:java}
2021-03-04T00:17:41.2017402Z [ERROR]
testMapAfterRepartitionHasCorrectParallelism2[Execution mode =
CLUSTER](org.apache.flink.api.scala.operators.PartitionITCase) Time elapsed:
300.117 s <<< ERROR!
2021-03-04T00:17:41.2018058Z
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-03-04T00:17:41.2018525Zat
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2021-03-04T00:17:41.2019563Zat
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2021-03-04T00:17:41.2020129Zat
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
2021-03-04T00:17:41.2021974Zat
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
2021-03-04T00:17:41.2022634Zat
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-03-04T00:17:41.2023118Zat
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-03-04T00:17:41.2023682Zat
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
2021-03-04T00:17:41.2024244Zat
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-03-04T00:17:41.2024749Zat
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-03-04T00:17:41.2025261Zat
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-03-04T00:17:41.2026070Zat
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-03-04T00:17:41.2026814Zat
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1066)
2021-03-04T00:17:41.2027633Zat
akka.dispatch.OnComplete.internal(Future.scala:264)
2021-03-04T00:17:41.2028245Zat
akka.dispatch.OnComplete.internal(Future.scala:261)
2021-03-04T00:17:41.2028796Zat
akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
2021-03-04T00:17:41.2029327Zat
akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
2021-03-04T00:17:41.2030017Zat
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-03-04T00:17:41.2030795Zat
org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
2021-03-04T00:17:41.2031885Zat
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
2021-03-04T00:17:41.2032678Zat
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
2021-03-04T00:17:41.2033428Zat
akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
2021-03-04T00:17:41.2034197Zat
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
2021-03-04T00:17:41.2035094Zat
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
2021-03-04T00:17:41.2035915Zat
scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
2021-03-04T00:17:41.2036617Zat
scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
2021-03-04T00:17:41.2037537Zat
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-03-04T00:17:41.2038019Zat
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
2021-03-04T00:17:41.2038554Zat
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
2021-03-04T00:17:41.2039117Zat
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
2021-03-04T00:17:41.2039671Zat
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
2021-03-04T00:17:41.2040159Zat
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
2021-03-04T00:17:41.2040632Zat
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
2021-03-04T00:17:41.2041086Zat
akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
2021-03-04T00:17:41.2041810Zat
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
2021-03-04T00:17:41.2042514Zat
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
2021-03-04T00:17:41.2042977Zat
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
2021-03-04T00:17:41.2043425Zat
[jira] [Updated] (FLINK-21588) unable to create catalogs for postgresql
[ https://issues.apache.org/jira/browse/FLINK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-21588: Component/s: (was: Deployment / Kubernetes) Connectors / JDBC > unable to create catalogs for postgresql > > > Key: FLINK-21588 > URL: https://issues.apache.org/jira/browse/FLINK-21588 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC >Affects Versions: 1.12.0 > Environment: debian,kubernetes environment >Reporter: Bhagi >Priority: Minor > Fix For: 1.12.0 > > Attachments: psql catalog error.PNG > > > i want to read the data from postgreSQL database. hence I am trying to > catalogs for postgres as mentioned in Flink document. but getting this error. > i have copied jar files into this path /opt/flink/lib/ > -rw-r- 1 flink flink 1005078 Mar 3 16:53 postgresql-42.2.19.jar > -rw-r- 1 flink flink 196416 Mar 3 16:53 > flink-connector-jdbc_2.11-1.12.0.jar > Please can you suggest how to fix this issue > !psql catalog error.PNG! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] rmetzger commented on a change in pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on a change in pull request #14737:
URL: https://github.com/apache/flink/pull/14737#discussion_r587203685
##
File path: flink-end-to-end-tests/flink-glue-schema-registry-test/pom.xml
##
@@ -0,0 +1,259 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+ flink-end-to-end-tests
+ org.apache.flink
+ 1.13-SNAPSHOT
+ ..
+
+ 4.0.0
+
+ flink-glue-schema-registry-test
+ Flink : E2E Tests : AWS Glue Schema Registry
+ jar
+
+
+ 4.5.9
+ 4.4.11
+ 1.11.754
+ 2.15.32
+ 1.0.2
+ 1.6.0
+ 4.1.53.Final
+ 29.0-jre
+
+
+
+
+
+
+
+
+
+
+
+ software.amazon.awssdk
+ http-client-spi
+ ${aws.sdkv2.version}
+
+
+
+ software.amazon.awssdk
+ aws-core
+ ${aws.sdkv2.version}
+
+
+
+ software.amazon.awssdk
+ protocol-core
+ ${aws.sdkv2.version}
+
+
+
+ software.amazon.awssdk
+ annotations
+ ${aws.sdkv2.version}
+
+
+
+ software.amazon.awssdk
+ utils
+ ${aws.sdkv2.version}
+
+
+
+ software.amazon.awssdk
+ apache-client
+ ${aws.sdkv2.version}
+
+
+
+ org.reactivestreams
+ reactive-streams
+ ${reactivestreams.version}
+
+
+
+ org.lz4
+ lz4-java
+ ${lz4.version}
+
+
+
+ software.amazon.awssdk
+ aws-json-protocol
+ ${aws.sdkv2.version}
+
+
+
+ software.amazon.awssdk
+ regions
+ ${aws.sdkv2.version}
+
+
+
+ software.amazon.awssdk
+ sdk-core
+ ${aws.sdkv2.version}
+
+
+
+ io.netty
+ netty-codec-http
+ ${netty.version}
+
+
+
+ software.amazon.awssdk
+ netty-nio-client
+ ${aws.sdkv2.version}
+ runtime
+
+
+
+ software.amazon.awssdk
+ auth
+ ${aws.sdkv2.version}
+
+
+
+ software.amazon.awssdk
+ metrics-spi
+ ${aws.sdkv2.version}
+
+
+
+ io.netty
+ netty-handler
+ ${netty.version}
+
+
+
+ com.google.guava
+ guava
+ ${guava.version}
+
+
+
+ org.apache.httpcomponents
+ httpclient
+ ${httpclient.version}
+
+
+
+ org.apache.httpcomponents
+ httpcore
+ ${httpcore.version}
+
+
+
+
+
+
+ org.apache.flink
[jira] [Commented] (FLINK-21588) unable to create catalogs for postgresql
[ https://issues.apache.org/jira/browse/FLINK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295041#comment-17295041 ] Matthias commented on FLINK-21588: -- Hi Bhagi, thanks for raising this issue. Can you verify that the jars are present in the lib folder of all Flink installations in your cluster? --- A short remark: I might be wrong but it sounds like a configuration issue. Issues like this are easier to be discussed in the [Flink mailing list|https://flink.apache.org/community.html#mailing-lists] and, therefore, are more likely to be answered there. > unable to create catalogs for postgresql > > > Key: FLINK-21588 > URL: https://issues.apache.org/jira/browse/FLINK-21588 > Project: Flink > Issue Type: Bug > Components: Deployment / Kubernetes >Affects Versions: 1.12.0 > Environment: debian,kubernetes environment >Reporter: Bhagi >Priority: Minor > Fix For: 1.12.0 > > Attachments: psql catalog error.PNG > > > i want to read the data from postgreSQL database. hence I am trying to > catalogs for postgres as mentioned in Flink document. but getting this error. > i have copied jar files into this path /opt/flink/lib/ > -rw-r- 1 flink flink 1005078 Mar 3 16:53 postgresql-42.2.19.jar > -rw-r- 1 flink flink 196416 Mar 3 16:53 > flink-connector-jdbc_2.11-1.12.0.jar > Please can you suggest how to fix this issue > !psql catalog error.PNG! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21596) CheckpointFailureManagerITCase.testAsyncCheckpointFailureTriggerJobFailed fail
[
https://issues.apache.org/jira/browse/FLINK-21596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guowei Ma updated FLINK-21596:
--
Labels: test-stability (was: )
> CheckpointFailureManagerITCase.testAsyncCheckpointFailureTriggerJobFailed
> fail
> ---
>
> Key: FLINK-21596
> URL: https://issues.apache.org/jira/browse/FLINK-21596
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.13.0
>Reporter: Guowei Ma
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14079=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=c2734c79-73b6-521c-e85a-67c7ecae9107
> {code:java}
> [ERROR]
> testAsyncCheckpointFailureTriggerJobFailed(org.apache.flink.test.checkpointing.CheckpointFailureManagerITCase)
> Time elapsed: 38.623 s <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 1
> milliseconds
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.test.util.TestUtils.submitJobAndWaitForResult(TestUtils.java:62)
> at
> org.apache.flink.test.checkpointing.CheckpointFailureManagerITCase.testAsyncCheckpointFailureTriggerJobFailed(CheckpointFailureManagerITCase.java:103)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21596) CheckpointFailureManagerITCase.testAsyncCheckpointFailureTriggerJobFailed fail
Guowei Ma created FLINK-21596:
-
Summary:
CheckpointFailureManagerITCase.testAsyncCheckpointFailureTriggerJobFailed fail
Key: FLINK-21596
URL: https://issues.apache.org/jira/browse/FLINK-21596
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 1.13.0
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14079=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=c2734c79-73b6-521c-e85a-67c7ecae9107
{code:java}
[ERROR]
testAsyncCheckpointFailureTriggerJobFailed(org.apache.flink.test.checkpointing.CheckpointFailureManagerITCase)
Time elapsed: 38.623 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 1
milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
at
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at
org.apache.flink.test.util.TestUtils.submitJobAndWaitForResult(TestUtils.java:62)
at
org.apache.flink.test.checkpointing.CheckpointFailureManagerITCase.testAsyncCheckpointFailureTriggerJobFailed(CheckpointFailureManagerITCase.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21564) CommonTestUtils.waitUntilCondition could fail with condition meets before
[
https://issues.apache.org/jira/browse/FLINK-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295040#comment-17295040
]
Kezhu Wang commented on FLINK-21564:
[~nicholasjiang] Not about simplification, but guarantee to caller that there
will be no exception once condition meet.
> CommonTestUtils.waitUntilCondition could fail with condition meets before
> -
>
> Key: FLINK-21564
> URL: https://issues.apache.org/jira/browse/FLINK-21564
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Reporter: Kezhu Wang
>Priority: Minor
>
> {code}
> public static void waitUntilCondition(
> SupplierWithException condition,
> Deadline timeout,
> long retryIntervalMillis,
> String errorMsg)
> throws Exception {
> while (timeout.hasTimeLeft() && !condition.get()) {
> final long timeLeft = Math.max(0, timeout.timeLeft().toMillis());
> Thread.sleep(Math.min(retryIntervalMillis, timeLeft));
> }
> if (!timeout.hasTimeLeft()) {
> throw new TimeoutException(errorMsg);
> }
> }
> {code}
> The timeout could run off between truth condition and last checking.
> Besides this, I also see time-out blocking condition in some tests, the
> combination could be worse.
> Not a big issue, but worth to be aware of and solved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21595) DispatcherResourceManagerComponent.deregisterApplication times out
[
https://issues.apache.org/jira/browse/FLINK-21595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guowei Ma updated FLINK-21595:
--
Summary: DispatcherResourceManagerComponent.deregisterApplication times out
(was: DispatcherResourceManagerComponent.deregisterApplication time-out)
> DispatcherResourceManagerComponent.deregisterApplication times out
> --
>
> Key: FLINK-21595
> URL: https://issues.apache.org/jira/browse/FLINK-21595
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Guowei Ma
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14079=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56
> {code:java}
> 2021-03-04T02:10:45.5546368Z org.apache.flink.util.FlinkException: Could not
> close resource.
> 2021-03-04T02:10:45.5546954Z at
> org.apache.flink.util.AutoCloseableAsync.close(AutoCloseableAsync.java:39)
> 2021-03-04T02:10:45.5547602Z at
> org.apache.flink.test.runtime.JobGraphRunningUtil.execute(JobGraphRunningUtil.java:62)
> 2021-03-04T02:10:45.5548315Z at
> org.apache.flink.test.runtime.BlockingShuffleITCase.testSortMergeBlockingShuffle(BlockingShuffleITCase.java:71)
> 2021-03-04T02:10:45.5548961Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-03-04T02:10:45.5549524Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-03-04T02:10:45.5550327Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-03-04T02:10:45.5551069Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-03-04T02:10:45.5551661Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-03-04T02:10:45.5552551Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-03-04T02:10:45.5553208Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-03-04T02:10:45.5553859Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-03-04T02:10:45.5554460Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-03-04T02:10:45.061Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-03-04T02:10:45.707Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2021-03-04T02:10:45.5556416Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-03-04T02:10:45.5556910Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-03-04T02:10:45.5557416Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2021-03-04T02:10:45.5557919Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2021-03-04T02:10:45.5558422Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2021-03-04T02:10:45.5558908Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2021-03-04T02:10:45.5559432Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2021-03-04T02:10:45.5560256Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2021-03-04T02:10:45.5560921Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2021-03-04T02:10:45.5561577Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 2021-03-04T02:10:45.5562260Z at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> 2021-03-04T02:10:45.5563068Z at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> 2021-03-04T02:10:45.5563720Z at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> 2021-03-04T02:10:45.5564341Z at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> 2021-03-04T02:10:45.5565320Z Caused by:
> java.util.concurrent.TimeoutException: Invocation of public abstract
> java.util.concurrent.CompletableFuture
> org.apache.flink.runtime.resourcemanager.ResourceManagerGateway.deregisterApplication(org.apache.flink.runtime.clusterframework.ApplicationStatus,java.lang.String)
> timed out.
> 2021-03-04T02:10:45.5566276Z at
> org.apache.flink.runtime.rpc.akka.$Proxy25.deregisterApplication(Unknown
> Source)
> 2021-03-04T02:10:45.5567034Z at
> org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent.deregisterApplication(DispatcherResourceManagerComponent.java:143)
>
[jira] [Created] (FLINK-21595) DispatcherResourceManagerComponent.deregisterApplication time-out
Guowei Ma created FLINK-21595:
-
Summary: DispatcherResourceManagerComponent.deregisterApplication
time-out
Key: FLINK-21595
URL: https://issues.apache.org/jira/browse/FLINK-21595
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14079=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56
{code:java}
2021-03-04T02:10:45.5546368Z org.apache.flink.util.FlinkException: Could not
close resource.
2021-03-04T02:10:45.5546954Zat
org.apache.flink.util.AutoCloseableAsync.close(AutoCloseableAsync.java:39)
2021-03-04T02:10:45.5547602Zat
org.apache.flink.test.runtime.JobGraphRunningUtil.execute(JobGraphRunningUtil.java:62)
2021-03-04T02:10:45.5548315Zat
org.apache.flink.test.runtime.BlockingShuffleITCase.testSortMergeBlockingShuffle(BlockingShuffleITCase.java:71)
2021-03-04T02:10:45.5548961Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-03-04T02:10:45.5549524Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-03-04T02:10:45.5550327Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-03-04T02:10:45.5551069Zat
java.lang.reflect.Method.invoke(Method.java:498)
2021-03-04T02:10:45.5551661Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-03-04T02:10:45.5552551Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-03-04T02:10:45.5553208Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-03-04T02:10:45.5553859Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-03-04T02:10:45.5554460Zat
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2021-03-04T02:10:45.061Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2021-03-04T02:10:45.707Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2021-03-04T02:10:45.5556416Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-03-04T02:10:45.5556910Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-03-04T02:10:45.5557416Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-03-04T02:10:45.5557919Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-03-04T02:10:45.5558422Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-03-04T02:10:45.5558908Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-03-04T02:10:45.5559432Zat
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
2021-03-04T02:10:45.5560256Zat
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
2021-03-04T02:10:45.5560921Zat
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
2021-03-04T02:10:45.5561577Zat
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
2021-03-04T02:10:45.5562260Zat
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2021-03-04T02:10:45.5563068Zat
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
2021-03-04T02:10:45.5563720Zat
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
2021-03-04T02:10:45.5564341Zat
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
2021-03-04T02:10:45.5565320Z Caused by: java.util.concurrent.TimeoutException:
Invocation of public abstract java.util.concurrent.CompletableFuture
org.apache.flink.runtime.resourcemanager.ResourceManagerGateway.deregisterApplication(org.apache.flink.runtime.clusterframework.ApplicationStatus,java.lang.String)
timed out.
2021-03-04T02:10:45.5566276Zat
org.apache.flink.runtime.rpc.akka.$Proxy25.deregisterApplication(Unknown Source)
2021-03-04T02:10:45.5567034Zat
org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent.deregisterApplication(DispatcherResourceManagerComponent.java:143)
2021-03-04T02:10:45.5568020Zat
org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent.lambda$deregisterApplicationAndClose$1(DispatcherResourceManagerComponent.java:129)
2021-03-04T02:10:45.5568915Zat
org.apache.flink.runtime.concurrent.FutureUtils.lambda$composeAfterwards$19(FutureUtils.java:704)
2021-03-04T02:10:45.5569586Zat
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-03-04T02:10:45.5570259Zat
[jira] [Closed] (FLINK-21576) Remove ExecutionVertex#getPreferredLocations
[
https://issues.apache.org/jira/browse/FLINK-21576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu closed FLINK-21576.
---
Resolution: Done
Done via 2233cb2ebad5908464e36ac890f0ab54bd57d35a
> Remove ExecutionVertex#getPreferredLocations
>
>
> Key: FLINK-21576
> URL: https://issues.apache.org/jira/browse/FLINK-21576
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Zhu Zhu
>Assignee: Zhu Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> {{ExecutionVertex#getPreferredLocations()}} is superseded by
> {{DefaultPreferredLocationsRetriever}} and is no longer used. Hence, we can
> remove it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21587) FineGrainedSlotManagerTest.testNotificationAboutNotEnoughResources is unstable
[
https://issues.apache.org/jira/browse/FLINK-21587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295035#comment-17295035
]
Guowei Ma commented on FLINK-21587:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14079=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=7c61167f-30b3-5893-cc38-a9e3d057e392
> FineGrainedSlotManagerTest.testNotificationAboutNotEnoughResources is unstable
> --
>
> Key: FLINK-21587
> URL: https://issues.apache.org/jira/browse/FLINK-21587
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Robert Metzger
>Assignee: Yangze Guo
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.13.0
>
>
> Happened in my WIP branch, but most likely unrelated to my change:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8925=logs=9dc1b5dc-bcfa-5f83-eaa7-0cb181ddc267=ab910030-93db-52a7-74a3-34a0addb481b
> Also note that the error is reproducable locally with DEBUG log level, but
> not with INFO:
> {code}
> [ERROR] Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 2.169 s <<< FAILURE! - in
> org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest
> [ERROR]
> testNotificationAboutNotEnoughResources(org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest)
> Time elapsed: 0.029 s <<< FAILURE!
> java.lang.AssertionError:
> Expected: a collection with size <1>
> but: collection size was <0>
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
> at org.junit.Assert.assertThat(Assert.java:923)
> at
> org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest$9.lambda$new$5(FineGrainedSlotManagerTest.java:548)
> at
> org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTestBase$Context.runTest(FineGrainedSlotManagerTestBase.java:197)
> at
> org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest$9.(FineGrainedSlotManagerTest.java:521)
> at
> org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest.testNotificationAboutNotEnoughResources(FineGrainedSlotManagerTest.java:507)
> at
> org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest.testNotificationAboutNotEnoughResources(FineGrainedSlotManagerTest.java:493)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21588) unable to create catalogs for postgresql
[ https://issues.apache.org/jira/browse/FLINK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias updated FLINK-21588: - Priority: Minor (was: Blocker) > unable to create catalogs for postgresql > > > Key: FLINK-21588 > URL: https://issues.apache.org/jira/browse/FLINK-21588 > Project: Flink > Issue Type: Bug > Components: Deployment / Kubernetes >Affects Versions: 1.12.0 > Environment: debian,kubernetes environment >Reporter: Bhagi >Priority: Minor > Fix For: 1.12.0 > > Attachments: psql catalog error.PNG > > > i want to read the data from postgreSQL database. hence I am trying to > catalogs for postgres as mentioned in Flink document. but getting this error. > i have copied jar files into this path /opt/flink/lib/ > -rw-r- 1 flink flink 1005078 Mar 3 16:53 postgresql-42.2.19.jar > -rw-r- 1 flink flink 196416 Mar 3 16:53 > flink-connector-jdbc_2.11-1.12.0.jar > Please can you suggest how to fix this issue > !psql catalog error.PNG! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-21564) CommonTestUtils.waitUntilCondition could fail with condition meets before
[
https://issues.apache.org/jira/browse/FLINK-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295033#comment-17295033
]
Nicholas Jiang edited comment on FLINK-21564 at 3/4/21, 6:46 AM:
-
[~kezhuw], according to the logic of the codes you provide, you just simplify
the check logic for whether has time left, right? And this simplify makes sense
to me.
cc [~chesnay]
was (Author: nicholasjiang):
[~kezhuw], according to the logic of the codes you provide, you just simplify
the check logic for whether has time left, right? And this simplify makes sense
to me.
> CommonTestUtils.waitUntilCondition could fail with condition meets before
> -
>
> Key: FLINK-21564
> URL: https://issues.apache.org/jira/browse/FLINK-21564
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Reporter: Kezhu Wang
>Priority: Minor
>
> {code}
> public static void waitUntilCondition(
> SupplierWithException condition,
> Deadline timeout,
> long retryIntervalMillis,
> String errorMsg)
> throws Exception {
> while (timeout.hasTimeLeft() && !condition.get()) {
> final long timeLeft = Math.max(0, timeout.timeLeft().toMillis());
> Thread.sleep(Math.min(retryIntervalMillis, timeLeft));
> }
> if (!timeout.hasTimeLeft()) {
> throw new TimeoutException(errorMsg);
> }
> }
> {code}
> The timeout could run off between truth condition and last checking.
> Besides this, I also see time-out blocking condition in some tests, the
> combination could be worse.
> Not a big issue, but worth to be aware of and solved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-21515) SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource is failing
[
https://issues.apache.org/jira/browse/FLINK-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295034#comment-17295034
]
Guowei Ma edited comment on FLINK-21515 at 3/4/21, 6:46 AM:
another case from 1.11 branch。
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14080=logs=f0ac5c25-1168-55a5-07ff-0e88223afed9=39a61cac-5c62-532f-d2c1-dea450a66708
{code:java}
2021-03-03T22:43:01.0479311Z "main" #1 prio=5 os_prio=0 cpu=1038548.77ms
elapsed=1039.83s tid=0x7f89fc028000 nid=0x61bc runnable
[0x7f8a057a9000]
2021-03-03T22:43:01.0479743Zjava.lang.Thread.State: RUNNABLE
2021-03-03T22:43:01.0480269Zat
org.apache.flink.streaming.runtime.tasks.StreamTaskMailboxTestHarness.waitForTaskCompletion(StreamTaskMailboxTestHarness.java:150)
2021-03-03T22:43:01.0481230Zat
org.apache.flink.streaming.runtime.tasks.SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource(SourceStreamTaskTest.java:606)
2021-03-03T22:43:01.0481813Zat
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.9.1/Native
Method)
2021-03-03T22:43:01.0482346Zat
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.9.1/NativeMethodAccessorImpl.java:62)
2021-03-03T22:43:01.0483002Zat
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.9.1/DelegatingMethodAccessorImpl.java:43)
2021-03-03T22:43:01.0483465Zat
java.lang.reflect.Method.invoke(java.base@11.0.9.1/Method.java:566)
2021-03-03T22:43:01.0483900Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-03-03T22:43:01.0484928Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-03-03T22:43:01.0485657Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-03-03T22:43:01.0486360Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-03-03T22:43:01.0487063Zat
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2021-03-03T22:43:01.0487959Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2021-03-03T22:43:01.0488707Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2021-03-03T22:43:01.0489318Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-03-03T22:43:01.0490038Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-03-03T22:43:01.0490658Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-03-03T22:43:01.0491332Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-03-03T22:43:01.0491964Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-03-03T22:43:01.0492594Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-03-03T22:43:01.0493165Zat
org.junit.runners.Suite.runChild(Suite.java:128)
2021-03-03T22:43:01.0493650Zat
org.junit.runners.Suite.runChild(Suite.java:27)
2021-03-03T22:43:01.0494201Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-03-03T22:43:01.0495008Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-03-03T22:43:01.0495651Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-03-03T22:43:01.0496332Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-03-03T22:43:01.0496971Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-03-03T22:43:01.0497558Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-03-03T22:43:01.0498770Zat
org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
2021-03-03T22:43:01.0499616Zat
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
2021-03-03T22:43:01.0500443Zat
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
2021-03-03T22:43:01.0501256Zat
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
2021-03-03T22:43:01.0501832Zat
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
2021-03-03T22:43:01.0502400Zat
org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
2021-03-03T22:43:01.0502928Zat
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2021-03-03T22:43:01.0503513Zat
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
2021-03-03T22:43:01.0504069Zat
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
2021-03-03T22:43:01.0504833Zat
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{code}
was (Author: maguowei):
[jira] [Comment Edited] (FLINK-21515) SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource is failing
[
https://issues.apache.org/jira/browse/FLINK-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295034#comment-17295034
]
Guowei Ma edited comment on FLINK-21515 at 3/4/21, 6:46 AM:
another case from 1.11
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14080=logs=f0ac5c25-1168-55a5-07ff-0e88223afed9=39a61cac-5c62-532f-d2c1-dea450a66708
{code:java}
2021-03-03T22:43:01.0479311Z "main" #1 prio=5 os_prio=0 cpu=1038548.77ms
elapsed=1039.83s tid=0x7f89fc028000 nid=0x61bc runnable
[0x7f8a057a9000]
2021-03-03T22:43:01.0479743Zjava.lang.Thread.State: RUNNABLE
2021-03-03T22:43:01.0480269Zat
org.apache.flink.streaming.runtime.tasks.StreamTaskMailboxTestHarness.waitForTaskCompletion(StreamTaskMailboxTestHarness.java:150)
2021-03-03T22:43:01.0481230Zat
org.apache.flink.streaming.runtime.tasks.SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource(SourceStreamTaskTest.java:606)
2021-03-03T22:43:01.0481813Zat
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.9.1/Native
Method)
2021-03-03T22:43:01.0482346Zat
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.9.1/NativeMethodAccessorImpl.java:62)
2021-03-03T22:43:01.0483002Zat
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.9.1/DelegatingMethodAccessorImpl.java:43)
2021-03-03T22:43:01.0483465Zat
java.lang.reflect.Method.invoke(java.base@11.0.9.1/Method.java:566)
2021-03-03T22:43:01.0483900Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-03-03T22:43:01.0484928Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-03-03T22:43:01.0485657Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-03-03T22:43:01.0486360Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-03-03T22:43:01.0487063Zat
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2021-03-03T22:43:01.0487959Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2021-03-03T22:43:01.0488707Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2021-03-03T22:43:01.0489318Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-03-03T22:43:01.0490038Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-03-03T22:43:01.0490658Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-03-03T22:43:01.0491332Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-03-03T22:43:01.0491964Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-03-03T22:43:01.0492594Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-03-03T22:43:01.0493165Zat
org.junit.runners.Suite.runChild(Suite.java:128)
2021-03-03T22:43:01.0493650Zat
org.junit.runners.Suite.runChild(Suite.java:27)
2021-03-03T22:43:01.0494201Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-03-03T22:43:01.0495008Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-03-03T22:43:01.0495651Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-03-03T22:43:01.0496332Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-03-03T22:43:01.0496971Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-03-03T22:43:01.0497558Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-03-03T22:43:01.0498770Zat
org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
2021-03-03T22:43:01.0499616Zat
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
2021-03-03T22:43:01.0500443Zat
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
2021-03-03T22:43:01.0501256Zat
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
2021-03-03T22:43:01.0501832Zat
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
2021-03-03T22:43:01.0502400Zat
org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
2021-03-03T22:43:01.0502928Zat
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2021-03-03T22:43:01.0503513Zat
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
2021-03-03T22:43:01.0504069Zat
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
2021-03-03T22:43:01.0504833Zat
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{code}
was (Author: maguowei):
[jira] [Closed] (FLINK-21515) SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource is failing
[
https://issues.apache.org/jira/browse/FLINK-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guowei Ma closed FLINK-21515.
-
Resolution: Fixed
> SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource is
> failing
>
>
> Key: FLINK-21515
> URL: https://issues.apache.org/jira/browse/FLINK-21515
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.12.2, 1.13.0
>Reporter: Matthias
>Assignee: Roman Khachatryan
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.13.0, 1.12.3
>
>
> We experience a test instability with
> {{SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource}}.
> The test is occassionally timing out.
> See [this
> build|https://dev.azure.com/mapohl/flink/_build/results?buildId=290=logs=cc649950-03e9-5fae-8326-2f1ad744b536=51cab6ca-669f-5dc0-221d-1e4f7dc4fc85=7846]
> being related to FLINK-21030.
> {noformat}
> "main" #1 prio=5 os_prio=0 tid=0x7f72fc00b800 nid=0x2133 runnable
> [0x7f73046ed000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskMailboxTestHarness.waitForTaskCompletion(StreamTaskMailboxTestHarness.java:147)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource(SourceStreamTaskTest.java:604)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
> This failure was reproducible on {{master}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21554) Remove the magical classpath setting stuff in docker-entrypoint.sh
[
https://issues.apache.org/jira/browse/FLINK-21554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-21554:
---
Labels: pull-request-available (was: )
> Remove the magical classpath setting stuff in docker-entrypoint.sh
> --
>
> Key: FLINK-21554
> URL: https://issues.apache.org/jira/browse/FLINK-21554
> Project: Flink
> Issue Type: Improvement
> Components: flink-docker
>Affects Versions: 1.13.0
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> After FLINK-21128 is done, we could remove the magical classpath setting
> stuff in {{docker-entrypoint.sh}}.
>
> {code:java}
> # Set the Flink related environmentsexport _FLINK_HOME_DETERMINED=true.
> $FLINK_HOME/bin/config.shexport
> FLINK_CLASSPATH="`constructFlinkClassPath`:$INTERNAL_HADOOP_CLASSPATHS"
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-docker] wangyang0918 opened a new pull request #69: [FLINK-21554] Remove the magical classpath setting stuff in docker-entrypoint.sh
wangyang0918 opened a new pull request #69: URL: https://github.com/apache/flink-docker/pull/69 After FLINK-21128 is done, we could remove the magical classpath setting stuff in docker-entrypoint.sh. ``` # Set the Flink related environments export _FLINK_HOME_DETERMINED=true . $FLINK_HOME/bin/config.sh export FLINK_CLASSPATH="`constructFlinkClassPath`:$INTERNAL_HADOOP_CLASSPATHS" ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Reopened] (FLINK-21515) SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource is failing
[
https://issues.apache.org/jira/browse/FLINK-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guowei Ma reopened FLINK-21515:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14080=logs=f0ac5c25-1168-55a5-07ff-0e88223afed9=39a61cac-5c62-532f-d2c1-dea450a66708
{code:java}
2021-03-03T22:43:01.0479311Z "main" #1 prio=5 os_prio=0 cpu=1038548.77ms
elapsed=1039.83s tid=0x7f89fc028000 nid=0x61bc runnable
[0x7f8a057a9000]
2021-03-03T22:43:01.0479743Zjava.lang.Thread.State: RUNNABLE
2021-03-03T22:43:01.0480269Zat
org.apache.flink.streaming.runtime.tasks.StreamTaskMailboxTestHarness.waitForTaskCompletion(StreamTaskMailboxTestHarness.java:150)
2021-03-03T22:43:01.0481230Zat
org.apache.flink.streaming.runtime.tasks.SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource(SourceStreamTaskTest.java:606)
2021-03-03T22:43:01.0481813Zat
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.9.1/Native
Method)
2021-03-03T22:43:01.0482346Zat
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.9.1/NativeMethodAccessorImpl.java:62)
2021-03-03T22:43:01.0483002Zat
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.9.1/DelegatingMethodAccessorImpl.java:43)
2021-03-03T22:43:01.0483465Zat
java.lang.reflect.Method.invoke(java.base@11.0.9.1/Method.java:566)
2021-03-03T22:43:01.0483900Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-03-03T22:43:01.0484928Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-03-03T22:43:01.0485657Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-03-03T22:43:01.0486360Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-03-03T22:43:01.0487063Zat
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2021-03-03T22:43:01.0487959Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2021-03-03T22:43:01.0488707Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2021-03-03T22:43:01.0489318Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-03-03T22:43:01.0490038Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-03-03T22:43:01.0490658Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-03-03T22:43:01.0491332Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-03-03T22:43:01.0491964Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-03-03T22:43:01.0492594Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-03-03T22:43:01.0493165Zat
org.junit.runners.Suite.runChild(Suite.java:128)
2021-03-03T22:43:01.0493650Zat
org.junit.runners.Suite.runChild(Suite.java:27)
2021-03-03T22:43:01.0494201Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-03-03T22:43:01.0495008Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-03-03T22:43:01.0495651Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-03-03T22:43:01.0496332Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-03-03T22:43:01.0496971Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-03-03T22:43:01.0497558Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-03-03T22:43:01.0498770Zat
org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
2021-03-03T22:43:01.0499616Zat
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
2021-03-03T22:43:01.0500443Zat
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
2021-03-03T22:43:01.0501256Zat
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
2021-03-03T22:43:01.0501832Zat
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
2021-03-03T22:43:01.0502400Zat
org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
2021-03-03T22:43:01.0502928Zat
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2021-03-03T22:43:01.0503513Zat
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
2021-03-03T22:43:01.0504069Zat
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
2021-03-03T22:43:01.0504833Zat
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{code}
> SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource is
> failing
>
[jira] [Comment Edited] (FLINK-21564) CommonTestUtils.waitUntilCondition could fail with condition meets before
[
https://issues.apache.org/jira/browse/FLINK-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295033#comment-17295033
]
Nicholas Jiang edited comment on FLINK-21564 at 3/4/21, 6:43 AM:
-
[~kezhuw], according to the logic of the codes you provide, you just simplify
the check logic for whether has time left, right? And this simplify makes sense
to me.
was (Author: nicholasjiang):
[~kezhuw], according to the logic of the codes you provide, you just simplify
the check logic for whether has time left, right?
> CommonTestUtils.waitUntilCondition could fail with condition meets before
> -
>
> Key: FLINK-21564
> URL: https://issues.apache.org/jira/browse/FLINK-21564
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Reporter: Kezhu Wang
>Priority: Minor
>
> {code}
> public static void waitUntilCondition(
> SupplierWithException condition,
> Deadline timeout,
> long retryIntervalMillis,
> String errorMsg)
> throws Exception {
> while (timeout.hasTimeLeft() && !condition.get()) {
> final long timeLeft = Math.max(0, timeout.timeLeft().toMillis());
> Thread.sleep(Math.min(retryIntervalMillis, timeLeft));
> }
> if (!timeout.hasTimeLeft()) {
> throw new TimeoutException(errorMsg);
> }
> }
> {code}
> The timeout could run off between truth condition and last checking.
> Besides this, I also see time-out blocking condition in some tests, the
> combination could be worse.
> Not a big issue, but worth to be aware of and solved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21564) CommonTestUtils.waitUntilCondition could fail with condition meets before
[
https://issues.apache.org/jira/browse/FLINK-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295033#comment-17295033
]
Nicholas Jiang commented on FLINK-21564:
[~kezhuw], according to the logic of the codes you provide, you just simplify
the check logic for whether has time left, right?
> CommonTestUtils.waitUntilCondition could fail with condition meets before
> -
>
> Key: FLINK-21564
> URL: https://issues.apache.org/jira/browse/FLINK-21564
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Reporter: Kezhu Wang
>Priority: Minor
>
> {code}
> public static void waitUntilCondition(
> SupplierWithException condition,
> Deadline timeout,
> long retryIntervalMillis,
> String errorMsg)
> throws Exception {
> while (timeout.hasTimeLeft() && !condition.get()) {
> final long timeLeft = Math.max(0, timeout.timeLeft().toMillis());
> Thread.sleep(Math.min(retryIntervalMillis, timeLeft));
> }
> if (!timeout.hasTimeLeft()) {
> throw new TimeoutException(errorMsg);
> }
> }
> {code}
> The timeout could run off between truth condition and last checking.
> Besides this, I also see time-out blocking condition in some tests, the
> combination could be worse.
> Not a big issue, but worth to be aware of and solved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21594) Flink - {"errors":["Service temporarily unavailable due to an ongoing leader election. Please refresh."]}
Bhagi created FLINK-21594:
-
Summary: Flink - {"errors":["Service temporarily unavailable due
to an ongoing leader election. Please refresh."]}
Key: FLINK-21594
URL: https://issues.apache.org/jira/browse/FLINK-21594
Project: Flink
Issue Type: Bug
Components: Deployment / Kubernetes
Affects Versions: 1.12.0
Environment: debian kubernetes
Reporter: Bhagi
Fix For: 1.12.0
Attachments: image-2021-03-04-11-59-24-095.png,
image-2021-03-04-12-06-12-504.png
Hi Team,
I deployed standalone Flink session cluster on kubernetes.. i have created the
HA with 3 job managers. But Flink Web UI is throwing an error saying
"\{"errors":["Service temporarily unavailable due to an ongoing leader
election. Please refresh."]}".
Please help to fix the issue with UI and HA.
Logs are showing with the Leader election and updated to the all configmaps..
below configmaps are created in the same namespace where flink cluster is
running.
111-dispatcher-leader
111-resourcemanager-leader
111-restserver-leader
but i have one doubt on configmap created, when i described above configmaps
showing type as flink-native-kubernetes.. But i created cluster with flink
standalone kubernetes.
configmap-type: high-availability
type: flink-native-kubernetes
2) Let me know how to confirm the whether flink is running on native kubernetes
or standalone kubernetes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21593) RocksDBListStatePerformanceTest.testRocksDbListStateAPI fail because of timeout
[
https://issues.apache.org/jira/browse/FLINK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guowei Ma updated FLINK-21593:
--
Description:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14080=logs=3b6ec2fd-a816-5e75-c775-06fb87cb6670=b33fdd4f-3de5-542e-2624-5d53167bb672
{code:java}
[ERROR]
testRocksDbListStateAPIs(org.apache.flink.contrib.streaming.state.benchmark.RocksDBListStatePerformanceTest)
Time elapsed: 8.245 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 2000
milliseconds
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:231)
at
org.apache.flink.contrib.streaming.state.benchmark.RocksDBListStatePerformanceTest.testRocksDbListStateAPIs(RocksDBListStatePerformanceTest.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}
was:
{code:java}
[ERROR]
testRocksDbListStateAPIs(org.apache.flink.contrib.streaming.state.benchmark.RocksDBListStatePerformanceTest)
Time elapsed: 8.245 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 2000
milliseconds
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:231)
at
org.apache.flink.contrib.streaming.state.benchmark.RocksDBListStatePerformanceTest.testRocksDbListStateAPIs(RocksDBListStatePerformanceTest.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}
> RocksDBListStatePerformanceTest.testRocksDbListStateAPI fail because of
> timeout
> ---
>
> Key: FLINK-21593
> URL: https://issues.apache.org/jira/browse/FLINK-21593
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.11.3
>Reporter: Guowei Ma
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14080=logs=3b6ec2fd-a816-5e75-c775-06fb87cb6670=b33fdd4f-3de5-542e-2624-5d53167bb672
> {code:java}
> [ERROR]
> testRocksDbListStateAPIs(org.apache.flink.contrib.streaming.state.benchmark.RocksDBListStatePerformanceTest)
> Time elapsed: 8.245 s <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 2000
> milliseconds
> at org.rocksdb.RocksDB.open(Native Method)
> at org.rocksdb.RocksDB.open(RocksDB.java:231)
> at
> org.apache.flink.contrib.streaming.state.benchmark.RocksDBListStatePerformanceTest.testRocksDbListStateAPIs(RocksDBListStatePerformanceTest.java:96)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at
[GitHub] [flink] xiaoHoly commented on a change in pull request #14531: [FLINK-20777][Connector][Kafka] Property "partition.discovery.interval.ms" shoule be enabled by default for unbounded mode, and
xiaoHoly commented on a change in pull request #14531:
URL: https://github.com/apache/flink/pull/14531#discussion_r587193000
##
File path:
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java
##
@@ -423,14 +430,15 @@ private void parseAndSetRequiredProperties() {
startingOffsetsInitializer.getAutoOffsetResetStrategy().name().toLowerCase(),
true);
-// If the source is bounded, do not run periodic partition discovery.
-if (maybeOverride(
-KafkaSourceOptions.PARTITION_DISCOVERY_INTERVAL_MS.key(),
-"-1",
-boundedness == Boundedness.BOUNDED)) {
-LOG.warn(
-"{} property is overridden to -1 because the source is
bounded.",
-KafkaSourceOptions.PARTITION_DISCOVERY_INTERVAL_MS);
+// If the source is bounded or stoppingOffsetsInitializer is
specified, do not run periodic
Review comment:
I think these should be exposed to users in the official documents
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-21593) RocksDBListStatePerformanceTest.testRocksDbListStateAPI fail because of timeout
Guowei Ma created FLINK-21593:
-
Summary: RocksDBListStatePerformanceTest.testRocksDbListStateAPI
fail because of timeout
Key: FLINK-21593
URL: https://issues.apache.org/jira/browse/FLINK-21593
Project: Flink
Issue Type: Bug
Components: Runtime / State Backends
Affects Versions: 1.11.3
Reporter: Guowei Ma
{code:java}
[ERROR]
testRocksDbListStateAPIs(org.apache.flink.contrib.streaming.state.benchmark.RocksDBListStatePerformanceTest)
Time elapsed: 8.245 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 2000
milliseconds
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:231)
at
org.apache.flink.contrib.streaming.state.benchmark.RocksDBListStatePerformanceTest.testRocksDbListStateAPIs(RocksDBListStatePerformanceTest.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21559) Python DataStreamTests::test_process_function failed on AZP
[
https://issues.apache.org/jira/browse/FLINK-21559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295030#comment-17295030
]
Huang Xingbo commented on FLINK-21559:
--
The expected result of this test depends on the order of element and watermark.
I think it is more stable to test related code with UT instead of the current
IT, which can better control the order of element and watermark.
> Python DataStreamTests::test_process_function failed on AZP
> ---
>
> Key: FLINK-21559
> URL: https://issues.apache.org/jira/browse/FLINK-21559
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
>Affects Versions: 1.13.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.0
>
>
> The Python test case {{DataStreamTests::test_process_function}} failed on AZP.
> {code}
> === short test summary info
>
> FAILED
> pyflink/datastream/tests/test_data_stream.py::DataStreamTests::test_process_function
> = 1 failed, 705 passed, 22 skipped, 303 warnings in 583.39s (0:09:43)
> ==
> ERROR: InvocationError for command /__w/3/s/flink-python/.tox/py38/bin/pytest
> --durations=20 (exited with code 1)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13992=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21103) E2e tests time out on azure
[
https://issues.apache.org/jira/browse/FLINK-21103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295028#comment-17295028
]
Guowei Ma commented on FLINK-21103:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14063=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> E2e tests time out on azure
> ---
>
> Key: FLINK-21103
> URL: https://issues.apache.org/jira/browse/FLINK-21103
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.1, 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12377=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> Creating worker2 ... done
> Jan 22 13:16:17 Waiting for hadoop cluster to come up. We have been trying
> for 0 seconds, retrying ...
> Jan 22 13:16:22 Waiting for hadoop cluster to come up. We have been trying
> for 5 seconds, retrying ...
> Jan 22 13:16:27 Waiting for hadoop cluster to come up. We have been trying
> for 10 seconds, retrying ...
> Jan 22 13:16:32 Waiting for hadoop cluster to come up. We have been trying
> for 15 seconds, retrying ...
> Jan 22 13:16:37 Waiting for hadoop cluster to come up. We have been trying
> for 20 seconds, retrying ...
> Jan 22 13:16:43 Waiting for hadoop cluster to come up. We have been trying
> for 26 seconds, retrying ...
> Jan 22 13:16:48 Waiting for hadoop cluster to come up. We have been trying
> for 31 seconds, retrying ...
> Jan 22 13:16:53 Waiting for hadoop cluster to come up. We have been trying
> for 36 seconds, retrying ...
> Jan 22 13:16:58 Waiting for hadoop cluster to come up. We have been trying
> for 41 seconds, retrying ...
> Jan 22 13:17:03 Waiting for hadoop cluster to come up. We have been trying
> for 46 seconds, retrying ...
> Jan 22 13:17:08 We only have 0 NodeManagers up. We have been trying for 0
> seconds, retrying ...
> 21/01/22 13:17:10 INFO client.RMProxy: Connecting to ResourceManager at
> master.docker-hadoop-cluster-network/172.19.0.3:8032
> 21/01/22 13:17:11 INFO client.AHSProxy: Connecting to Application History
> server at master.docker-hadoop-cluster-network/172.19.0.3:10200
> Jan 22 13:17:11 We now have 2 NodeManagers up.
> ==
> === WARNING: This E2E Run took already 80% of the allocated time budget of
> 250 minutes ===
> ==
> ==
> === WARNING: This E2E Run will time out in the next few minutes. Starting to
> upload the log output ===
> ==
> ##[error]The task has timed out.
> Async Command Start: Upload Artifact
> Uploading 1 files
> File upload succeed.
> Upload '/tmp/_e2e_watchdog.output.0' to file container:
> '#/11824779/e2e-timeout-logs'
> Associated artifact 140921 with build 12377
> Async Command End: Upload Artifact
> Async Command Start: Upload Artifact
> Uploading 1 files
> File upload succeed.
> Upload '/tmp/_e2e_watchdog.output.1' to file container:
> '#/11824779/e2e-timeout-logs'
> Associated artifact 140921 with build 12377
> Async Command End: Upload Artifact
> Async Command Start: Upload Artifact
> Uploading 1 files
> File upload succeed.
> Upload '/tmp/_e2e_watchdog.output.2' to file container:
> '#/11824779/e2e-timeout-logs'
> Associated artifact 140921 with build 12377
> Async Command End: Upload Artifact
> Finishing: Run e2e tests
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20329) Elasticsearch7DynamicSinkITCase hangs
[
https://issues.apache.org/jira/browse/FLINK-20329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295027#comment-17295027
]
Guowei Ma commented on FLINK-20329:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14048=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20
> Elasticsearch7DynamicSinkITCase hangs
> -
>
> Key: FLINK-20329
> URL: https://issues.apache.org/jira/browse/FLINK-20329
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ElasticSearch
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10052=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20
> {code}
> 2020-11-24T16:04:05.9260517Z [INFO] Running
> org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch7DynamicSinkITCase
> 2020-11-24T16:19:25.5481231Z
> ==
> 2020-11-24T16:19:25.5483549Z Process produced no output for 900 seconds.
> 2020-11-24T16:19:25.5484064Z
> ==
> 2020-11-24T16:19:25.5484498Z
> ==
> 2020-11-24T16:19:25.5484882Z The following Java processes are running (JPS)
> 2020-11-24T16:19:25.5485475Z
> ==
> 2020-11-24T16:19:25.5694497Z Picked up JAVA_TOOL_OPTIONS:
> -XX:+HeapDumpOnOutOfMemoryError
> 2020-11-24T16:19:25.7263048Z 16192 surefirebooter5057948964630155904.jar
> 2020-11-24T16:19:25.7263515Z 18566 Jps
> 2020-11-24T16:19:25.7263709Z 959 Launcher
> 2020-11-24T16:19:25.7411148Z
> ==
> 2020-11-24T16:19:25.7427013Z Printing stack trace of Java process 16192
> 2020-11-24T16:19:25.7427369Z
> ==
> 2020-11-24T16:19:25.7484365Z Picked up JAVA_TOOL_OPTIONS:
> -XX:+HeapDumpOnOutOfMemoryError
> 2020-11-24T16:19:26.0848776Z 2020-11-24 16:19:26
> 2020-11-24T16:19:26.0849578Z Full thread dump OpenJDK 64-Bit Server VM
> (25.275-b01 mixed mode):
> 2020-11-24T16:19:26.0849831Z
> 2020-11-24T16:19:26.0850185Z "Attach Listener" #32 daemon prio=9 os_prio=0
> tid=0x7fc148001000 nid=0x48e7 waiting on condition [0x]
> 2020-11-24T16:19:26.0850595Zjava.lang.Thread.State: RUNNABLE
> 2020-11-24T16:19:26.0850814Z
> 2020-11-24T16:19:26.0851375Z "testcontainers-ryuk" #31 daemon prio=5
> os_prio=0 tid=0x7fc251232000 nid=0x3fb0 in Object.wait()
> [0x7fc1012c4000]
> 2020-11-24T16:19:26.0854688Zjava.lang.Thread.State: TIMED_WAITING (on
> object monitor)
> 2020-11-24T16:19:26.0855379Z at java.lang.Object.wait(Native Method)
> 2020-11-24T16:19:26.0855844Z at
> org.testcontainers.utility.ResourceReaper.lambda$null$1(ResourceReaper.java:142)
> 2020-11-24T16:19:26.0857272Z - locked <0x8e2bd2d0> (a
> java.util.ArrayList)
> 2020-11-24T16:19:26.0857977Z at
> org.testcontainers.utility.ResourceReaper$$Lambda$93/1981729428.run(Unknown
> Source)
> 2020-11-24T16:19:26.0858471Z at
> org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
> 2020-11-24T16:19:26.0858961Z at
> org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:133)
> 2020-11-24T16:19:26.0859422Z at
> org.testcontainers.utility.ResourceReaper$$Lambda$92/40191541.run(Unknown
> Source)
> 2020-11-24T16:19:26.0859788Z at java.lang.Thread.run(Thread.java:748)
> 2020-11-24T16:19:26.0860030Z
> 2020-11-24T16:19:26.0860371Z "process reaper" #24 daemon prio=10 os_prio=0
> tid=0x7fc0f803b800 nid=0x3f92 waiting on condition [0x7fc10296e000]
> 2020-11-24T16:19:26.0860913Zjava.lang.Thread.State: TIMED_WAITING
> (parking)
> 2020-11-24T16:19:26.0861387Z at sun.misc.Unsafe.park(Native Method)
> 2020-11-24T16:19:26.0862495Z - parking to wait for <0x8814bf30> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> 2020-11-24T16:19:26.0863253Z at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-11-24T16:19:26.0863760Z at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> 2020-11-24T16:19:26.0864274Z at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> 2020-11-24T16:19:26.0864762Z at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> 2020-11-24T16:19:26.0865299Z at
>
[jira] [Commented] (FLINK-21592) RemoveSingleAggregateRule fails due to nullability mismatch
[
https://issues.apache.org/jira/browse/FLINK-21592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295024#comment-17295024
]
Rui Li commented on FLINK-21592:
I think the root cause is because {{RemoveSingleAggregateRule::onMatch}}
assumes {{SqlSingleValueAggFunction}}
returns a
[nullable|https://github.com/apache/flink/blob/release-1.12.2/flink-table/flink-table-planner-blink/src/main/java/org/apache/calcite/sql2rel/RelDecorrelator.java#L1872]
value, which is wrong because {{SqlSingleValueAggFunction}}'s return type
inference is {{ReturnTypes.ARG0}} and doesn't guarantee nullability.
> RemoveSingleAggregateRule fails due to nullability mismatch
> ---
>
> Key: FLINK-21592
> URL: https://issues.apache.org/jira/browse/FLINK-21592
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Rui Li
>Priority: Major
> Fix For: 1.13.0
>
>
> The test case to reproduce the issue:
> {code}
> @Test
> public void test() throws Exception {
> tableEnv.executeSql("create table foo(x int,y int)");
> tableEnv.executeSql("create table bar(i int,s string)");
> System.out.println(tableEnv.explainSql("select (select count(x)-1
> from foo where foo.y=bar.i) from bar"));
> }
> {code}
> Error stack trace is:
> {noformat}
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(BIGINT NOT NULL $f0) NOT NULL
> expression type is RecordType(BIGINT $f0) NOT NULL
> set is
> rel#94:LogicalAggregate.NONE.any.[](input=HepRelVertex#93,group={},agg#0=SINGLE_VALUE($0))
> expression is LogicalProject($f0=[CAST(-($0, 1)):BIGINT])
> LogicalAggregate(group=[{}], agg#0=[COUNT($0)])
> LogicalProject(x=[$0])
> LogicalFilter(condition=[=($1, $cor0.i)])
> LogicalTableScan(table=[[test-catalog, default, foo]])
> at
> org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:381)
> at
> org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:58)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:268)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:283)
> at
> org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1881)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:243)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:346)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:192)
> at
> org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:169)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkDecorrelateProgram.optimize(FlinkDecorrelateProgram.scala:41)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21115) Add AggregatingState and corresponding StateDescriptor for Python DataStream API
[ https://issues.apache.org/jira/browse/FLINK-21115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295023#comment-17295023 ] Wei Zhong commented on FLINK-21115: --- Merged via 044f8e17237d0fdbff96ca973817c932a3c985ba > Add AggregatingState and corresponding StateDescriptor for Python DataStream > API > > > Key: FLINK-21115 > URL: https://issues.apache.org/jira/browse/FLINK-21115 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Wei Zhong >Assignee: Wei Zhong >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21559) Python DataStreamTests::test_process_function failed on AZP
[
https://issues.apache.org/jira/browse/FLINK-21559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295025#comment-17295025
]
Shuiqiang Chen commented on FLINK-21559:
Thank you for reporting the issue, I am looking at it.
> Python DataStreamTests::test_process_function failed on AZP
> ---
>
> Key: FLINK-21559
> URL: https://issues.apache.org/jira/browse/FLINK-21559
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
>Affects Versions: 1.13.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.0
>
>
> The Python test case {{DataStreamTests::test_process_function}} failed on AZP.
> {code}
> === short test summary info
>
> FAILED
> pyflink/datastream/tests/test_data_stream.py::DataStreamTests::test_process_function
> = 1 failed, 705 passed, 22 skipped, 303 warnings in 583.39s (0:09:43)
> ==
> ERROR: InvocationError for command /__w/3/s/flink-python/.tox/py38/bin/pytest
> --durations=20 (exited with code 1)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13992=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-21115) Add AggregatingState and corresponding StateDescriptor for Python DataStream API
[ https://issues.apache.org/jira/browse/FLINK-21115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zhong closed FLINK-21115. - Resolution: Fixed > Add AggregatingState and corresponding StateDescriptor for Python DataStream > API > > > Key: FLINK-21115 > URL: https://issues.apache.org/jira/browse/FLINK-21115 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Wei Zhong >Assignee: Wei Zhong >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-21434) When UDAF return ROW type, and the number of fields is more than 14, the crash happend
[
https://issues.apache.org/jira/browse/FLINK-21434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zhong closed FLINK-21434.
-
Resolution: Fixed
> When UDAF return ROW type, and the number of fields is more than 14, the
> crash happend
> --
>
> Key: FLINK-21434
> URL: https://issues.apache.org/jira/browse/FLINK-21434
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.12.1
> Environment: python 3.7.5
> pyflink 1.12.1
>Reporter: awayne
>Assignee: Wei Zhong
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.13.0, 1.12.3
>
>
> Code(a simple udaf to return a Row containing 15 fields):
> {code:python}
> from pyflink.common import Row
> from pyflink.table.udf import AggregateFunction, udaf
> from pyflink.table import DataTypes, EnvironmentSettings,
> StreamTableEnvironment
> class Test(AggregateFunction):
> def create_accumulator(self):
> return Row(0, 0)
> def get_value(self, accumulator):
> return Row(1.23, 1.23, 1.23, 1.23, 1.23, 1.23, 1.23,
>1.23, 1.23, 1.23, 1.23, 1.23, 1.23, 1.23, 1.23)
> def accumulate(self, accumulator, a, b):
> pass
> def get_result_type(self):
> return DataTypes.ROW([
> DataTypes.FIELD("f1", DataTypes.FLOAT()),
> DataTypes.FIELD("f2", DataTypes.FLOAT()),
> DataTypes.FIELD("f3", DataTypes.FLOAT()),
> DataTypes.FIELD("f4", DataTypes.FLOAT()),
> DataTypes.FIELD("f5", DataTypes.FLOAT()),
> DataTypes.FIELD("f6", DataTypes.FLOAT()),
> DataTypes.FIELD("f7", DataTypes.FLOAT()),
> DataTypes.FIELD("f8", DataTypes.FLOAT()),
> DataTypes.FIELD("f9", DataTypes.FLOAT()),
> DataTypes.FIELD("f10", DataTypes.FLOAT()),
> DataTypes.FIELD("f11", DataTypes.FLOAT()),
> DataTypes.FIELD("f12", DataTypes.FLOAT()),
> DataTypes.FIELD("f13", DataTypes.FLOAT()),
> DataTypes.FIELD("f14", DataTypes.FLOAT()),
> DataTypes.FIELD("f15", DataTypes.FLOAT())
> ])
> def get_accumulator_type(self):
> return DataTypes.ROW([
> DataTypes.FIELD("f1", DataTypes.BIGINT()),
> DataTypes.FIELD("f2", DataTypes.BIGINT())])
> def udaf_test():
> env_settings =
> EnvironmentSettings.new_instance().in_streaming_mode().use_blink_planner().build()
> table_env = StreamTableEnvironment.create(environment_settings=env_settings)
> test = udaf(Test())
> table_env.execute_sql("""
> CREATE TABLE print_sink (
> `name` STRING,
> `agg` ROW f5 FLOAT, f6 FLOAT, f7 FLOAT, f8 FLOAT,
> f9 FLOAT, f10 FLOAT, f11 FLOAT, f12 FLOAT,
> f13 FLOAT, f14 FLOAT, f15 FLOAT>
> ) WITH (
> 'connector' = 'print'
> )
> """)
> table = table_env.from_elements([(1, 2, "Lee")], ['value', 'count', 'name'])
> result_table = table.group_by(table.name)\
> .select(table.name, test(table.value, table.count))
> result_table.execute_insert("print_sink").wait()
> if __name__ == "__main__":
> udaf_test()
> {code}
> Exception:
> {code:java}
> Caused by: java.io.EOFException
> at java.base/java.io.DataInputStream.readInt(DataInputStream.java:397)
> at java.base/java.io.DataInputStream.readFloat(DataInputStream.java:451)
> at
> org.apache.flink.api.common.typeutils.base.FloatSerializer.deserialize(FloatSerializer.java:72)
> at
> org.apache.flink.api.common.typeutils.base.FloatSerializer.deserialize(FloatSerializer.java:30)
> at
> org.apache.flink.table.runtime.typeutils.serializers.python.RowDataSerializer.deserialize(RowDataSerializer.java:106)
> at
> org.apache.flink.table.runtime.typeutils.serializers.python.RowDataSerializer.deserialize(RowDataSerializer.java:49)
> at
> org.apache.flink.table.runtime.typeutils.serializers.python.RowDataSerializer.deserialize(RowDataSerializer.java:106
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21434) When UDAF return ROW type, and the number of fields is more than 14, the crash happend
[
https://issues.apache.org/jira/browse/FLINK-21434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295022#comment-17295022
]
Wei Zhong commented on FLINK-21434:
---
Fixed via:
- master: 1b0e592cf56005bff430252b0d59b8b6da1347f8
- release-1.12: 591cd3cf5e65d6726178652107c6751c5705141a
> When UDAF return ROW type, and the number of fields is more than 14, the
> crash happend
> --
>
> Key: FLINK-21434
> URL: https://issues.apache.org/jira/browse/FLINK-21434
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.12.1
> Environment: python 3.7.5
> pyflink 1.12.1
>Reporter: awayne
>Assignee: Wei Zhong
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.13.0, 1.12.3
>
>
> Code(a simple udaf to return a Row containing 15 fields):
> {code:python}
> from pyflink.common import Row
> from pyflink.table.udf import AggregateFunction, udaf
> from pyflink.table import DataTypes, EnvironmentSettings,
> StreamTableEnvironment
> class Test(AggregateFunction):
> def create_accumulator(self):
> return Row(0, 0)
> def get_value(self, accumulator):
> return Row(1.23, 1.23, 1.23, 1.23, 1.23, 1.23, 1.23,
>1.23, 1.23, 1.23, 1.23, 1.23, 1.23, 1.23, 1.23)
> def accumulate(self, accumulator, a, b):
> pass
> def get_result_type(self):
> return DataTypes.ROW([
> DataTypes.FIELD("f1", DataTypes.FLOAT()),
> DataTypes.FIELD("f2", DataTypes.FLOAT()),
> DataTypes.FIELD("f3", DataTypes.FLOAT()),
> DataTypes.FIELD("f4", DataTypes.FLOAT()),
> DataTypes.FIELD("f5", DataTypes.FLOAT()),
> DataTypes.FIELD("f6", DataTypes.FLOAT()),
> DataTypes.FIELD("f7", DataTypes.FLOAT()),
> DataTypes.FIELD("f8", DataTypes.FLOAT()),
> DataTypes.FIELD("f9", DataTypes.FLOAT()),
> DataTypes.FIELD("f10", DataTypes.FLOAT()),
> DataTypes.FIELD("f11", DataTypes.FLOAT()),
> DataTypes.FIELD("f12", DataTypes.FLOAT()),
> DataTypes.FIELD("f13", DataTypes.FLOAT()),
> DataTypes.FIELD("f14", DataTypes.FLOAT()),
> DataTypes.FIELD("f15", DataTypes.FLOAT())
> ])
> def get_accumulator_type(self):
> return DataTypes.ROW([
> DataTypes.FIELD("f1", DataTypes.BIGINT()),
> DataTypes.FIELD("f2", DataTypes.BIGINT())])
> def udaf_test():
> env_settings =
> EnvironmentSettings.new_instance().in_streaming_mode().use_blink_planner().build()
> table_env = StreamTableEnvironment.create(environment_settings=env_settings)
> test = udaf(Test())
> table_env.execute_sql("""
> CREATE TABLE print_sink (
> `name` STRING,
> `agg` ROW f5 FLOAT, f6 FLOAT, f7 FLOAT, f8 FLOAT,
> f9 FLOAT, f10 FLOAT, f11 FLOAT, f12 FLOAT,
> f13 FLOAT, f14 FLOAT, f15 FLOAT>
> ) WITH (
> 'connector' = 'print'
> )
> """)
> table = table_env.from_elements([(1, 2, "Lee")], ['value', 'count', 'name'])
> result_table = table.group_by(table.name)\
> .select(table.name, test(table.value, table.count))
> result_table.execute_insert("print_sink").wait()
> if __name__ == "__main__":
> udaf_test()
> {code}
> Exception:
> {code:java}
> Caused by: java.io.EOFException
> at java.base/java.io.DataInputStream.readInt(DataInputStream.java:397)
> at java.base/java.io.DataInputStream.readFloat(DataInputStream.java:451)
> at
> org.apache.flink.api.common.typeutils.base.FloatSerializer.deserialize(FloatSerializer.java:72)
> at
> org.apache.flink.api.common.typeutils.base.FloatSerializer.deserialize(FloatSerializer.java:30)
> at
> org.apache.flink.table.runtime.typeutils.serializers.python.RowDataSerializer.deserialize(RowDataSerializer.java:106)
> at
> org.apache.flink.table.runtime.typeutils.serializers.python.RowDataSerializer.deserialize(RowDataSerializer.java:49)
> at
> org.apache.flink.table.runtime.typeutils.serializers.python.RowDataSerializer.deserialize(RowDataSerializer.java:106
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15080: [FLINK-21587][coordination] Harden FineGrainedSlotManagerTest and AbstractFineGrainedSlotManagerITCase
flinkbot edited a comment on pull request #15080: URL: https://github.com/apache/flink/pull/15080#issuecomment-790241029 ## CI report: * ac7a34047e2ffe43d2b6c42f233ac6e908e0ba84 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14088) * 3be3e2f724caeb9ef4fdd1310a788976110f1a44 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14092) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15049: [FLINK-21190][runtime-web] Expose exception history
flinkbot edited a comment on pull request #15049: URL: https://github.com/apache/flink/pull/15049#issuecomment-787719691 ## CI report: * 2cbffce55c35f7e163739d07f88e480870a0fc37 UNKNOWN * ecdc2416f67f8cf580a65b30374c1a0bc08b1c0e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14078) * 54de76d97aea0dacc6515abc6c74be4bffea5994 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] becketqin commented on a change in pull request #14531: [FLINK-20777][Connector][Kafka] Property "partition.discovery.interval.ms" shoule be enabled by default for unbounded mode, and
becketqin commented on a change in pull request #14531:
URL: https://github.com/apache/flink/pull/14531#discussion_r587183656
##
File path:
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java
##
@@ -423,14 +430,15 @@ private void parseAndSetRequiredProperties() {
startingOffsetsInitializer.getAutoOffsetResetStrategy().name().toLowerCase(),
true);
-// If the source is bounded, do not run periodic partition discovery.
-if (maybeOverride(
-KafkaSourceOptions.PARTITION_DISCOVERY_INTERVAL_MS.key(),
-"-1",
-boundedness == Boundedness.BOUNDED)) {
-LOG.warn(
-"{} property is overridden to -1 because the source is
bounded.",
-KafkaSourceOptions.PARTITION_DISCOVERY_INTERVAL_MS);
+// If the source is bounded or stoppingOffsetsInitializer is
specified, do not run periodic
Review comment:
Should we update the comments here as well? It probably should be
something like:
"We only disable the periodic partition discovery in the following cases:
1. The source is bounded.
2. The source has stopping offsets and there is no explicit partition
discovery interval setting."
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-21592) RemoveSingleAggregateRule fails due to nullability mismatch
[
https://issues.apache.org/jira/browse/FLINK-21592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated FLINK-21592:
---
Description:
The test case to reproduce the issue:
{code}
@Test
public void test() throws Exception {
tableEnv.executeSql("create table foo(x int,y int)");
tableEnv.executeSql("create table bar(i int,s string)");
System.out.println(tableEnv.explainSql("select (select count(x)-1 from
foo where foo.y=bar.i) from bar"));
}
{code}
Error stack trace is:
{noformat}
java.lang.AssertionError: Cannot add expression of different type to set:
set type is RecordType(BIGINT NOT NULL $f0) NOT NULL
expression type is RecordType(BIGINT $f0) NOT NULL
set is
rel#94:LogicalAggregate.NONE.any.[](input=HepRelVertex#93,group={},agg#0=SINGLE_VALUE($0))
expression is LogicalProject($f0=[CAST(-($0, 1)):BIGINT])
LogicalAggregate(group=[{}], agg#0=[COUNT($0)])
LogicalProject(x=[$0])
LogicalFilter(condition=[=($1, $cor0.i)])
LogicalTableScan(table=[[test-catalog, default, foo]])
at
org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:381)
at
org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:58)
at
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:268)
at
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:283)
at
org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1881)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
at
org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
at
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:243)
at
org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
at
org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
at
org.apache.calcite.sql2rel.RelDecorrelator.removeCorrelationViaRule(RelDecorrelator.java:346)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:192)
at
org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:169)
at
org.apache.flink.table.planner.plan.optimize.program.FlinkDecorrelateProgram.optimize(FlinkDecorrelateProgram.scala:41)
{noformat}
was:
The test case to reproduce the issue:
{code}
@Test
public void test() throws Exception {
tableEnv.executeSql("create table foo(x int,y int)");
tableEnv.executeSql("create table bar(i int,s string)");
System.out.println(tableEnv.explainSql("select (select count(x)-1 from
foo where foo.y=bar.i) from bar"));
}
{code}
> RemoveSingleAggregateRule fails due to nullability mismatch
> ---
>
> Key: FLINK-21592
> URL: https://issues.apache.org/jira/browse/FLINK-21592
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Rui Li
>Priority: Major
> Fix For: 1.13.0
>
>
> The test case to reproduce the issue:
> {code}
> @Test
> public void test() throws Exception {
> tableEnv.executeSql("create table foo(x int,y int)");
> tableEnv.executeSql("create table bar(i int,s string)");
> System.out.println(tableEnv.explainSql("select (select count(x)-1
> from foo where foo.y=bar.i) from bar"));
> }
> {code}
> Error stack trace is:
> {noformat}
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(BIGINT NOT NULL $f0) NOT NULL
> expression type is RecordType(BIGINT $f0) NOT NULL
> set is
> rel#94:LogicalAggregate.NONE.any.[](input=HepRelVertex#93,group={},agg#0=SINGLE_VALUE($0))
> expression is LogicalProject($f0=[CAST(-($0, 1)):BIGINT])
> LogicalAggregate(group=[{}], agg#0=[COUNT($0)])
> LogicalProject(x=[$0])
> LogicalFilter(condition=[=($1, $cor0.i)])
> LogicalTableScan(table=[[test-catalog, default, foo]])
> at
> org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:381)
> at
> org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:58)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:268)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:283)
> at
> org.apache.calcite.sql2rel.RelDecorrelator$RemoveSingleAggregateRule.onMatch(RelDecorrelator.java:1881)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at
[jira] [Created] (FLINK-21592) RemoveSingleAggregateRule fails due to nullability mismatch
Rui Li created FLINK-21592:
--
Summary: RemoveSingleAggregateRule fails due to nullability
mismatch
Key: FLINK-21592
URL: https://issues.apache.org/jira/browse/FLINK-21592
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Reporter: Rui Li
Fix For: 1.13.0
The test case to reproduce the issue:
{code}
@Test
public void test() throws Exception {
tableEnv.executeSql("create table foo(x int,y int)");
tableEnv.executeSql("create table bar(i int,s string)");
System.out.println(tableEnv.explainSql("select (select count(x)-1 from
foo where foo.y=bar.i) from bar"));
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21103) E2e tests time out on azure
[
https://issues.apache.org/jira/browse/FLINK-21103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-21103:
--
Component/s: (was: Runtime / State Backends)
Thanks for the follow up [~dwysakowicz] and [~rmetzger]!
I'm removing `State Backends` from the JIRA component field since there seems
to be no more clue on state backend related issues. Please feel free to add it
back if any new findings or I'm missing anything. Thanks.
> E2e tests time out on azure
> ---
>
> Key: FLINK-21103
> URL: https://issues.apache.org/jira/browse/FLINK-21103
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.1, 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12377=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> Creating worker2 ... done
> Jan 22 13:16:17 Waiting for hadoop cluster to come up. We have been trying
> for 0 seconds, retrying ...
> Jan 22 13:16:22 Waiting for hadoop cluster to come up. We have been trying
> for 5 seconds, retrying ...
> Jan 22 13:16:27 Waiting for hadoop cluster to come up. We have been trying
> for 10 seconds, retrying ...
> Jan 22 13:16:32 Waiting for hadoop cluster to come up. We have been trying
> for 15 seconds, retrying ...
> Jan 22 13:16:37 Waiting for hadoop cluster to come up. We have been trying
> for 20 seconds, retrying ...
> Jan 22 13:16:43 Waiting for hadoop cluster to come up. We have been trying
> for 26 seconds, retrying ...
> Jan 22 13:16:48 Waiting for hadoop cluster to come up. We have been trying
> for 31 seconds, retrying ...
> Jan 22 13:16:53 Waiting for hadoop cluster to come up. We have been trying
> for 36 seconds, retrying ...
> Jan 22 13:16:58 Waiting for hadoop cluster to come up. We have been trying
> for 41 seconds, retrying ...
> Jan 22 13:17:03 Waiting for hadoop cluster to come up. We have been trying
> for 46 seconds, retrying ...
> Jan 22 13:17:08 We only have 0 NodeManagers up. We have been trying for 0
> seconds, retrying ...
> 21/01/22 13:17:10 INFO client.RMProxy: Connecting to ResourceManager at
> master.docker-hadoop-cluster-network/172.19.0.3:8032
> 21/01/22 13:17:11 INFO client.AHSProxy: Connecting to Application History
> server at master.docker-hadoop-cluster-network/172.19.0.3:10200
> Jan 22 13:17:11 We now have 2 NodeManagers up.
> ==
> === WARNING: This E2E Run took already 80% of the allocated time budget of
> 250 minutes ===
> ==
> ==
> === WARNING: This E2E Run will time out in the next few minutes. Starting to
> upload the log output ===
> ==
> ##[error]The task has timed out.
> Async Command Start: Upload Artifact
> Uploading 1 files
> File upload succeed.
> Upload '/tmp/_e2e_watchdog.output.0' to file container:
> '#/11824779/e2e-timeout-logs'
> Associated artifact 140921 with build 12377
> Async Command End: Upload Artifact
> Async Command Start: Upload Artifact
> Uploading 1 files
> File upload succeed.
> Upload '/tmp/_e2e_watchdog.output.1' to file container:
> '#/11824779/e2e-timeout-logs'
> Associated artifact 140921 with build 12377
> Async Command End: Upload Artifact
> Async Command Start: Upload Artifact
> Uploading 1 files
> File upload succeed.
> Upload '/tmp/_e2e_watchdog.output.2' to file container:
> '#/11824779/e2e-timeout-logs'
> Associated artifact 140921 with build 12377
> Async Command End: Upload Artifact
> Finishing: Run e2e tests
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-21552) The managed memory was not released if exception was thrown in createPythonExecutionEnvironment
[
https://issues.apache.org/jira/browse/FLINK-21552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song closed FLINK-21552.
Resolution: Fixed
Fixed via
* master (1.13): 1e71b1c35c25063ce80abd35d1ac59e437c858c3
* release-1.12: 64647490f3e96bdbdfe535654c667f1ead0b026c
> The managed memory was not released if exception was thrown in
> createPythonExecutionEnvironment
> ---
>
> Key: FLINK-21552
> URL: https://issues.apache.org/jira/browse/FLINK-21552
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Runtime / Coordination
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Xintong Song
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0, 1.12.3
>
>
> If there is exception thrown in
> [createPythonExecutionEnvironment|https://github.com/apache/flink/blob/3796e59f79a90bd8ad5e6fc37458e2d6cce23139/flink-python/src/main/java/org/apache/flink/streaming/api/runners/python/beam/BeamPythonFunctionRunner.java#L248],
> the job will failed with the following exception:
> {code:java}
> org.apache.flink.runtime.memory.MemoryAllocationException: Could not created
> the shared memory resource of size 611948962. Not enough memory left to
> reserve from the slot's managed memory.
> at
> org.apache.flink.runtime.memory.MemoryManager.lambda$getSharedMemoryResourceForManagedMemory$5(MemoryManager.java:536)
> at
> org.apache.flink.runtime.memory.SharedResources.createResource(SharedResources.java:126)
> at
> org.apache.flink.runtime.memory.SharedResources.getOrAllocateSharedResource(SharedResources.java:72)
> at
> org.apache.flink.runtime.memory.MemoryManager.getSharedMemoryResourceForManagedMemory(MemoryManager.java:555)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.open(BeamPythonFunctionRunner.java:250)
> at
> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.open(AbstractPythonFunctionOperator.java:113)
> at
> org.apache.flink.table.runtime.operators.python.AbstractStatelessFunctionOperator.open(AbstractStatelessFunctionOperator.java:116)
> at
> org.apache.flink.table.runtime.operators.python.scalar.AbstractPythonScalarFunctionOperator.open(AbstractPythonScalarFunctionOperator.java:88)
> at
> org.apache.flink.table.runtime.operators.python.scalar.AbstractRowDataPythonScalarFunctionOperator.open(AbstractRowDataPythonScalarFunctionOperator.java:70)
> at
> org.apache.flink.table.runtime.operators.python.scalar.RowDataPythonScalarFunctionOperator.open(RowDataPythonScalarFunctionOperator.java:59)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:428)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:543)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:93)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:533)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:573)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
> at java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.flink.runtime.memory.MemoryReservationException: Could
> not allocate 611948962 bytes, only 0 bytes are remaining. This usually
> indicates that you are requesting more memory than you have reserved.
> However, when running an old JVM version it can also be caused by slow
> garbage collection. Try to upgrade to Java 8u72 or higher if running on an
> old Java version.
> at
> org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:170)
> at
> org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:84)
> at
> org.apache.flink.runtime.memory.MemoryManager.reserveMemory(MemoryManager.java:423)
> at
> org.apache.flink.runtime.memory.MemoryManager.lambda$getSharedMemoryResourceForManagedMemory$5(MemoryManager.java:534)
> ... 17 more
> {code}
> The reason is that the reserved managed memory was not added back to the
> MemoryManager when Job failed because of exceptions thrown in
> createPythonExecutionEnvironment. This causes that there is no managed memory
> to allocate during failover.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14841: [FLINK-21232][YARN][Common] Introduce pluggable Hadoop delegation token providers
flinkbot edited a comment on pull request #14841: URL: https://github.com/apache/flink/pull/14841#issuecomment-772182984 ## CI report: * 60e427103cca73ac1fc29fb18893e3996edb1c7a UNKNOWN * cbaf10a07272ce5b3019e98a51cb0ef7f1cdd86b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14075) * 76442a5286c4d6916506fdd6dc877065b3c95b39 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14095) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14800: [FLINK-21192][python] Support setting namespace in RemoteKeyedStateBackend
flinkbot edited a comment on pull request #14800: URL: https://github.com/apache/flink/pull/14800#issuecomment-769581041 ## CI report: * c4c73b24c730472d974aa3d874e0a5c2e116ca51 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14087) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong closed pull request #15057: [FLINK-21552][runtime] Unreserve managed memory if OpaqueMemoryResour…
xintongsong closed pull request #15057: URL: https://github.com/apache/flink/pull/15057 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] KarmaGYZ commented on pull request #15080: [FLINK-21587][coordination] Harden FineGrainedSlotManagerTest and AbstractFineGrainedSlotManagerITCase
KarmaGYZ commented on pull request #15080: URL: https://github.com/apache/flink/pull/15080#issuecomment-790315596 cc @xintongsong This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk merged pull request #15069: [FLINK-21576][runtime] Remove legacy ExecutionVertex#getPreferredLocations()
zhuzhurk merged pull request #15069: URL: https://github.com/apache/flink/pull/15069 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #15069: [FLINK-21576][runtime] Remove legacy ExecutionVertex#getPreferredLocations()
zhuzhurk commented on pull request #15069: URL: https://github.com/apache/flink/pull/15069#issuecomment-790307951 The failed test case is a known issue FLINK-21574. Merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14841: [FLINK-21232][YARN][Common] Introduce pluggable Hadoop delegation token providers
flinkbot edited a comment on pull request #14841: URL: https://github.com/apache/flink/pull/14841#issuecomment-772182984 ## CI report: * 60e427103cca73ac1fc29fb18893e3996edb1c7a UNKNOWN * cbaf10a07272ce5b3019e98a51cb0ef7f1cdd86b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14075) * 76442a5286c4d6916506fdd6dc877065b3c95b39 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15069: [FLINK-21576][runtime] Remove legacy ExecutionVertex#getPreferredLocations()
flinkbot edited a comment on pull request #15069: URL: https://github.com/apache/flink/pull/15069#issuecomment-789495972 ## CI report: * b75bc5b5a449ad252c6ba49e42484ca7228dc4de Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14086) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15066: [FLINK-21488] [connectors/jdbc] Embed JobID into XID's global transaction ID
flinkbot edited a comment on pull request #15066: URL: https://github.com/apache/flink/pull/15066#issuecomment-789284217 ## CI report: * 57eb15aee5784699b0a167dd1f763a3dd03dab81 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14085) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15048: [FLINK-21477][table-planner-blink] Lookup joins now deal with intermediate table scans containing just the table source correctly
flinkbot edited a comment on pull request #15048: URL: https://github.com/apache/flink/pull/15048#issuecomment-787621304 ## CI report: * a1687a0edd159a5b8be339a59c08a79ddfee5069 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14089) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13965) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15081: [FLINK-21553][table-runtime-blink] Copy record if needed when flush window buffer records to state
flinkbot edited a comment on pull request #15081: URL: https://github.com/apache/flink/pull/15081#issuecomment-790283597 ## CI report: * e5392081f205893b2f504a66e9ea3e9d7d0c0432 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14094) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15081: [FLINK-21553][table-runtime-blink] Copy record if needed when flush window buffer records to state
flinkbot commented on pull request #15081: URL: https://github.com/apache/flink/pull/15081#issuecomment-790283597 ## CI report: * e5392081f205893b2f504a66e9ea3e9d7d0c0432 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20496) RocksDB partitioned index filter option
[
https://issues.apache.org/jira/browse/FLINK-20496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li closed FLINK-20496.
-
Resolution: Implemented
Merged into master via:
be78727ae125c036cbb297d020e8a7ad23aae083
f7439740e8e023d458d2ac0cb2a58682eb9b6beb
Will add some release note and create new JIRA to add some instructions on when
and how to use this feature in our document later.
> RocksDB partitioned index filter option
> ---
>
> Key: FLINK-20496
> URL: https://issues.apache.org/jira/browse/FLINK-20496
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Affects Versions: 1.10.2, 1.11.2, 1.12.0
>Reporter: YufeiLiu
>Assignee: YufeiLiu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> When using RocksDBStateBackend and enabling
> {{state.backend.rocksdb.memory.managed}} and
> {{state.backend.rocksdb.memory.fixed-per-slot}}, flink will strictly limited
> rocksdb memory usage which contains "write buffer" and "block cache". With
> these options rocksdb stores index and filters in block cache, because in
> default options index/filters can grows unlimited.
> But it's lead another issue, if high-priority cache(configure by
> {{state.backend.rocksdb.memory.high-prio-pool-ratio}}) can't fit all
> index/filters blocks, it will load all metadata from disk when cache missed,
> and program went extremely slow. According to [Partitioned Index
> Filters|https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters][1],
> we can enable two-level index having acceptable performance when
> index/filters cache missed.
> Enable these options can get over 10x faster in my case[2], I think we can
> add an option {{state.backend.rocksdb.partitioned-index-filters}} and default
> value is false, so we can use this feature easily.
> [1] Partitioned Index Filters:
> https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters
> [2] Deduplicate scenario, state.backend.rocksdb.memory.fixed-per-slot=256M,
> SSD, elapsed time 4.91ms -> 0.33ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] carp84 closed pull request #14341: [FLINK-20496][state backends] RocksDB partitioned index/filters option.
carp84 closed pull request #14341: URL: https://github.com/apache/flink/pull/14341 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-21553) WindowDistinctAggregateITCase#testHopWindow_Cube is unstable
[
https://issues.apache.org/jira/browse/FLINK-21553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17294981#comment-17294981
]
Andy edited comment on FLINK-21553 at 3/4/21, 4:26 AM:
---
[~jark] [~dwysakowicz] [~guoweima] Now when flush buffer to state,
`CombineRecordsFunction` only copy window key because window key is reused.
However, it forgets to copy record which is also reused. I think it's the root
cause of the failure case. For above failed case, there exists `count(distinct
distinctKey)` in sql, distinctKey is the UK of MapState. If pushed object
directly to state when flush buffer to state, it may be updated after it is
pushed into HeapStateBackend because it is a reused object in
`AbstractBytesMultiMap`.
{code:java}
//代码占位符
@Test
def testHopWindow_Cube(): Unit = {
System.setProperty("org.codehaus.janino.source_debugging.enable", "true")
System.setProperty("org.codehaus.janino.source_debugging.dir",
"/Users/zhangjing/IdeaProjects/flink/flink-table/flink-table-planner-blink/src/main/java")
val inputData: Seq[Row] = List(
row("2020-10-10 00:00:01", "Hi", "a"),
row("2020-10-10 00:00:03", "Comment#1", "a"),
row("2020-10-10 00:00:04", null, "a"),
row("2020-10-10 00:00:07", "Hello", "b"),
row("2020-10-10 00:00:06", "Hi", "b"), // out of order
row("2020-10-10 00:00:08", "Comment#2", "a")
)
val dataId = TestValuesTableFactory.registerData(inputData)
tEnv.executeSql(
s"""
|CREATE TABLE T2 (
| `ts` STRING,
| `string` STRING,
| `name` STRING,
| `rowtime` AS TO_TIMESTAMP(`ts`),
| WATERMARK for `rowtime` AS `rowtime` - INTERVAL '1' SECOND
|) WITH (
| 'connector' = 'values',
| 'data-id' = '$dataId',
| 'failing-source' = 'true'
|)
|""".stripMargin)
val sql =
"""
|SELECT
| GROUPING_ID(`name`),
| `name`,
| window_start,
| window_end,
| COUNT(DISTINCT `string`)
|FROM TABLE(
| HOP(TABLE T2, DESCRIPTOR(rowtime), INTERVAL '5' SECOND, INTERVAL '10'
SECOND))
|GROUP BY CUBE(`name`), window_start, window_end
""".stripMargin
val sink = new TestingAppendSink
tEnv.sqlQuery(sql).toAppendStream[Row].addSink(sink)
env.execute()
val data = Seq(
"0,a,2020-10-09T23:59:55,2020-10-10T00:00:05,2",
"0,a,2020-10-10T00:00,2020-10-10T00:00:10,3",
"0,a,2020-10-10T00:00:05,2020-10-10T00:00:15,1",
"0,b,2020-10-10T00:00,2020-10-10T00:00:10,2",
"0,b,2020-10-10T00:00:05,2020-10-10T00:00:15,2",
"1,null,2020-10-09T23:59:55,2020-10-10T00:00:05,2",
"1,null,2020-10-10T00:00,2020-10-10T00:00:10,4",
"1,null,2020-10-10T00:00:05,2020-10-10T00:00:15,3"
)
assertEquals(
data.sorted.mkString("\n"),
sink.getAppendResults.sorted.mkString("\n"))
}
{code}
I simplify the failure case as above case, which could fails frequently (1 per
3~4 times based on *HeapStateBackend + SplitDistinct: false*) on my local
machine. The state in HeapStateBackend is following pic1 when test pass, while
state are pic2 or pic3 when test failure.
!image-2021-03-04-12-05-59-802.png|width=1308,height=292!
!image-2021-03-04-12-07-53-566.png|width=1420,height=361!
!image-2021-03-04-12-08-07-097.png|width=1297,height=211!
After copy record, the case could always be passed.
was (Author: qingru zhang):
[~jark] [~dwysakowicz] [~guoweima] Now when flush buffer to state,
`CombineRecordsFunction` only copy window key because window key is reused.
However, it forgets to copy record which is also reused. I think it's the root
cause of the failure case. For above failed case, there exists `count(distinct
distinctKey)` in sql, distinctKey is the UK of MapState. If pushed object
directly to state when flush buffer to state, it may be updated after it is
pushed into HeapStateBackend because it is a reused object in
`AbstractBytesMultiMap`.
{code:java}
//代码占位符
@Test
def testHopWindow_Cube(): Unit = {
System.setProperty("org.codehaus.janino.source_debugging.enable", "true")
System.setProperty("org.codehaus.janino.source_debugging.dir",
"/Users/zhangjing/IdeaProjects/flink/flink-table/flink-table-planner-blink/src/main/java")
val inputData: Seq[Row] = List(
row("2020-10-10 00:00:01", "Hi", "a"),
row("2020-10-10 00:00:03", "Comment#1", "a"),
row("2020-10-10 00:00:04", null, "a"),
row("2020-10-10 00:00:07", "Hello", "b"),
row("2020-10-10 00:00:06", "Hi", "b"), // out of order
row("2020-10-10 00:00:08", "Comment#2", "a")
)
val dataId = TestValuesTableFactory.registerData(inputData)
tEnv.executeSql(
s"""
|CREATE TABLE T2 (
| `ts` STRING,
| `string` STRING,
| `name` STRING,
| `rowtime` AS TO_TIMESTAMP(`ts`),
| WATERMARK for `rowtime` AS `rowtime` - INTERVAL '1' SECOND
|) WITH (
| 'connector' = 'values',
|
[jira] [Comment Edited] (FLINK-21553) WindowDistinctAggregateITCase#testHopWindow_Cube is unstable
[
https://issues.apache.org/jira/browse/FLINK-21553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17294981#comment-17294981
]
Andy edited comment on FLINK-21553 at 3/4/21, 4:24 AM:
---
[~jark] [~dwysakowicz] [~guoweima] Now when flush buffer to state,
`CombineRecordsFunction` only copy window key because window key is reused.
However, it forgets to copy record which is also reused. I think it's the root
cause of the failure case. For above failed case, there exists `count(distinct
distinctKey)` in sql, distinctKey is the UK of MapState. If pushed object
directly to state when flush buffer to state, it may be updated after it is
pushed into HeapStateBackend because it is a reused object in
`AbstractBytesMultiMap`.
{code:java}
//代码占位符
@Test
def testHopWindow_Cube(): Unit = {
System.setProperty("org.codehaus.janino.source_debugging.enable", "true")
System.setProperty("org.codehaus.janino.source_debugging.dir",
"/Users/zhangjing/IdeaProjects/flink/flink-table/flink-table-planner-blink/src/main/java")
val inputData: Seq[Row] = List(
row("2020-10-10 00:00:01", "Hi", "a"),
row("2020-10-10 00:00:03", "Comment#1", "a"),
row("2020-10-10 00:00:04", null, "a"),
row("2020-10-10 00:00:07", "Hello", "b"),
row("2020-10-10 00:00:06", "Hi", "b"), // out of order
row("2020-10-10 00:00:08", "Comment#2", "a")
)
val dataId = TestValuesTableFactory.registerData(inputData)
tEnv.executeSql(
s"""
|CREATE TABLE T2 (
| `ts` STRING,
| `string` STRING,
| `name` STRING,
| `rowtime` AS TO_TIMESTAMP(`ts`),
| WATERMARK for `rowtime` AS `rowtime` - INTERVAL '1' SECOND
|) WITH (
| 'connector' = 'values',
| 'data-id' = '$dataId',
| 'failing-source' = 'true'
|)
|""".stripMargin)
val sql =
"""
|SELECT
| GROUPING_ID(`name`),
| `name`,
| window_start,
| window_end,
| COUNT(DISTINCT `string`)
|FROM TABLE(
| HOP(TABLE T2, DESCRIPTOR(rowtime), INTERVAL '5' SECOND, INTERVAL '10'
SECOND))
|GROUP BY CUBE(`name`), window_start, window_end
""".stripMargin
val sink = new TestingAppendSink
tEnv.sqlQuery(sql).toAppendStream[Row].addSink(sink)
env.execute()
val data = Seq(
"0,a,2020-10-09T23:59:55,2020-10-10T00:00:05,2",
"0,a,2020-10-10T00:00,2020-10-10T00:00:10,3",
"0,a,2020-10-10T00:00:05,2020-10-10T00:00:15,1",
"0,b,2020-10-10T00:00,2020-10-10T00:00:10,2",
"0,b,2020-10-10T00:00:05,2020-10-10T00:00:15,2",
"1,null,2020-10-09T23:59:55,2020-10-10T00:00:05,2",
"1,null,2020-10-10T00:00,2020-10-10T00:00:10,4",
"1,null,2020-10-10T00:00:05,2020-10-10T00:00:15,3"
)
assertEquals(
data.sorted.mkString("\n"),
sink.getAppendResults.sorted.mkString("\n"))
}
{code}
I simplify the failure case as above case, which could fails frequently (1 per
3~4 times based on HeapStateBackend + SplitDistinct: false) on my local
machine. The state in HeapStateBackend is following pic1 when test pass, while
state are pic2 or pic3 when test failure.
!image-2021-03-04-12-05-59-802.png|width=1308,height=292!
!image-2021-03-04-12-07-53-566.png|width=1420,height=361!
!image-2021-03-04-12-08-07-097.png|width=1297,height=211!
After copy record, the case could always be passed.
was (Author: qingru zhang):
[~jark] [~dwysakowicz] [~guoweima] Now when flush buffer to state,
`CombineRecordsFunction` only copy window key because window key is reused.
However, it forgets to copy record which is also reused. I think it's the root
cause of the failure case. For above failed case, there exists `count(distinct
distinctKey)` in sql, distinctKey is the UK of MapState. If pushed object
directly to state when flush buffer to state, it may be updated after it is
pushed into HeapStateBackend because it is a reused object in
`AbstractBytesMultiMap`.
{code:java}
//代码占位符
@Test
def testHopWindow_Cube(): Unit = {
System.setProperty("org.codehaus.janino.source_debugging.enable", "true")
System.setProperty("org.codehaus.janino.source_debugging.dir",
"/Users/zhangjing/IdeaProjects/flink/flink-table/flink-table-planner-blink/src/main/java")
val inputData: Seq[Row] = List(
row("2020-10-10 00:00:01", "Hi", "a"),
row("2020-10-10 00:00:03", "Comment#1", "a"),
row("2020-10-10 00:00:04", null, "a"),
row("2020-10-10 00:00:07", "Hello", "b"),
row("2020-10-10 00:00:06", "Hi", "b"), // out of order
row("2020-10-10 00:00:08", "Comment#2", "a")
)
val dataId = TestValuesTableFactory.registerData(inputData)
tEnv.executeSql(
s"""
|CREATE TABLE T2 (
| `ts` STRING,
| `string` STRING,
| `name` STRING,
| `rowtime` AS TO_TIMESTAMP(`ts`),
| WATERMARK for `rowtime` AS `rowtime` - INTERVAL '1' SECOND
|) WITH (
| 'connector' = 'values',
|
[GitHub] [flink] flinkbot commented on pull request #15081: [FLINK-21553][table-runtime-blink] Copy record if needed when flush window buffer records to state
flinkbot commented on pull request #15081: URL: https://github.com/apache/flink/pull/15081#issuecomment-790279431 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit e5392081f205893b2f504a66e9ea3e9d7d0c0432 (Thu Mar 04 04:24:42 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
