[GitHub] [flink] wangyang0918 commented on a change in pull request #18854: [FLINK-25648][Kubernetes] Avoid redundant to query Kubernetes deployment when creating task manager pods
wangyang0918 commented on a change in pull request #18854:
URL: https://github.com/apache/flink/pull/18854#discussion_r825264353
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java
##
@@ -130,15 +132,16 @@ public void
createJobManagerComponent(KubernetesJobManagerSpecification kubernet
@Override
public CompletableFuture createTaskManagerPod(KubernetesPod
kubernetesPod) {
Review comment:
I think we could create two TM pods and verify that getting the
jobmanager deployment should only happen once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18980: [FLINK-26421] Use only EnvironmentSettings to configure the environment
flinkbot edited a comment on pull request #18980: URL: https://github.com/apache/flink/pull/18980#issuecomment-1059275933 ## CI report: * 2f723bbfa1c373bf0ce15fb75c12dda1f34124ac Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32938) * c632e6dc69c54b968b0d157cf1903a675b108d7f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32953) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25256) Savepoints do not work with ExternallyInducedSources
[
https://issues.apache.org/jira/browse/FLINK-25256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17505212#comment-17505212
]
Brian Zhou commented on FLINK-25256:
Hi [~dwysakowicz] , I want to ask what the influence is of this issue, for
example, what kind of application or state backend settings is not able to
recover correctly.
We have internally tried to reproduce this issue for some basic applications.
We've found that when savepointing, all of the states are stored in the
metadata, which makes the job to recover correctly. We have tried both stateful
and stateless apps, both filesystem and rocksdb backend, and always seeing the
metadata file in the savepoint directory.
I understand from the code, it does look incorrect in checkpointing when
treating the externally induced source, but we need to get the influence to
help us make plans in our product releases.
> Savepoints do not work with ExternallyInducedSources
>
>
> Key: FLINK-25256
> URL: https://issues.apache.org/jira/browse/FLINK-25256
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0, 1.13.3, 1.12.7
>Reporter: Dawid Wysakowicz
>Assignee: Arvid Heise
>Priority: Major
>
> It is not possible to take a proper savepoint with
> {{ExternallyInducedSource}} or {{ExternallyInducedSourceReader}} (both legacy
> and FLIP-27 versions). The problem is that we're hardcoding
> {{CheckpointOptions}} in the {{triggerHook}}.
> The outcome of current state is that operators would try to take checkpoints
> in the checkpoint location whereas the {{CheckpointCoordinator}} will write
> metadata for those states in the savepoint location.
> Moreover the situation gets even weirder (I have not checked it entirely), if
> we have a mixture of {{ExternallyInducedSource(s)}} and regular sources. In
> such a case the location and format at which the state of a particular task
> is persisted depends on the order of barriers arrival. If a barrier from a
> regular source arrives last the task takes a savepoint, on the other hand if
> last barrier is from an externally induced source it will take a checkpoint.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18980: [FLINK-26421] Use only EnvironmentSettings to configure the environment
flinkbot edited a comment on pull request #18980: URL: https://github.com/apache/flink/pull/18980#issuecomment-1059275933 ## CI report: * 2f723bbfa1c373bf0ce15fb75c12dda1f34124ac Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32938) * c632e6dc69c54b968b0d157cf1903a675b108d7f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19068: [FLINK-26601][Connector/HBase]Flink HBase HBaseRowDataAsyncLookupFunc…
flinkbot edited a comment on pull request #19068: URL: https://github.com/apache/flink/pull/19068#issuecomment-1065814892 ## CI report: * e04b67a0573bf115aa3c6d6d212df01c8230a5f7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32948) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) * 195bf4369bf1eb0cdc1727847e63ce0d6fd31f66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32952) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) * 195bf4369bf1eb0cdc1727847e63ce0d6fd31f66 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) * 195bf4369bf1eb0cdc1727847e63ce0d6fd31f66 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) * 195bf4369bf1eb0cdc1727847e63ce0d6fd31f66 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) * c0d46182cddc810486f7c60baa189073b0397e1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32951) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) * c0d46182cddc810486f7c60baa189073b0397e1a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) * 195bf4369bf1eb0cdc1727847e63ce0d6fd31f66 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) * 195bf4369bf1eb0cdc1727847e63ce0d6fd31f66 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) * c0d46182cddc810486f7c60baa189073b0397e1a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) * c0d46182cddc810486f7c60baa189073b0397e1a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) * 195bf4369bf1eb0cdc1727847e63ce0d6fd31f66 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) * c0d46182cddc810486f7c60baa189073b0397e1a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) * 195bf4369bf1eb0cdc1727847e63ce0d6fd31f66 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26565) Use lateTrigger when the window maxtimestap of data is less than currentwatermark and it is not discarded because the allow latency parameter
[ https://issues.apache.org/jira/browse/FLINK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17505187#comment-17505187 ] hehuiyuan commented on FLINK-26565: --- If it is a new window when a latency data that is the first data in the window to which it belongs arrived , but the watermark does not arrived the time (window.maxTimestamp + allow-latency), is it more reasonable to reuse latertrigger ? [~jark] > Use lateTrigger when the window maxtimestap of data is less than > currentwatermark and it is not discarded because the allow latency parameter > --- > > Key: FLINK-26565 > URL: https://issues.apache.org/jira/browse/FLINK-26565 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API, Table SQL / Planner, Table SQL / Runtime >Reporter: hehuiyuan >Priority: Minor > Labels: pull-request-available > Attachments: image-2022-03-10-11-27-52-891.png > > > Use lateTrigger when the window maxtimestap of data is less than > currentwatermark . > The data is not discarded because the allow latency parameter. > The lateTrigger is set and the window just appeared with this data. > !image-2022-03-10-11-27-52-891.png|width=543,height=318! -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) * 195bf4369bf1eb0cdc1727847e63ce0d6fd31f66 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) * c0d46182cddc810486f7c60baa189073b0397e1a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19069: [FLINK-24862][Connectors / Hive][backport]Fix user-defined hive udaf/udtf cannot be used normally in hive dialect for flink1.14
flinkbot edited a comment on pull request #19069: URL: https://github.com/apache/flink/pull/19069#issuecomment-1065823671 ## CI report: * 008b52783d14aff212e36afb2def20690c68a81c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32950) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) * c0d46182cddc810486f7c60baa189073b0397e1a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * ff0bcbd841960c3e524a62c3c45f426707fb5306 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32946) * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) * 195bf4369bf1eb0cdc1727847e63ce0d6fd31f66 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19069: [FLINK-24862][Connectors / Hive][backport]Fix user-defined hive udaf/udtf cannot be used normally in hive dialect for flink1.14
flinkbot commented on pull request #19069: URL: https://github.com/apache/flink/pull/19069#issuecomment-1065823671 ## CI report: * 008b52783d14aff212e36afb2def20690c68a81c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) * c0d46182cddc810486f7c60baa189073b0397e1a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] hehuiyuan opened a new pull request #19069: [FLINK-24862][Connectors / Hive][backport]Fix user-defined hive udaf/udtf cannot be used normally in hive dialect for flink1.14
hehuiyuan opened a new pull request #19069: URL: https://github.com/apache/flink/pull/19069 Backport of https://github.com/apache/flink/pull/17761 ## What is the purpose of the change Fix user-defined hive udaf/udtf cannot be used normally in hive dialect ## Brief change log FunctionCatalog#validateAndPrepareFunction method skip validate TableFunctionDefinition ## Verifying this change This change added tests and can be verified as follows: - Added unit test HiveDialectITCase#testTemporaryFunctionUDAF for create temporary udaf function ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no ) - The runtime per-record code paths (performance sensitive): (no ) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no ) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? ( no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) * c0d46182cddc810486f7c60baa189073b0397e1a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] realdengziqi commented on a change in pull request #18982: [FLINK-26334][datastream] Fix getWindowStartWithOffset in TimeWindow.java
realdengziqi commented on a change in pull request #18982:
URL: https://github.com/apache/flink/pull/18982#discussion_r824733805
##
File path:
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/window/TimeWindow.java
##
@@ -220,7 +220,9 @@ public TimeWindowSerializerSnapshot() {
* @return window start
*/
public static long getWindowStartWithOffset(long timestamp, long offset,
long windowSize) {
-return timestamp - (timestamp - offset + windowSize) % windowSize;
+return timestamp
+- (timestamp - offset) % windowSize
+- (windowSize & (timestamp - offset) >> 63);
Review comment:
Thanks for reviewing our code, we made a mistake. What we want to change
is `org.apache.flink.streaming.api.windowing.windows.TimeWindow`, not
`org.apache.flink.table.runtime.operators.window.TimeWindow`
In the new commit, we find the class we want to change and modify the unit
test.
Please ignore this change.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) * c0d46182cddc810486f7c60baa189073b0397e1a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * fa874e5d8f48d444e7a33de55ab931ef5a2b Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32885) * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32949) * fc25bf608689248a0f33913c0b37bf4b8725afe5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * fa874e5d8f48d444e7a33de55ab931ef5a2b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32885) * c54dcadb4e91c8c3a3ce593fdeec6a7cef83b8ef UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
gaoyunhaii commented on a change in pull request #19052:
URL: https://github.com/apache/flink/pull/19052#discussion_r825246703
##
File path:
flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/sink/utils/PartSizeOnlyRollingPolicy.java
##
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.utils;
+
+import org.apache.flink.streaming.api.functions.sink.filesystem.PartFileInfo;
+import org.apache.flink.streaming.api.functions.sink.filesystem.RollingPolicy;
+
+import java.io.IOException;
+
+/** A testing rolling policy to ensure in-progress file exists. */
+public class PartSizeOnlyRollingPolicy implements
RollingPolicy {
Review comment:
I think this could be merged with PartSizeAndCheckpointRollingPolicy by
passing shouldRollOnCheckpoint as parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19068: [FLINK-26601][Connector/HBase]Flink HBase HBaseRowDataAsyncLookupFunc…
flinkbot edited a comment on pull request #19068: URL: https://github.com/apache/flink/pull/19068#issuecomment-1065814892 ## CI report: * e04b67a0573bf115aa3c6d6d212df01c8230a5f7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32948) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19068: [FLINK-26601][Connector/HBase]Flink HBase HBaseRowDataAsyncLookupFunc…
flinkbot commented on pull request #19068: URL: https://github.com/apache/flink/pull/19068#issuecomment-1065814892 ## CI report: * e04b67a0573bf115aa3c6d6d212df01c8230a5f7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26601) Flink HBase HBaseRowDataAsyncLookupFunction threadPool not close cause container has so many 'hbase-async-lookup-worker' thread
[ https://issues.apache.org/jira/browse/FLINK-26601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-26601: --- Labels: pull-request-available (was: ) > Flink HBase HBaseRowDataAsyncLookupFunction threadPool not close cause > container has so many 'hbase-async-lookup-worker' thread > > > Key: FLINK-26601 > URL: https://issues.apache.org/jira/browse/FLINK-26601 > Project: Flink > Issue Type: Bug > Components: Connectors / HBase >Affects Versions: 1.13.0 >Reporter: zhisheng >Priority: Major > Labels: pull-request-available > Attachments: image-2022-03-11-15-31-19-206.png, > image-2022-03-11-15-32-23-721.png, image-2022-03-11-15-32-46-701.png, > image-2022-03-11-15-32-58-084.png > > > [https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-hbase-2.2/src/main/java/org/apache/flink/connector/hbase2/source/HBaseRowDataAsyncLookupFunction.java#L111] > the thread pool create in the open function, but not gracefulShutdown in the > close function, when the job failover, it will renew the thread pool > !image-2022-03-11-15-32-58-084.png|width=516,height=288! > !image-2022-03-11-15-32-23-721.png|width=532,height=274! > !image-2022-03-11-15-31-19-206.png|width=555,height=369! > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] zhisheng17 opened a new pull request #19068: [FLINK-26601][Connector/HBase]Flink HBase HBaseRowDataAsyncLookupFunc…
zhisheng17 opened a new pull request #19068: URL: https://github.com/apache/flink/pull/19068 …tion threadPool not close cause container has so many 'hbase-async-lookup-worker' thread ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * fa874e5d8f48d444e7a33de55ab931ef5a2b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32885) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
flinkbot edited a comment on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1064736594 ## CI report: * fa874e5d8f48d444e7a33de55ab931ef5a2b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32885) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #19052: [FLINK-26580] FileSink Compactor is not properly processing in-progress files.
gaoyunhaii commented on pull request #19052: URL: https://github.com/apache/flink/pull/19052#issuecomment-1065804214 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * ff0bcbd841960c3e524a62c3c45f426707fb5306 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32946) * 84846361d6db21a5ac721494ebb5badf23b274a1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32947) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * ff0bcbd841960c3e524a62c3c45f426707fb5306 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32946) * 84846361d6db21a5ac721494ebb5badf23b274a1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * ff0bcbd841960c3e524a62c3c45f426707fb5306 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32946) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * ff0bcbd841960c3e524a62c3c45f426707fb5306 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32946) * 84846361d6db21a5ac721494ebb5badf23b274a1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * ff0bcbd841960c3e524a62c3c45f426707fb5306 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32946) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * ff0bcbd841960c3e524a62c3c45f426707fb5306 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32946) * 84846361d6db21a5ac721494ebb5badf23b274a1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * ff0bcbd841960c3e524a62c3c45f426707fb5306 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32946) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * ff0bcbd841960c3e524a62c3c45f426707fb5306 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32946) * 84846361d6db21a5ac721494ebb5badf23b274a1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * ff0bcbd841960c3e524a62c3c45f426707fb5306 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32946) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * ff0bcbd841960c3e524a62c3c45f426707fb5306 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32946) * 84846361d6db21a5ac721494ebb5badf23b274a1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * ff0bcbd841960c3e524a62c3c45f426707fb5306 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32946) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * d178dd422b702f51ee85b41e43ec4b306c60aa8c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32914) * ff0bcbd841960c3e524a62c3c45f426707fb5306 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32946) * 84846361d6db21a5ac721494ebb5badf23b274a1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * d178dd422b702f51ee85b41e43ec4b306c60aa8c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32914) * ff0bcbd841960c3e524a62c3c45f426707fb5306 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32946) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * d178dd422b702f51ee85b41e43ec4b306c60aa8c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32914) * ff0bcbd841960c3e524a62c3c45f426707fb5306 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * d178dd422b702f51ee85b41e43ec4b306c60aa8c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32914) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * d178dd422b702f51ee85b41e43ec4b306c60aa8c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32914) * ff0bcbd841960c3e524a62c3c45f426707fb5306 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * d178dd422b702f51ee85b41e43ec4b306c60aa8c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32914) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * d178dd422b702f51ee85b41e43ec4b306c60aa8c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32914) * ff0bcbd841960c3e524a62c3c45f426707fb5306 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * d178dd422b702f51ee85b41e43ec4b306c60aa8c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32914) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * d178dd422b702f51ee85b41e43ec4b306c60aa8c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32914) * ff0bcbd841960c3e524a62c3c45f426707fb5306 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] syhily edited a comment on pull request #18838: [FLINK-26177][Connector/pulsar] Use testcontainer pulsar runtime instead o…
syhily edited a comment on pull request #18838: URL: https://github.com/apache/flink/pull/18838#issuecomment-1064890584 * 1st run: [SUCCESS](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32872=results) * 2nd run: [SUCCESS](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32904=results) * 3rd run: [SUCCESS](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32939=results) * 4th run: [SUCCESS](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32942=results) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18838: [FLINK-26177][Connector/pulsar] Use testcontainer pulsar runtime instead o…
flinkbot edited a comment on pull request #18838: URL: https://github.com/apache/flink/pull/18838#issuecomment-1044141081 ## CI report: * dd09bb45c7c18e20ce9cc2fa75713aa70bfb4c2e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32942) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19067: [FLINK-26614]Upsert Kafka SQL Connector:Support startup mode of timestamp and specific offsets
flinkbot edited a comment on pull request #19067: URL: https://github.com/apache/flink/pull/19067#issuecomment-1065421908 ## CI report: * 981b73590bf770f200b4fcca1f6dbfcb4361cf2a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32941) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18838: [FLINK-26177][Connector/pulsar] Use testcontainer pulsar runtime instead o…
flinkbot edited a comment on pull request #18838: URL: https://github.com/apache/flink/pull/18838#issuecomment-1044141081 ## CI report: * 826d744793c3a11c0e41cb7fb7422566a94501b1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32939) * dd09bb45c7c18e20ce9cc2fa75713aa70bfb4c2e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32942) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18838: [FLINK-26177][Connector/pulsar] Use testcontainer pulsar runtime instead o…
flinkbot edited a comment on pull request #18838: URL: https://github.com/apache/flink/pull/18838#issuecomment-1044141081 ## CI report: * 826d744793c3a11c0e41cb7fb7422566a94501b1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32939) * dd09bb45c7c18e20ce9cc2fa75713aa70bfb4c2e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18982: [FLINK-26334][datastream] Fix getWindowStartWithOffset in TimeWindow.java
flinkbot edited a comment on pull request #18982: URL: https://github.com/apache/flink/pull/18982#issuecomment-1059437393 ## CI report: * 5bf80617fee372c23728e2c4ba4192198e34205e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32931) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19066: [FLINK-26596][runtime] Adds leadership loss handling
flinkbot edited a comment on pull request #19066: URL: https://github.com/apache/flink/pull/19066#issuecomment-1065357860 ## CI report: * 0fd2f11bda7d4c42a407a0fa8f4811ce38ae53a2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32940) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] syhily edited a comment on pull request #18838: [FLINK-26177][Connector/pulsar] Use testcontainer pulsar runtime instead o…
syhily edited a comment on pull request #18838: URL: https://github.com/apache/flink/pull/18838#issuecomment-1064890584 * 1st run: [SUCCESS](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32872=results) * 2nd run: [SUCCESS](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32904=results) * 3rd run: [SUCCESS](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32939=results) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18838: [FLINK-26177][Connector/pulsar] Use testcontainer pulsar runtime instead o…
flinkbot edited a comment on pull request #18838: URL: https://github.com/apache/flink/pull/18838#issuecomment-1044141081 ## CI report: * 826d744793c3a11c0e41cb7fb7422566a94501b1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32939) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18980: [FLINK-26421] Use only EnvironmentSettings to configure the environment
flinkbot edited a comment on pull request #18980: URL: https://github.com/apache/flink/pull/18980#issuecomment-1059275933 ## CI report: * 2f723bbfa1c373bf0ce15fb75c12dda1f34124ac Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32938) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19067: [FLINK-26614]Upsert Kafka SQL Connector:Support startup mode of timestamp and specific offsets
flinkbot edited a comment on pull request #19067: URL: https://github.com/apache/flink/pull/19067#issuecomment-1065421908 ## CI report: * 981b73590bf770f200b4fcca1f6dbfcb4361cf2a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32941) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26614) Upsert Kafka SQL Connector:Support startup mode of timestamp and specific offsets
[ https://issues.apache.org/jira/browse/FLINK-26614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-26614: --- Labels: pull-request-available (was: ) > Upsert Kafka SQL Connector:Support startup mode of timestamp and specific > offsets > - > > Key: FLINK-26614 > URL: https://issues.apache.org/jira/browse/FLINK-26614 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.14.4 >Reporter: Ning Ma >Priority: Blocker > Labels: pull-request-available > Fix For: 1.14.4 > > > When we run Flink SQL using upsert-kafka connector, it cannot support startup > mode of timestamp or specific offsets. It is determined to read data from the > earliest. > I don't think it's reasonable. Because the data stored in Kafka is incomplete > that it only saves data for 7 days in most cases. So why don't we consume > data from the specified timestamp or offsets, and this can improve the > efficiency of reading data from Kafka. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #19067: [FLINK-26614]Upsert Kafka SQL Connector:Support startup mode of timestamp and specific offsets
flinkbot commented on pull request #19067: URL: https://github.com/apache/flink/pull/19067#issuecomment-1065421908 ## CI report: * 981b73590bf770f200b4fcca1f6dbfcb4361cf2a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-5058) taskManagerMemory attribute set wrong value in FlinkShell
[
https://issues.apache.org/jira/browse/FLINK-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-5058:
--
Labels: pull-request-available (was: )
> taskManagerMemory attribute set wrong value in FlinkShell
> -
>
> Key: FLINK-5058
> URL: https://issues.apache.org/jira/browse/FLINK-5058
> Project: Flink
> Issue Type: Bug
> Components: Scala Shell
>Affects Versions: 1.1.3
>Reporter: iteblog
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.4, 1.2.0
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> In FlinkShell.scala, someone mistakenly set {{slots}} value to
> {{taskManagerMemory}} attribute as follow:
> {code}
> // set configuration from user input
> yarnConfig.jobManagerMemory.foreach((jmMem) => args ++= Seq("-yjm",
> jmMem.toString))
> yarnConfig.slots.foreach((tmMem) => args ++= Seq("-ytm", tmMem.toString))
> yarnConfig.name.foreach((name) => args ++= Seq("-ynm", name.toString))
> yarnConfig.queue.foreach((queue) => args ++= Seq("-yqu", queue.toString))
> yarnConfig.slots.foreach((slots) => args ++= Seq("-ys", slots.toString))
> {code}
> Note on the third line: {{yarnConfig.slots.foreach((tmMem) => args ++=
> Seq("-ytm", tmMem.toString))}} , we set {{slots}} value to {{-ytm}}
> attribute, the right code should be:
> {code}
> // set configuration from user input
> yarnConfig.jobManagerMemory.foreach((jmMem) => args ++= Seq("-yjm",
> jmMem.toString))
> yarnConfig.taskManagerMemory.foreach((tmMem) => args ++= Seq("-ytm",
> tmMem.toString))
> yarnConfig.name.foreach((name) => args ++= Seq("-ynm", name.toString))
> yarnConfig.queue.foreach((queue) => args ++= Seq("-yqu", queue.toString))
> yarnConfig.slots.foreach((slots) => args ++= Seq("-ys", slots.toString))
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #19064: [BP-1.14][FLINK-24538][runtime][tests] Fix race condition when offering information to leader event queue in TestingRetrievalBase
flinkbot edited a comment on pull request #19064: URL: https://github.com/apache/flink/pull/19064#issuecomment-1065256305 ## CI report: * c815902e29a5af05121a07abc52c012b174f2c37 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32934) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] mnmaning opened a new pull request #19067: [FLINK-5058]Upsert Kafka SQL Connector:Support startup mode of timestamp and specific offsets
mnmaning opened a new pull request #19067: URL: https://github.com/apache/flink/pull/19067 …ific offsets ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19065: [FLINK-26047][yarn] Support usrlib in HDFS for YARN application mode
flinkbot edited a comment on pull request #19065: URL: https://github.com/apache/flink/pull/19065#issuecomment-1065263534 ## CI report: * e70d083bb0be3f9ca0dbb1aae4815c1a353fbea6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32935) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26256) AWS SDK Async Event Loop Group Classloader Issue
[
https://issues.apache.org/jira/browse/FLINK-26256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17505084#comment-17505084
]
Zichen Liu commented on FLINK-26256:
Possible solutions:
# Create a separate ELG per subtask. *Unfortunately this would create too many
ELGs and therefore thread pools incurring a large and needless resource cost.*
# Create a separate ELG per Flink job. *This would be the most ideal solution
if it were possible. Unfortunately, it seems that it is impossible to get any
unique identifier e.g. jobId, hash etc for the job in the sink/sinkWriter
context. The Sink.InitContext was a good potential candidate, but sadly none of
the fields could be used as a truly unique identifier.*
# Attach the correct classloader to ELG spawned threads. *I'm not sure there
can be a correct classloader without creating a new one. AFAIK, the following
are available: application, plugin and user classloaders. It seems from
[here|https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#:~:text=In%20setups%20where,code%20jar(s).]
that each of these is per cluster rather than per job. I believe the only way
to get a classloader per job is to create them manually.*
I would be very interested in any suggestions on how we can proceed with this
issue, maybe [~dmvk] [~dannycranmer] can help?
> AWS SDK Async Event Loop Group Classloader Issue
>
>
> Key: FLINK-26256
> URL: https://issues.apache.org/jira/browse/FLINK-26256
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Reporter: Danny Cranmer
>Assignee: Zichen Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> h3. Background
> AWS SDK v2 async clients use a Netty async client for Kinesis Data
> Streams/Firehose sink and Kinesis Data Streams EFO consumer. The SDK creates
> a shared thread pool for Netty to use for network operations when one is not
> configured. The thread pool is managed by a shared ELG (event loop group),
> and this is stored in a static field. We do not configure this for the AWS
> connectors in the Flink codebase.
> When threads are spawned within the ELG, they inherit the context classloader
> from the current thread. If the ELG is created from a shared classloader, for
> instance Flink parent classloader, or MiniCluster parent classloader,
> multiple Flink jobs can share the same ELG. When an ELG thread is spawned
> from a Flink job, it will inherit the Flink user classloader. When this job
> completes/fails, the classloader is destroyed, however the Netty thread is
> still referencing it, and this leads to below exception.
> h3. Impact
> This issue *does not* impact jobs deployed to TM when AWS SDK v2 is loaded
> via the Flink User Classloader. It is expected this is the standard
> deployment configuration.
> This issue is known to impact:
> - Flink mini cluster, for example in integration tests (FLINK-26064)
> - Flink cluster loading AWS SDK v2 via parent classloader
> h3. Suggested solution
> There are a few possible solutions, as discussed
> https://github.com/apache/flink/pull/18733
> 1. Create a separate ELG per Flink job
> 2. Create a separate ELG per subtask
> 3. Attach the correct classloader to ELG spawned threads
> h3. Error Stack
> (shortened stack trace, as full is too large)
> {noformat}
> Feb 09 20:05:04 java.util.concurrent.ExecutionException:
> software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
> HTTP request: Trying to access closed classloader. Please check if you store
> classloaders directly or indirectly in static fields. If the stacktrace
> suggests that the leak occurs in a third party library and cannot be fixed
> immediately, you can disable this check with the configuration
> 'classloader.check-leaked-classloader'.
> Feb 09 20:05:04 at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> Feb 09 20:05:04 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> (...)
> Feb 09 20:05:04 Caused by:
> software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
> HTTP request: Trying to access closed classloader. Please check if you store
> classloaders directly or indirectly in static fields. If the stacktrace
> suggests that the leak occurs in a third party library and cannot be fixed
> immediately, you can disable this check with the configuration
> 'classloader.check-leaked-classloader'.
> Feb 09 20:05:04 at
> software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
> Feb 09 20:05:04 at
> software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
> Feb
[GitHub] [flink] flinkbot edited a comment on pull request #18982: [FLINK-26334][datastream] Fix getWindowStartWithOffset in TimeWindow.java
flinkbot edited a comment on pull request #18982: URL: https://github.com/apache/flink/pull/18982#issuecomment-1059437393 ## CI report: * 5bf80617fee372c23728e2c4ba4192198e34205e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32931) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18982: [FLINK-26334][datastream] Fix getWindowStartWithOffset in TimeWindow.java
flinkbot edited a comment on pull request #18982: URL: https://github.com/apache/flink/pull/18982#issuecomment-1059437393 ## CI report: * 5bf80617fee372c23728e2c4ba4192198e34205e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32931) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] realdengziqi commented on pull request #18982: [FLINK-26334][datastream] Fix getWindowStartWithOffset in TimeWindow.java
realdengziqi commented on pull request #18982: URL: https://github.com/apache/flink/pull/18982#issuecomment-1065390926 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] realdengziqi removed a comment on pull request #18982: [FLINK-26334][datastream] Fix getWindowStartWithOffset in TimeWindow.java
realdengziqi removed a comment on pull request #18982: URL: https://github.com/apache/flink/pull/18982#issuecomment-1065235908 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18982: [FLINK-26334][datastream] Fix getWindowStartWithOffset in TimeWindow.java
flinkbot edited a comment on pull request #18982: URL: https://github.com/apache/flink/pull/18982#issuecomment-1059437393 ## CI report: * 5bf80617fee372c23728e2c4ba4192198e34205e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32931) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26616) AdaptiveSchedulerITCase.testExceptionHistoryIsRetrievableFromTheRestAPI failed with a timeout
[
https://issues.apache.org/jira/browse/FLINK-26616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17505051#comment-17505051
]
Sebastian Mattheis commented on FLINK-26616:
Same here:
[https://dev.azure.com/smattheis/Flink/_build/results?buildId=39=logs=0a15d512-44ac-5ba5-97ab-13a5d066c22c=9a028d19-6c4b-5a4e-d378-03fca149d0b1=6426]
{code:java}
Mar 11 15:35:07 [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time
elapsed: 80.865 s <<< FAILURE! - in
org.apache.flink.test.scheduling.AdaptiveSchedulerITCase
Mar 11 15:35:07 [ERROR]
org.apache.flink.test.scheduling.AdaptiveSchedulerITCase.testExceptionHistoryIsRetrievableFromTheRestAPI
Time elapsed: 60.122 s <<< ERROR!
Mar 11 15:35:07 java.util.concurrent.TimeoutException: Condition was not met in
given timeout.
...
Mar 11 15:35:07 at
org.apache.flink.test.scheduling.AdaptiveSchedulerITCase.testExceptionHistoryIsRetrievableFromTheRestAPI(AdaptiveSchedulerITCase.java:268)
...
{code}
> AdaptiveSchedulerITCase.testExceptionHistoryIsRetrievableFromTheRestAPI
> failed with a timeout
> -
>
> Key: FLINK-26616
> URL: https://issues.apache.org/jira/browse/FLINK-26616
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Priority: Critical
> Fix For: 1.15.0
>
>
> {{AdaptiveSchedulerITCase.}} failed in [this
> build|https://dev.azure.com/mapohl/flink/_build/results?buildId=855=logs=0a15d512-44ac-5ba5-97ab-13a5d066c22c=9a028d19-6c4b-5a4e-d378-03fca149d0b1=5778]
> due to a timeout.
> {code}
> Mar 11 14:41:36 [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 76.177 s <<< FAILURE! - in
> org.apache.flink.test.scheduling.AdaptiveSchedulerITCase
> Mar 11 14:41:36 [ERROR]
> org.apache.flink.test.scheduling.AdaptiveSchedulerITCase.testExceptionHistoryIsRetrievableFromTheRestAPI
> Time elapsed: 60.146 s <<< ERROR!
> Mar 11 14:41:36 java.util.concurrent.TimeoutException: Condition was not met
> in given timeout.
> Mar 11 14:41:36 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:167)
> Mar 11 14:41:36 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:145)
> Mar 11 14:41:36 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:137)
> Mar 11 14:41:36 at
> org.apache.flink.test.scheduling.AdaptiveSchedulerITCase.testExceptionHistoryIsRetrievableFromTheRestAPI(AdaptiveSchedulerITCase.java:268)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
