[GitHub] [flink] FangYongs commented on a diff in pull request #23301: [FLINK-32848][tests][JUnit5 migration] Migrate flink-runtime/rpc tests to JUnit5
FangYongs commented on code in PR #23301:
URL: https://github.com/apache/flink/pull/23301#discussion_r1328268676
##
flink-runtime/src/test/java/org/apache/flink/runtime/rpc/RpcSSLAuthITCase.java:
##
@@ -84,9 +82,11 @@ public void testConnectFailure() throws Exception {
try {
// to test whether the test is still good:
-// - create actorSystem2 with sslConfig1 (same as actorSystem1)
and see that both can
+// - create actorSystem2 with sslConfig1 (same as actorSystem1)
+// and see that both can
Review Comment:
Why is there a new line here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] FangYongs commented on a diff in pull request #23301: [FLINK-32848][tests][JUnit5 migration] Migrate flink-runtime/rpc tests to JUnit5
FangYongs commented on code in PR #23301:
URL: https://github.com/apache/flink/pull/23301#discussion_r1328266864
##
flink-runtime/src/test/java/org/apache/flink/runtime/rpc/AsyncCallsTest.java:
##
@@ -117,22 +113,22 @@ private void
runScheduleWithNoDelayTest(RpcEndpointFactory factory) throws Excep
Duration.ofSeconds(30L));
String str = result.get(30, TimeUnit.SECONDS);
-assertEquals("test", str);
+assertThat(str).isEqualTo("test");
// validate that no concurrent access happened
-assertFalse("Rpc Endpoint had concurrent access",
concurrentAccess.get());
+assertThat(concurrentAccess).isFalse();
Review Comment:
Use `assertThat(...).withFailMessage("Rpc Endpoint had concurrent
access").isFalse()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #23435: [FLINK-33033][tests] Add haservice benchmark test for dispatcher
flinkbot commented on PR #23435: URL: https://github.com/apache/flink/pull/23435#issuecomment-1722782353 ## CI report: * 0f70393f2b864837821a10790d5549b5f1c72989 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33033) Add haservice micro benchmark for olap
[ https://issues.apache.org/jira/browse/FLINK-33033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33033: --- Labels: pull-request-available (was: ) > Add haservice micro benchmark for olap > -- > > Key: FLINK-33033 > URL: https://issues.apache.org/jira/browse/FLINK-33033 > Project: Flink > Issue Type: Sub-task > Components: Benchmarks >Affects Versions: 1.19.0 >Reporter: Fang Yong >Assignee: Fang Yong >Priority: Major > Labels: pull-request-available > > Add micro benchmarks of haservice for olap to improve the performance for > short-lived jobs -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] FangYongs opened a new pull request, #23435: [FLINK-33033][tests] Add haservice benchmark test for dispatcher
FangYongs opened a new pull request, #23435: URL: https://github.com/apache/flink/pull/23435 ## What is the purpose of the change Add benchmark test classes for high availability service in dispatcher, they will be used in flink-benchmark project ## Brief change log - Add `DispatcherHighAvailableServiceBenchmark` for zookeeper and none high availability service - Submit job to dispatcher with different haservices in `DispatcherHighAvailableServiceBenchmark` ## Verifying this change This change added tests and can be verified as follows: - Added `DispatcherHighAvailabilityServiceBenchmarkTest` to submit jobs in `DispatcherHighAvailabilityServiceBenchmark` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) no - The serializers: (yes / no / don't know) no - The runtime per-record code paths (performance sensitive): (yes / no / don't know) no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) no - The S3 file system connector: (yes / no / don't know) no ## Documentation - Does this pull request introduce a new feature? (yes / no) no - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-33033) Add haservice micro benchmark for olap
[ https://issues.apache.org/jira/browse/FLINK-33033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fang Yong reassigned FLINK-33033: - Assignee: Fang Yong > Add haservice micro benchmark for olap > -- > > Key: FLINK-33033 > URL: https://issues.apache.org/jira/browse/FLINK-33033 > Project: Flink > Issue Type: Sub-task > Components: Benchmarks >Affects Versions: 1.19.0 >Reporter: Fang Yong >Assignee: Fang Yong >Priority: Major > > Add micro benchmarks of haservice for olap to improve the performance for > short-lived jobs -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33102) Document the standalone autoscaler
Rui Fan created FLINK-33102: --- Summary: Document the standalone autoscaler Key: FLINK-33102 URL: https://issues.apache.org/jira/browse/FLINK-33102 Project: Flink Issue Type: Sub-task Reporter: Rui Fan Assignee: Rui Fan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33101) Add the integration test for standalone autoscaler
Rui Fan created FLINK-33101: --- Summary: Add the integration test for standalone autoscaler Key: FLINK-33101 URL: https://issues.apache.org/jira/browse/FLINK-33101 Project: Flink Issue Type: Sub-task Reporter: Rui Fan Assignee: Samrat Deb -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32947) Standalone autoscaler supports the metric reporter
[ https://issues.apache.org/jira/browse/FLINK-32947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-32947: Parent: FLINK-32723 Issue Type: Sub-task (was: New Feature) > Standalone autoscaler supports the metric reporter > -- > > Key: FLINK-32947 > URL: https://issues.apache.org/jira/browse/FLINK-32947 > Project: Flink > Issue Type: Sub-task > Components: Autoscaler >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > > The flink kubernetes operator itself sets up the metrics reporters and > provides the base metric groups, for standalone implementation we need to > create a new metric reporter otherwise we cannot report the metrics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32947) Standalone autoscaler supports the metric reporter
[ https://issues.apache.org/jira/browse/FLINK-32947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-32947: Summary: Standalone autoscaler supports the metric reporter (was: Autoscaler standalone mode supports the metric report) > Standalone autoscaler supports the metric reporter > -- > > Key: FLINK-32947 > URL: https://issues.apache.org/jira/browse/FLINK-32947 > Project: Flink > Issue Type: New Feature > Components: Autoscaler >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > > FLIP-334 is decoupling the autoscaler and kubernetes, the autoscaler can run > with standalone mode after FLIP-334. > The flink kubernetes operator itself sets up the metrics reporters and > provides the base metric groups, for standalone implementation we need to > create a new metric reporter otherwise we cannot report the metrics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32947) Standalone autoscaler supports the metric reporter
[ https://issues.apache.org/jira/browse/FLINK-32947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-32947: Description: The flink kubernetes operator itself sets up the metrics reporters and provides the base metric groups, for standalone implementation we need to create a new metric reporter otherwise we cannot report the metrics. (was: FLIP-334 is decoupling the autoscaler and kubernetes, the autoscaler can run with standalone mode after FLIP-334. The flink kubernetes operator itself sets up the metrics reporters and provides the base metric groups, for standalone implementation we need to create a new metric reporter otherwise we cannot report the metrics.) > Standalone autoscaler supports the metric reporter > -- > > Key: FLINK-32947 > URL: https://issues.apache.org/jira/browse/FLINK-32947 > Project: Flink > Issue Type: New Feature > Components: Autoscaler >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > > The flink kubernetes operator itself sets up the metrics reporters and > provides the base metric groups, for standalone implementation we need to > create a new metric reporter otherwise we cannot report the metrics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-connector-pulsar] ruanhang1993 opened a new pull request, #61: [FLINK-30400][build] Stop bundling flink-connector-base
ruanhang1993 opened a new pull request, #61: URL: https://github.com/apache/flink-connector-pulsar/pull/61 ## Purpose of the change Stop bundling flink-connector-base. ## Verifying this change This change is a code cleanup without any test coverage. ## Significant changes - [x] Dependencies have been added or upgraded - [ ] Public API has been changed (Public API is any class annotated with `@Public(Evolving)`) - [ ] Serializers have been changed - [ ] New feature has been introduced - If yes, how is this documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-33100) Implement YarnJobFetcher for Standalone Autoscaler
Rui Fan created FLINK-33100: --- Summary: Implement YarnJobFetcher for Standalone Autoscaler Key: FLINK-33100 URL: https://issues.apache.org/jira/browse/FLINK-33100 Project: Flink Issue Type: Sub-task Reporter: Rui Fan Assignee: Rui Fan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #23434: Update create.md , ifx spell error, change statment to statement
flinkbot commented on PR #23434: URL: https://github.com/apache/flink/pull/23434#issuecomment-1722687634 ## CI report: * c80bf2d29cf4097101fa7af63c093d0656173fc6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-33099) Support the Standalone Autoscaler
Rui Fan created FLINK-33099: --- Summary: Support the Standalone Autoscaler Key: FLINK-33099 URL: https://issues.apache.org/jira/browse/FLINK-33099 Project: Flink Issue Type: Sub-task Reporter: Rui Fan Assignee: Rui Fan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33098) Support kubernetes autoscaler using generic interface
Rui Fan created FLINK-33098: --- Summary: Support kubernetes autoscaler using generic interface Key: FLINK-33098 URL: https://issues.apache.org/jira/browse/FLINK-33098 Project: Flink Issue Type: Sub-task Reporter: Rui Fan Assignee: Rui Fan # Moving all classes aren't related to kubernetes to flink-autoscaler module # Support kubernetes autoscaler using generic interface # Removing the flink-kubernetes-operator-autoscaler module # Removing the option prefix(kubernetes.operator.) for all options and update the doc(All old option names are marked with withDeprecatedKeys to ensure the compatibility.) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33090) CheckpointsCleaner clean individual checkpoint states in parallel
[ https://issues.apache.org/jira/browse/FLINK-33090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766182#comment-17766182 ] Hangxiang Yu commented on FLINK-33090: -- Sure. Thanks for the pr. I will take a look asap. > CheckpointsCleaner clean individual checkpoint states in parallel > - > > Key: FLINK-33090 > URL: https://issues.apache.org/jira/browse/FLINK-33090 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.17.1 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Major > Labels: pull-request-available > > Currently CheckpointsCleaner clean multiple checkpoints in parallel with > JobManager's ioExecutor, however each checkpoint states is cleaned > sequentially. With thousands of StateObjects to clean this can take long time > on some checkpoint storage, if longer than the checkpoint interval this > prevents new checkpointing. > The proposal is to use the same ioExecutor to clean up each checkpoints > states in parallel as well. From my local testing, with default settings for > ioExecutor thread pool for xK state files this can reduce clean up time from > 10 minutes to <1 minute. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #23433: Update match_recognize.md , preterite of seek fix ,change seeked to sought
flinkbot commented on PR #23433: URL: https://github.com/apache/flink/pull/23433#issuecomment-1722685234 ## CI report: * e33b707b2ea7be3f44b55e5b3e3adef9974c95b8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] yanfangli85 opened a new pull request, #23434: Yanfangli85 patch 4
yanfangli85 opened a new pull request, #23434: URL: https://github.com/apache/flink/pull/23434 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] yanfangli85 opened a new pull request, #23433: Update match_recognize.md , preterite of seek fix ,change seeked to sought
yanfangli85 opened a new pull request, #23433: URL: https://github.com/apache/flink/pull/23433 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-32723) FLIP-334 : Decoupling autoscaler and kubernetes and support the Standalone Autoscaler
[ https://issues.apache.org/jira/browse/FLINK-32723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-32723: Summary: FLIP-334 : Decoupling autoscaler and kubernetes and support the Standalone Autoscaler (was: FLIP-334 : Decoupling autoscaler and kubernetes) > FLIP-334 : Decoupling autoscaler and kubernetes and support the Standalone > Autoscaler > - > > Key: FLINK-32723 > URL: https://issues.apache.org/jira/browse/FLINK-32723 > Project: Flink > Issue Type: New Feature > Components: Autoscaler, Kubernetes Operator >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > > This is an umbrella jira for decoupling autoscaler and kubernetes. > https://cwiki.apache.org/confluence/x/x4qzDw -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #23432: Update with.md defintion spell error ，fix with definition
flinkbot commented on PR #23432: URL: https://github.com/apache/flink/pull/23432#issuecomment-1722680234 ## CI report: * 54d5bd42aec689dfe8e9114ca99f5540dcf03bc8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-33097) Initialize the generic autoscaler module and interfaces
Rui Fan created FLINK-33097: --- Summary: Initialize the generic autoscaler module and interfaces Key: FLINK-33097 URL: https://issues.apache.org/jira/browse/FLINK-33097 Project: Flink Issue Type: Sub-task Reporter: Rui Fan Assignee: Rui Fan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32947) Autoscaler standalone mode supports the metric report
[ https://issues.apache.org/jira/browse/FLINK-32947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-32947: Parent: (was: FLINK-32723) Issue Type: New Feature (was: Sub-task) > Autoscaler standalone mode supports the metric report > - > > Key: FLINK-32947 > URL: https://issues.apache.org/jira/browse/FLINK-32947 > Project: Flink > Issue Type: New Feature > Components: Autoscaler >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > > FLIP-334 is decoupling the autoscaler and kubernetes, the autoscaler can run > with standalone mode after FLIP-334. > The flink kubernetes operator itself sets up the metrics reporters and > provides the base metric groups, for standalone implementation we need to > create a new metric reporter otherwise we cannot report the metrics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] yanfangli85 opened a new pull request, #23432: Update with.md defintion spell error ，fix with definition
yanfangli85 opened a new pull request, #23432: URL: https://github.com/apache/flink/pull/23432 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #23431: spell error : continous to continuous
flinkbot commented on PR #23431: URL: https://github.com/apache/flink/pull/23431#issuecomment-1722675574 ## CI report: * e6a1ca496563f7b382f58b96252140424a8bfe76 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22128) Window aggregation should have unique keys
[
https://issues.apache.org/jira/browse/FLINK-22128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766177#comment-17766177
]
Jingsong Lee commented on FLINK-22128:
--
[~337361...@qq.com] I think yes, this still should be done. But I don't have
time to research more details. CC [~lincoln.86xy]
> Window aggregation should have unique keys
> --
>
> Key: FLINK-22128
> URL: https://issues.apache.org/jira/browse/FLINK-22128
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Jingsong Lee
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> auto-unassigned
>
> We should add match method in {{FlinkRelMdUniqueKeys for
> StreamPhysicalWindowAggregate}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] yanfangli85 opened a new pull request, #23431: spell error : continous to continuous
yanfangli85 opened a new pull request, #23431: URL: https://github.com/apache/flink/pull/23431 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] elkhand commented on pull request #23406: [FLINK-32884] [flink-clients] PyFlink remote execution should support URLs with paths and https scheme
elkhand commented on PR #23406: URL: https://github.com/apache/flink/pull/23406#issuecomment-1722651735 > Tests are failing due to the new config options: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53293=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461 > > You need to follow the steps listed in the error message to generate the documentation for the config options. (line 32660) Thank you, fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21648) FLIP-151: Incremental snapshots for heap-based state backend
[ https://issues.apache.org/jira/browse/FLINK-21648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-21648: --- Labels: auto-deprioritized-major auto-unassigned stale-assigned (was: auto-deprioritized-major auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > FLIP-151: Incremental snapshots for heap-based state backend > > > Key: FLINK-21648 > URL: https://issues.apache.org/jira/browse/FLINK-21648 > Project: Flink > Issue Type: New Feature > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: auto-deprioritized-major, auto-unassigned, stale-assigned > Fix For: 1.18.0 > > > Umbrella ticket for > [https://cwiki.apache.org/confluence/display/FLINK/FLIP-151%3A+Incremental+snapshots+for+heap-based+state+backend] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32658) State should not be silently removed when ignore-unclaimed-state is false
[ https://issues.apache.org/jira/browse/FLINK-32658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32658: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > State should not be silently removed when ignore-unclaimed-state is false > - > > Key: FLINK-32658 > URL: https://issues.apache.org/jira/browse/FLINK-32658 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.18.0, 1.17.1 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > Labels: pull-request-available, stale-assigned > Attachments: screenshot-1.png > > > When ignore-unclaimed-state is false and the old state is removed, flink > should throw exception. It's similar to removing a stateful operator. > This case occurs not only when the user removes state, but also when the > operator is replaced. > For example: upgrade FlinkKafkaConsumer to KafkaSource. All logical are not > changed, so the operator id isn't changed. The KafkaSource cannot resume from > the state of FlinkKafkaConsumer. However, the new flink job can start, and > the state is silently removed in the new job.(The old state is not physically > discarded, it is still stored in the state backend, but the new code will > never use it.) > It also brings an additional problem: the KafkaSource will snapshot 2 states, > it includes the new state of KafkaSource, and the union list state of > FlinkKafkaConsumer. Whenever a job resumes from checkpoint, the union List > state is inflated. Eventually the state size of kafka offset exceeded 200MB. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30699) Improved getRandomString method code format in the StringUtils class
[ https://issues.apache.org/jira/browse/FLINK-30699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-30699: --- Labels: auto-deprioritized-minor pull-request-available (was: pull-request-available stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Improved getRandomString method code format in the StringUtils class > > > Key: FLINK-30699 > URL: https://issues.apache.org/jira/browse/FLINK-30699 > Project: Flink > Issue Type: Improvement > Components: API / Core >Reporter: Bingye Chen >Priority: Not a Priority > Labels: auto-deprioritized-minor, pull-request-available > Attachments: image-2023-01-16-18-13-56-912.png, > image-2023-01-16-18-14-12-939.png > > > > !image-2023-01-16-18-13-56-912.png|width=398,height=148! > !image-2023-01-16-18-14-12-939.png|width=398,height=114! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32575) Unified the Cpu of JobManager Name
[ https://issues.apache.org/jira/browse/FLINK-32575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32575: --- Labels: auto-deprioritized-major pull-request-available (was: pull-request-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Unified the Cpu of JobManager Name > -- > > Key: FLINK-32575 > URL: https://issues.apache.org/jira/browse/FLINK-32575 > Project: Flink > Issue Type: New Feature >Reporter: Bo Cui >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > To set the jm CPU of the Yarn, use the `yarn.appmaster.vcores`. To set the jm > cpu of the k8s, use `kubernetes.jobmanager.cpu`. when there are yarn and k8s > clusters, managing these configurations is difficult. Add a unified name for > them for ease of use. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32806) EmbeddedJobResultStore keeps the non-dirty job entries forever
[
https://issues.apache.org/jira/browse/FLINK-32806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32806:
---
Labels: stale-assigned starter (was: starter)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> EmbeddedJobResultStore keeps the non-dirty job entries forever
> --
>
> Key: FLINK-32806
> URL: https://issues.apache.org/jira/browse/FLINK-32806
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.18.0, 1.17.1, 1.19.0
>Reporter: Matthias Pohl
>Assignee: hk__lrzy
>Priority: Major
> Labels: stale-assigned, starter
>
> The {{EmbeddedJobResultStore}} keeps the entries of cleaned-up jobs in-memory
> forever. We might want to add a TTL to have those entries be removed after a
> certain amount of time to allow maintaining the memory footprint of the
> {{EmbeddedJobResultStore}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33058) Support for JSON-encoded Avro
[ https://issues.apache.org/jira/browse/FLINK-33058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766109#comment-17766109 ] Dale Lane commented on FLINK-33058: --- hi [~rskraba] - thanks very much, a review would be much appreciated. As for use cases, I should give some context. I work for IBM - we sell a Kafka distribution with a schema registry that comes with serdes clients offering both binary and JSON-encoded Avro support. As a part of this, I've worked with many customers who use and value JSON-encoding. As you suggest, sometimes this is a temporary thing, related to the phase of a project - I've seen some customers who will use JSON-encoding during development, and when they're ready to go into test/prod phases they flip the switch to binary-encoding. However, there have also been times where I've seen customers use JSON-encoding even in production - generally where the topic throughput is low enough that any performance issues are outweighed by the benefits of greater readability and compatibility that JSON-encoding offers. Don't get me wrong, I don't dispute at all that binary-encoding is the more common choice, and comes with major network and disk usage improvements - so it makes sense that Flink would've started with that. But I would love to enable my customers to use Flink with their JSON-encoded Avro topics in the same way that they're able to use other tools, which is what prompted me to offer the pull request. > Support for JSON-encoded Avro > - > > Key: FLINK-33058 > URL: https://issues.apache.org/jira/browse/FLINK-33058 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: Dale Lane >Priority: Minor > Labels: avro, flink, flink-formats, pull-request-available > > Avro supports two serialization encoding methods: binary and JSON > cf. [https://avro.apache.org/docs/1.11.1/specification/#encodings] > flink-avro currently has a hard-coded assumption that Avro data is > binary-encoded (and cannot process Avro data that has been JSON-encoded). > I propose adding a new optional format option to flink-avro: *avro.encoding* > It will support two options: 'binary' and 'json'. > It unset, it will default to 'binary' to maintain compatibility/consistency > with current behaviour. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] mxm commented on pull request #23406: [FLINK-32884] [flink-clients] PyFlink remote execution should support URLs with paths and https scheme
mxm commented on PR #23406: URL: https://github.com/apache/flink/pull/23406#issuecomment-1722527946 Tests are failing due to the new config options: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53293=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461 You need to follow the steps listed in the error message to generate the documentation for the config options. (line 32660) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ikstewa commented on pull request #8532: [FLINK-12541][REST] Support to submit Python Table API jobs via REST API
ikstewa commented on PR #8532: URL: https://github.com/apache/flink/pull/8532#issuecomment-1722511536 Is there still plans to support pyflink job submission via REST api? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-33096) Flink on k8s，if one taskmanager pod was crashed，the whole flink job will be failed
wawa created FLINK-33096: Summary: Flink on k8s，if one taskmanager pod was crashed，the whole flink job will be failed Key: FLINK-33096 URL: https://issues.apache.org/jira/browse/FLINK-33096 Project: Flink Issue Type: Bug Components: Deployment / Kubernetes Affects Versions: 1.14.3 Reporter: wawa The Flink version is 1.14.3, and the job is submitted to Kubernetes using the Native Kubernetes application mode. During the scheduling process, when a TaskManager pod crashes due to an exception, Kubernetes will attempt to start a new TaskManager pod. However, the scheduling process is halted immediately, resulting in the entire Flink job being terminated. On the other hand, if the JobManager pod crashes, Kubernetes is able to successfully schedule a new JobManager pod. This observation was made during application usage. Can you please help analyze the underlying issue? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] JunRuiLee commented on pull request #23430: test
JunRuiLee commented on PR #23430: URL: https://github.com/apache/flink/pull/23430#issuecomment-1722467291 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #23430: test
flinkbot commented on PR #23430: URL: https://github.com/apache/flink/pull/23430#issuecomment-1722403773 ## CI report: * 49789711a2baf37a3139e01ebe5fc6fe21b64712 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JunRuiLee opened a new pull request, #23430: test
JunRuiLee opened a new pull request, #23430: URL: https://github.com/apache/flink/pull/23430 run ci -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JunRuiLee closed pull request #23429: test
JunRuiLee closed pull request #23429: test URL: https://github.com/apache/flink/pull/23429 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
