This is an automated email from the ASF dual-hosted git repository. gary pushed a change to branch FLINK-14651 in repository https://gitbox.apache.org/repos/asf/flink.git.
discard 639a9c2 [FLINK-14780][runtime] Register restart metrics only when start scheduling discard b596484 Improve test_streaming_bucketing discard 41a6c94 Improve Streaming File Sink end-to-end test discard af85012 fix test_queryable_state_restart_tm discard 3f7158b Ignore exception JobException: Recovery is suppressed by discard 445fe89 Ignore AbstractTaskManagerProcessFailureRecoveryTest.testTaskManagerProcessFailure discard 52d70d3 [FLINK-14651][runtime] Fix Kafka08ITCase#testFailOnNoBroker discard a9f4e7d [FLINK-14651][runtime] Fix SchedulerNGFactoryFactoryTest#createLegacySchedulerFactoryByDefault discard be4da97 [FLINK-14651][runtime] Fix KafkaConsumerTestBase#runFailOnNoBrokerTest discard a5f3292 [FLINK-14651][travis] Disable scheduler_ng Travis stages discard f1889bc [FLINK-14636][runtime] Remove NoOpFailoverStrategy discard baf1d0a [FLINK-14600][runtime] Enable DefaultScheduler by default discard 2c7052a Temporarily disable notifications from TravisCI add 3171edf [FLINK-14239] Fix the max watermark in StreamSource may arrive the downstream operator early add f9ba614 [hotfix][test] Fix some code styles in StreamSourceOperatorWatermarksTest and StreamSourceOperatorLatencyMetricsTest add 3f0f6f2 [FLINK-13729][docs] Update website generation dependencies add 135472e [FLINK-13725][docs] use sassc for faster doc generation add 8fc632d [hotfix][docs] Temporarily disable liveserve add cb7e904 [FLINK-13726][docs] Build docs with jekyll 4.0.0.pre.beta1 add a8868dd [FLINK-13791][docs] Speed up sidenav by using group_by add 84c96b3 [FLINK-14642] Add support for copying null values to the TupleSerializer and CaseClassSerializer add f074d1f [FLINK-14758] Add Executor-related interfaces + wire their discovery to environments. add a9b1266 [FLINK-14472][runtime] Implement back-pressure monitor with non-blocking outputs add 60f7aa0 [FLINK-14708][runtime] Introduce RestartAllStrategy for NG scheduler add 56c4664 [FLINK-14131][runtime] Introduce FailoverStrategyFactoryLoader to load NG failover strategy factories add 6abc6d9 [FLINK-14131][runtime] DefaultSchedulerFactory uses FailoverStrategyFactoryLoader to load failover strategy factory add 738cda3 [FLINK-14682][tests] Enable AbstractTaskManagerProcessFailureRecoveryTest to pass with new DefaultScheduler add 966c1f2 [FLINK-14680][runtime,tests] Enable KafkaConsumerTestBase#runFailOnNoBrokerTest to pass with DefaultScheduler add ed7e7b7 Temporarily disable notifications from TravisCI add eab85fc [FLINK-14600][runtime] Enable DefaultScheduler by default add 6b06800 [FLINK-14636][runtime] Remove NoOpFailoverStrategy add a11f501 [FLINK-14651][travis] Disable scheduler_ng Travis stages add 6439d61 [FLINK-14651][runtime] Fix SchedulerNGFactoryFactoryTest#createLegacySchedulerFactoryByDefault add 559aa58 Ignore exception JobException: Recovery is suppressed by add 6f17bbe fix test_queryable_state_restart_tm add 0517454 Improve Streaming File Sink end-to-end test add fb1aa63 Improve test_streaming_bucketing add 8c1235c [FLINK-14780][runtime] Register restart metrics only when start scheduling This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (639a9c2) \ N -- N -- N refs/heads/FLINK-14651 (8c1235c) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: docs/.gitignore | 3 +- docs/Gemfile | 19 +- docs/Gemfile.lock | 86 +++-- docs/README.md | 3 +- docs/_includes/generated/web_configuration.html | 4 +- docs/_includes/sidenav.html | 7 +- docs/build_docs.sh | 2 +- docs/fig/back_pressure_sampling.png | Bin 17635 -> 17525 bytes docs/monitoring/back_pressure.md | 18 +- docs/monitoring/back_pressure.zh.md | 22 +- .../connectors/kafka/KafkaConsumerTestBase.java | 15 +- .../java/typeutils/runtime/TupleSerializer.java | 8 + .../flink/configuration/ConfigConstants.java | 4 +- .../org/apache/flink/configuration/WebOptions.java | 8 +- .../execution/DefaultExecutorServiceLoader.java | 80 ++++ .../org/apache/flink/core/execution/Executor.java | 37 ++ .../flink/core/execution/ExecutorFactory.java | 40 ++ .../core/execution/ExecutorServiceLoader.java | 39 ++ .../flink/api/java/ExecutionEnvironment.java | 53 ++- .../flink/api/java/ExecutorDiscoveryTest.java | 85 +++++ ...org.apache.flink.core.execution.ExecutorFactory | 16 + .../flink/runtime/executiongraph/Execution.java | 28 +- .../flip1/FailoverStrategyFactoryLoader.java | 69 ++++ .../failover/flip1/RestartAllStrategy.java | 66 ++++ .../runtime/jobgraph/tasks/AbstractInvokable.java | 12 - .../jobmanager/slots/TaskManagerGateway.java | 26 +- .../jobmaster/JobManagerSharedServices.java | 31 +- .../runtime/jobmaster/RpcTaskManagerGateway.java | 17 +- ...Response.java => TaskBackPressureResponse.java} | 27 +- .../BackPressureRequestCoordinator.java | 369 ++++++++++++++++++ .../legacy/backpressure/BackPressureStats.java | 87 +++++ .../backpressure/BackPressureStatsTracker.java | 2 +- .../backpressure/BackPressureStatsTrackerImpl.java | 231 ++++------- .../backpressure/OperatorBackPressureStats.java | 68 +--- .../legacy/backpressure/StackTraceSample.java | 119 ------ .../backpressure/StackTraceSampleCoordinator.java | 392 ------------------- .../flink/runtime/scheduler/DefaultScheduler.java | 7 +- .../runtime/scheduler/DefaultSchedulerFactory.java | 4 +- .../taskexecutor/BackPressureSampleService.java | 111 ++++++ ...leTask.java => BackPressureSampleableTask.java} | 10 +- .../taskexecutor/StackTraceSampleService.java | 128 ------- .../flink/runtime/taskexecutor/TaskExecutor.java | 38 +- .../runtime/taskexecutor/TaskExecutorGateway.java | 9 +- .../runtime/taskexecutor/TaskManagerRunner.java | 14 +- .../TaskStackTraceSampleableTaskAdapter.java | 58 --- .../org/apache/flink/runtime/taskmanager/Task.java | 26 +- .../executiongraph/ExecutionJobVertexTest.java | 26 +- .../flip1/FailoverStrategyFactoryLoaderTest.java | 72 ++++ .../failover/flip1/RestartAllStrategyTest.java | 55 +++ .../utils/SimpleAckingTaskManagerGateway.java | 9 +- .../TestingJobManagerSharedServicesBuilder.java | 14 +- .../BackPressureRequestCoordinatorTest.java | 343 +++++++++++++++++ .../BackPressureStatsTrackerImplTest.java | 293 ++++++++------ .../StackTraceSampleCoordinatorTest.java | 421 --------------------- .../BackPressureSampleServiceTest.java | 123 ++++++ .../taskexecutor/StackTraceSampleServiceTest.java | 189 --------- .../TaskExecutorPartitionLifecycleTest.java | 8 +- .../taskexecutor/TaskExecutorSubmissionTest.java | 154 +++----- .../runtime/taskexecutor/TaskExecutorTest.java | 6 +- .../TaskSubmissionTestEnvironment.java | 4 +- .../runtime/taskexecutor/TestingTaskExecutor.java | 6 +- .../taskexecutor/TestingTaskExecutorGateway.java | 11 +- .../apache/flink/runtime/taskmanager/TaskTest.java | 5 +- .../runtime/testtasks/OutputBlockedInvokable.java | 47 +++ .../api/scala/typeutils/CaseClassSerializer.scala | 17 +- .../api/environment/LocalStreamEnvironment.java | 4 - .../environment/StreamExecutionEnvironment.java | 54 ++- .../streaming/api/operators/StreamSource.java | 19 +- .../streaming/runtime/tasks/SourceStreamTask.java | 6 - .../environment/ExecutorDiscoveryTest.java | 86 +++++ .../StreamSourceOperatorLatencyMetricsTest.java | 9 +- .../StreamSourceOperatorWatermarksTest.java | 191 ++++++---- ...org.apache.flink.core.execution.ExecutorFactory | 16 + ...tractTaskManagerProcessFailureRecoveryTest.java | 3 +- 74 files changed, 2557 insertions(+), 2132 deletions(-) create mode 100644 flink-core/src/main/java/org/apache/flink/core/execution/DefaultExecutorServiceLoader.java create mode 100644 flink-core/src/main/java/org/apache/flink/core/execution/Executor.java create mode 100644 flink-core/src/main/java/org/apache/flink/core/execution/ExecutorFactory.java create mode 100644 flink-core/src/main/java/org/apache/flink/core/execution/ExecutorServiceLoader.java create mode 100644 flink-java/src/test/java/org/apache/flink/api/java/ExecutorDiscoveryTest.java create mode 100644 flink-java/src/test/resources/META-INF/services/org.apache.flink.core.execution.ExecutorFactory create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/failover/flip1/FailoverStrategyFactoryLoader.java create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/failover/flip1/RestartAllStrategy.java rename flink-runtime/src/main/java/org/apache/flink/runtime/messages/{StackTraceSampleResponse.java => TaskBackPressureResponse.java} (72%) create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/backpressure/BackPressureRequestCoordinator.java create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/backpressure/BackPressureStats.java delete mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/backpressure/StackTraceSample.java delete mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/backpressure/StackTraceSampleCoordinator.java create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/BackPressureSampleService.java rename flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/{StackTraceSampleableTask.java => BackPressureSampleableTask.java} (76%) delete mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/StackTraceSampleService.java delete mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskStackTraceSampleableTaskAdapter.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/failover/flip1/FailoverStrategyFactoryLoaderTest.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/failover/flip1/RestartAllStrategyTest.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/legacy/backpressure/BackPressureRequestCoordinatorTest.java delete mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/legacy/backpressure/StackTraceSampleCoordinatorTest.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/BackPressureSampleServiceTest.java delete mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/StackTraceSampleServiceTest.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/testtasks/OutputBlockedInvokable.java create mode 100644 flink-streaming-java/src/test/java/org/apache/flink/streaming/environment/ExecutorDiscoveryTest.java create mode 100644 flink-streaming-java/src/test/resources/META-INF/services/org.apache.flink.core.execution.ExecutorFactory