[ https://issues.apache.org/jira/browse/FLINK-8001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16241927#comment-16241927 ]
ASF GitHub Bot commented on FLINK-8001: --------------------------------------- GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/4969 (release-1.3) [FLINK-8001] [kafka] Mark subtask as IDLE instead of emitting Long.MAX_VALUE watermark ## What is the purpose of the change This is a backport of #4967 to `release-1.3`. The additional change, compared to #4967, is that instead of emitting the `Long.MAX_VALUE` watermark when a subtask has no Kafka partitions, `sourceContext.markAsTemporarilyIdle()` is used instead. ## Brief change log Only contains one commit, which addresses the above described issue. ## Verifying this change A new test is added to AbstractFetcherTest, that verifies when there is no subscribed partitions and the periodic watermark emitter fires, no watermark is emitted. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? n/a You can merge this pull request into a Git repository by running: $ git pull https://github.com/tzulitai/flink FLINK-8001-1.3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4969.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4969 ---- commit 85fac04706937c3ad096da60609f264bd4355f55 Author: Tzu-Li (Gordon) Tai <tzuli...@apache.org> Date: 2017-11-07T12:12:23Z [FLINK-8001] [kafka] Mark consumer subtask as IDLE if it doesn't have partitions ---- > Mark Kafka Consumer as idle if it doesn't have partitions > --------------------------------------------------------- > > Key: FLINK-8001 > URL: https://issues.apache.org/jira/browse/FLINK-8001 > Project: Flink > Issue Type: Bug > Reporter: Aljoscha Krettek > Assignee: Tzu-Li (Gordon) Tai > Priority: Blocker > Fix For: 1.4.0, 1.3.3 > > > In Flink 1.3.x the Kafka Consumer will emit a {{Long.MAX_VALUE}} watermark if > it has zero partitions assigned. If this happens and other parallel instances > of the Kafka Consumer are marked as idle (which currently never happens by > default but does happen in custom forks of our Kafka code) this means that > the watermark jumps to {{Long.MAX_VALUE}} downstream. > In Flink 1.4.x this happens implicitly in the {{PeriodicWatermarkEmitter}} in > {{AbstractFetcher}} where the watermark is {{Long.MAX_VALUE}} if we don't > have any partitions. This should be changed to mark the source as idle > instead, if we don't have any partitions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)