[
https://issues.apache.org/jira/browse/TWILL-204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Terence Yim updated TWILL-204:
------------------------------
Fix Version/s: (was: 0.12.0)
> AM should keep Kafka broker up until log consumers finish
> ---------------------------------------------------------
>
> Key: TWILL-204
> URL: https://issues.apache.org/jira/browse/TWILL-204
> Project: Apache Twill
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.9.0, 0.10.0, 0.11.0
> Reporter: Martin Serrano
>
> I'm encountering a situation on a fast machine where the Kafka log
> aggregation topic is not empty when the system shuts down. The scenario:
> log consumer consumes all messages
> consumer sleeps (500ms) due to empty queue
> containers exit, posting final log messages about why
> controller notices containers are down and terminates consumers.
> consumer is interrupted from sleep and but has been canceled so it does
> not get the rest of the messages.
> This scenario can be really confusing during development because an error may
> be missed (as in my case) if it falls into the final log messages. Before I
> file a ticket and fix this, I wanted to get some feedback. Looking at
> org.apache.twill.internal.kafka.client.SimpleKafkaConsumer it seems this
> behavior could be intentional given this log message (line 384):
> LOG.debug("Unable to fetch messages on {}, kafka consumer service
> shutdown is in progress.", topicPart);
> My opinion is that final messages logged by a container are likely to be
> critical in diagnosing errors and that twill should do whatever it can to
> forward them before shutting things down.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
