Pramod Immaneni created APEXMALHAR-2518:
-------------------------------------------
Summary: Kafka input operator stops reading tuples when there is a
UNKNOWN_MEMBER_ID error during committed offset processing
Key: APEXMALHAR-2518
URL: https://issues.apache.org/jira/browse/APEXMALHAR-2518
Project: Apache Apex Malhar
Issue Type: Bug
Reporter: Pramod Immaneni
Assignee: Pramod Immaneni
Kafka 0.9 operator stores offsets that are completely processed and no longer
needed (committed offsets) back in kafka. It does so by making a kafka API
call. If the response from kafka server to this call comes back with an
UNKNOWN_MEMBER_ID error, it results in the kafka consumer state changing to
needing partition re-assignment and no further messages are returned by the
consumer. There are a couple of other errors that result in the same state
including when rebalance is in progress.
What exactly caused this error is not known but the following is the likely
reason due to the conditions surrounding the application. When the operator has
temporarily stalled due to back-pressure exerted by the slow downstream, it
will eventually stall the operator kafka consumer thread that is reading
messages from kafka. This will result in the thread not making any kafka
consumer API calls and it will result in no heartbeats being sent to kafka
server. This can cause the server to evict the consumer after a timeout period.
This could have been the cause for the UNKNOWN_MEMBER_ID error.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
