GitHub user shanthoosh opened a pull request:
https://github.com/apache/samza/pull/499
SAMZA-1695: Clear events in ScheduleAfterDebounceTime on session expiration
Scenario:
Let's assume there're three processors in the group [P1, P2, P3] and P1 is
the leader.
1. Leader processor(P1) loses connectivity with a zookeeper server in the
ensemble and it's ephemeral processor node is deleted(due to session
expiration).
2. Immediate successor(P2) to the leader(P1) finds out that the leader is
dead and declares itself as leader. Processor P2 Schedules onProcessorChange to
publish JobModel.
3. ZkClient connection retry logic helps the Leader(P1) to reconnect to
another zkServer in the ensemble and it joins as follower.
4. Processor P1 acts on the stale buffered event in the debounce
queue(which it received when it's a leader) and acts as leader. At this point,
there're two processors acting as leader(P1 & P2). If P1 proceeds to execute
leader actions before P2, P2 will fail(and in worst case can cause state
corruption).
Sample exception logs:
https://gist.github.com/shanthoosh/55410fe4ebf3cfb65281b35f16397cad
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shanthoosh/samza
remove_events_from_debounce_queue_on_session_expiry
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/499.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #499
----
commit f83c1203bc042fcef4ebcd28b8c517cd5badbe7d
Author: Shanthoosh Venkataraman <santhoshvenkat1988@...>
Date: 2018-04-30T19:12:39Z
SAMZA-1695: Clear events in ScheduleAfterDebounceTime on session expiration.
----
---
