GitHub user shanthoosh opened a pull request:
https://github.com/apache/samza/pull/515
SAMZA-1711: Re-enable existing standalone integration tests.
**Changes:**
* Enable all existing standalone integration tests except
`TestZkStreamProcessorSession`(`TestZkStreamProcessorSession` is flaky. It
spawns `x` StreamProcessors and kills one StreamProcessor through zookeeper
session expiration. Sleeps for 5 seconds and proceeds to do validation. If the
rebalancing phase takes longer the sleep time, validation fails).
* Remove zookeeper unavailable unit test from LocalApplicationRunner(Race
condition in zookeeper shutdown fails other tests). The deleted test will be
added back in a separate test class.
* Increase zookeeper server minimum session timeout from 6 seconds to 120
seconds.
* Add assertions to validate if kafka topics setup were successful before
the unit tests.
**Validation:**
Verified by running the following script on top of this patch.
```bash
i=0
while [ $i -lt 50 ]; do
i=`expr $i + 1`
echo "Run " +$i
./gradlew clean :samza-test:test
-Dtest.single="TestZkLocalApplicationRunner" --debug --stacktrace >>
~/test-logs_10
./gradlew clean :samza-test:test -Dtest.single="TestZkStreamProcessor"
--debug --stacktrace >> ~/test-logs_10
./gradlew clean :samza-test:test -Dtest.single="TestStreamProcessor"
--debug --stacktrace >> ~/test-logs_10
./gradlew clean :samza-test:test
-Dtest.single="TestZkStreamProcessorFailures" --debug --stacktrace >>
~/test-logs_10
done;
```
**Result:**
```bash
[svenkata@svenkata-ld2 samza]$ grep 'BUILD SUCCESS' ~/test-logs_10 | wc -l
200
[svenkata@svenkata-ld2 samza]$ grep 'BUILD FAIL' ~/test-logs_10 | wc -l
0
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shanthoosh/samza turn_all_integration_tests_on
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/515.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #515
----
commit 549efb7876fbad9ca3789eb68cdb73e495627b4f
Author: Shanthoosh Venkataraman <svenkataraman@...>
Date: 2018-04-30T04:28:14Z
SAMZA-1692: Standalone stability fixes.
- Currently, on session expiration processorListener with incorrect
generationId is registered with zookeeper(ZkUtils generationId is incremented
on reconnect but the generationId in processorListener is zero all the time).
When a session reconnect happens to a processor successive to leader, leader
expiration event will be skipped. This will prevent leader re-election on a
current leader death and will stall the processors group. Fix is to
reinstantiate and then register processorChangeListener on session expiration.
- Add processorId to debounce thread name (this can aid debugging when
multiple processors are running within a jvm).
- After ScheduleAfterDebounceTime queue is shutdown, don't accept new
schedule requests. Current ZkJobCoordinator shutdown sequence comprise of the
following steps
- Shutdown the ScheduleAfterDebounceTime queue.
- Stop the zkClient and relinquish it's resources.
After we shutdown ScheduleAfterDebounceTime and before zkclient is stopped,
any new operations can be scheduled in ScheduleAfterDebounceTime queue. This
will result in RejectedExecutionException, since executorService is stopped.
```
Caused by: java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@23f962a8
rejected from java.util.concurrent.ScheduledThreadPoolExecutor@43408be8
```
commit 6a07a960bbb2baf5d3149d0384ba5e98ecfd82fc
Author: Shanthoosh Venkataraman <santhoshvenkat1988@...>
Date: 2018-05-07T20:23:58Z
Review comments.
commit 5ea977cf7b469653c2a588abeb3f46c344cfccfc
Author: Shanthoosh Venkataraman <santhoshvenkat1988@...>
Date: 2018-05-09T23:30:48Z
Reenable all disabled standalone integration tests.
----
---
