Some very high level comments. I think it's perfectly fine to use CountDownLatch to wait for events, as is done in many jsr166 tests.
For the tricky problem of waiting for GC, see GcFinalization https://code.google.com/p/guava-libraries/source/browse/guava-testlib/src/com/google/common/testing/GcFinalization.java
