Hello Eric, 1) The rebalance failures is mainly on ZK session timeout, you could try to increase your zk session timeout value and see if that helps.
2) The new consumer in 0.9 re-write will resolve this problem by getting rid of the ZK dependency and use a centralized coordinator for rebalance logic. Guozhang On Wed, Oct 22, 2014 at 9:32 AM, Eric Bottard <ebott...@pivotal.io> wrote: > For the purpose of a unit/integration test for Spring XD, I am creating > several consumers (in the same group) in quick succession. With just 3 > consumers, this triggers 2 rebalances that (on some machines) can't be > dealt with in the default 4*2000ms and fails. > > I have created a simple use case that reproduces this out of the context of > Spring XD. It can be found in the main() method of [1]. If it does not fail > on your machine, I believe bumping the number of creations to 4 or 5 may do > it.. > > So, I have a couple of questions: > - on my machine, when using various combinations of rebalance.backoff.ms > and rebalance.max.retries, this failure always seems to happen after 30s, > whatever the combination. On some other (more powerful) machines, it seems > to never fail. Is this actually cpu bound? 30s sounds a lot like twice the > default tick, so is this at all related to an ephemeral node timing out? > - given that this is for the purposes of an integration test, is there any > other parameter that I could tweak to have the system settle down faster? > - Is this something that is likely to change with the 0.9 rewrite (I saw > that the current de-centralized rebalancing mechanics are the cause of > other issues)? > > As a workaround, I tried waiting for nodes to re-appear in ZooKeeper (code > at [2]), but this is still very slow (not to mention that this would be > very intrusive to the tests I'm trying to write) > > Lastly, I should mention that I do need to create 3 consumers sequentially > (as opposed to say, use a 3 threads consumer). The test in question simply > happens to mimic the creation of consumers that may well be on 3 separate > machines > > Best, > > [1] https://gist.github.com/ericbottard/91aa9ee114c6091e5b7b > [2] https://gist.github.com/ericbottard/de87f1edc8eee2b9bee5 > > -- > Eric Bottard > -- -- Guozhang
