We have a topology which consists of a KafkaSpout and a persistence bolt
which is used
to write in a memory database and then forward tuples to other bolts. The
issue we are facing is that the topology does not scale as expected.
Specifically when we scale our topology using 2 machines - 2 workers(8
cores - 16 parallesism units) everything seems to scale normally. We use 2
KafkaSpouts which distribute tuples to 2 persistence bolts almost evenly
(we use field grouping based on device id between spouts and bolts).
But our problem starts when we scale to 4 machines - 4 workers. Then even
though the 4 KafkaSpouts emit almost the same number of tuples those are
not distributed to the 4 persistence bolts. In fact half of those emitted
tuples result in only 1 bolt and another bolt does not get any tuples (the
2 other get a small number o f tuples).
My question is why this is happening ? It is getting much worse if we
increase the number of executors for the persistence bolts (e.g. 8
executors result only half of the bolts get any tuples), but at the same
point when we increase the number of tasks to a large number 64 for example
then every bolt gets some tuples. Is this the normal behavior ? How can we
resolve this issue ?
