I'm not sure that KafkaSpout rebalancing blocks spout thread, but if it is, it shouldn't be.
In fact any of Spout methods shouldn't be blocked longer to ensure message timeout works properly, since spout event handling (including calling nextTuple()) is done in single thread. For general solution for this issue, we may be better to adopt multi-threads (actually two, for calling nextTuple(), and for handling ack/fail messages) on Spout from JStorm, but then any existing Spout implementation should ensure thread-safety, which most of Spout implementations need to be reimplemented. I would also like to hear opinions about this symptom from anyone familiar with storm-kafka-client. - Jungtaek Lim (HeartSaVioR) 2017년 3월 23일 (목) 오후 4:33, Antti Järvinen <[email protected]>님이 작성: > Hi, > > opened ticket https://issues.apache.org/jira/browse/STORM-2426 about > behaviour which causes first tuples to be emitted by Kafka Spout to fail > after startup with long rebalancing period. Storm version 1.0.2. > > It seems that the system tick tuples that should be received every 30 > seconds are delayed while the KafkaSpout is rebalancing and then received > inside short time period when then rebalancing ends. This causes the > RotatingMap to rotate multiple times in short time that causes tuples that > were just emitted to fail on timeout. > > Did not find the actual piece of code that causes the tuples to be delayed. > Is this wanted/expected behaviour of the SpoutExecutor or could the delayed > ticks be somehow ignored? > > I'm willing to work on issue if there is something to be changed but need > some handholding on why the delaying happens. > > -- > Antti Järvinen > Alupark Oy > 044 5783 084 > [email protected] >
