Hello,
I'm writing a storm topology with a multilang bolt and see that worker is
crashing consistently. My data flow is like this,
KafkaSpout -> KafkaDecoderBolt ->
> RankingBolt(Multilang) -> ...
Problem seems to be with the KafkaDecoderBolt -> RankingBolt pipeline.
Only way I'm able to avoid the crash is by placing a small sleep of 30ms at
the end of KafkaDecoderBolt. This suggests that KafkaDecoderBolt is writing
way faster that multilang bolt is able to process.
This is indeed true. Multilang bolt takes ~100ms to process and
KafkaDecoderBolt takes just 2ms. I'm trying to understand which queue I'm
blowing up.
I tried keeping max outstanding tuple to low value hoping that spout would
throttle if multilang bolt is not able to ack fast enough. That didn't seem
to help.
> config.setMaxSpoutPending(50);
I also tried increasing executor buffer but that didn't seem to help even a
small bit. Didn't change network receiver/sender buffer as my tests were on
a single worker.
> config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 16384*2);
> config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 16384*2);
> config.put(Config.TOPOLOGY_WORKER_CHILDOPTS, "-Xmx4g -Xms1024m");
I know I need higher executors for multilang bolt but looks like I need one
more config to tweak. Not sure which one.
Below is the exception and logging at DEBUG level.
2015-04-21T15:35:24.954-0400 b.s.d.task [INFO] Emitting: RankingBolt
> __ack_ack [-5027869909211355112 3891537668110238939]
> 2015-04-21T15:35:24.954-0400 b.s.d.executor [INFO] Processing received
> message source: RankingBolt:4, stream: __ack_ack, id: {},
> [-5027869909211355112 3891537668110238939]
> 2015-04-21T15:35:24.954-0400 b.s.t.ShellBolt [INFO] ShellLog pid:26767,
> name:RankingBolt Processed = 1499
> 2015-04-21T15:35:24.955-0400 b.s.t.ShellBolt [ERROR] Halting process:
> ShellBolt died.
> java.lang.RuntimeException: backtype.storm.multilang.NoOutputException:
> Pipe to subprocess seems to be broken! No output read.
> Serializer Exception:
> Execution halted
> at
> backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:101)
> ~[storm-core-0.9.4.jar:0.9.4]
> at
> backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318)
> ~[storm-core-0.9.4.jar:0.9.4]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
Srikanth
