Hi All, I have figured out a way to deal with this. Just need to increase the parallelism hint of the component which is taking longer to process or it's Capacity is near to one in storm ui. This helped me to generate files faster.
However another question which I have is regarding CPU utilization. I have executed my topologies with storm 0.9.6 and 2.4.0. Storm 0.9.6 uses ~30 % of 16 core RHEL 8 server to process. However storm 2.4.0 uses ~65% to process the same . The difference is huge. Please note we are emitting 1000 records from Spout and this is coded in java. Storm configurations used in both versions topology.max.spout.pending:200 topology.message.timeout.secs:300 storm.messaging transport:"backtype.storm.messaging.netty.Context" nimbus.task.timeout.secs:240 nimbus.thrift.max_buffer_size:20480000 nimbus.childopts:"-Xmx2048m -Djava.net.preferIPv4Stack=true" worker. childopts:"-Xmx20g -XX:+UseG1GC -Djava.net.preferIPv4Stack=true" supervisor.childopts:"-Xmx2048m -Djava.net.preferIPv4Stack=true" How we could trace why storm 2.4.0 is taking such high CPU utilization? Any configuration changes needed for storm 2.4.0 while migrating from 0.9.6? Regards, Viraj Kadu On Thu, 24 Aug, 2023, 1:49 am v.s kadu, <virajkadu...@gmail.com> wrote: > Hello Team, > > I am migrating to apache storm 2.4.0 from apache storm 0.9.6 But the > performance in apache storm 2.4.0 is slower. > > I read that wait strategy plays role in performance Exiting application on > apache storm 0.9.6 has following wait strategy configuration > > topology.spout.wait.strategy: backtype.storm.spout.SleepWaitStrategy > topology.disruptor.wait.strategy: com.lmax.disruptor.BlockingWaitStrategy > topology.disruptor.wait.timeout.millis: 1000 > topology.sleep.spout.wait.strategy.time.ms: 1 > > Tried tweaking wait strategy parameter in apache storm 2.4.0. Following > tests i had performed > 1. > Kept all defaults as it is > > 2. > topology.bolt.wait.progressive.level1.count:0 > topology.bolt.wait.progressive.level2.count:0 > topology.bolt.wait.progressive.level3.sleep.millis:0 > topology.backpressure.wait.progressive.level1.count:0 > topology.backpressure.wait.progressive.level2.count:0 > topology.backpressure.wait.progressive.level3.sleep.millis:0 > > 3. > topology.spout.wait.strategy:org.apache.storm.policy.WaitStrategy > topology.backpressure.wait.strategy:org.apache.storm.policy.WaitStrategy > topology.bolt.wait.strategy:org.apache.storm.policy.WaitStrategy > topology.bolt.wait.microsec:0 > > But performance was lot slower in all scenarios. What other factors we can > check for performance testing while migrating? How to improve performance > in apache storm 2.4.0? > > Regards, > Viraj Kadu >
