Hello Viraj, Looks like you need to run profiler. Does this page help? Dynamic Worker Profiling
| | | | Dynamic Worker Profiling | | | --Bipin On Monday, August 28, 2023 at 06:34:41 AM PDT, Aishwarya Soni <aishwarya.ash...@gmail.com> wrote: I also observed that we do need more CPU power since 2.x versions. Try increasing the supervisor CPU and its service memory/RAM. Since you are giving 20Gb XmX for workers, we need supervisors to be in a much beefier state to provide that heap. We are a streaming app and for us, the supervisor CPU utilization can go close to 16 CPU cores+. On Mon, Aug 28, 2023 at 5:13 PM v.s kadu <virajkadu...@gmail.com> wrote: Hi All, I have figured out a way to deal with this. Just need to increase the parallelism hint of the component which is taking longer to process or it's Capacity is near to one in storm ui. This helped me to generate files faster. However another question which I have is regarding CPU utilization. I have executed my topologies with storm 0.9.6 and 2.4.0.Storm 0.9.6 uses ~30 % of 16 core RHEL 8 server to process. However storm 2.4.0 uses ~65% to process the same . The difference is huge. Please note we are emitting 1000 records from Spout and this is coded in java. Storm configurations used in both versions topology.max.spout.pending:200topology.message.timeout.secs:300storm.messaging transport:"backtype.storm.messaging.netty.Context"nimbus.task.timeout.secs:240nimbus.thrift.max_buffer_size:20480000nimbus.childopts:"-Xmx2048m -Djava.net.preferIPv4Stack=true" worker. childopts:"-Xmx20g -XX:+UseG1GC -Djava.net.preferIPv4Stack=true" supervisor.childopts:"-Xmx2048m -Djava.net.preferIPv4Stack=true" How we could trace why storm 2.4.0 is taking such high CPU utilization? Any configuration changes needed for storm 2.4.0 while migrating from 0.9.6? Regards, Viraj Kadu On Thu, 24 Aug, 2023, 1:49 am v.s kadu, <virajkadu...@gmail.com> wrote: Hello Team, I am migrating to apache storm 2.4.0 from apache storm 0.9.6 But the performance in apache storm 2.4.0 is slower. I read that wait strategy plays role in performance Exiting application on apache storm 0.9.6 has following wait strategy configuration topology.spout.wait.strategy: backtype.storm.spout.SleepWaitStrategy topology.disruptor.wait.strategy: com.lmax.disruptor.BlockingWaitStrategy topology.disruptor.wait.timeout.millis: 1000 topology.sleep.spout.wait.strategy.time.ms: 1 Tried tweaking wait strategy parameter in apache storm 2.4.0. Following tests i had performed 1. Kept all defaults as it is 2. topology.bolt.wait.progressive.level1.count:0 topology.bolt.wait.progressive.level2.count:0 topology.bolt.wait.progressive.level3.sleep.millis:0 topology.backpressure.wait.progressive.level1.count:0 topology.backpressure.wait.progressive.level2.count:0 topology.backpressure.wait.progressive.level3.sleep.millis:0 3. topology.spout.wait.strategy:org.apache.storm.policy.WaitStrategy topology.backpressure.wait.strategy:org.apache.storm.policy.WaitStrategy topology.bolt.wait.strategy:org.apache.storm.policy.WaitStrategy topology.bolt.wait.microsec:0 But performance was lot slower in all scenarios. What other factors we can check for performance testing while migrating? How to improve performance in apache storm 2.4.0? Regards,Viraj Kadu