Hello, Thanks for responding. I have used all default settings on the Kafka broker as well as the camel Kafka endpoint.
I debugged the issue a little bit further, I noticed that in the kafkaConsumer.java <https://github.com/apache/kafka/blob/62abe01bee0396515253373e906253bd0e081cb9/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1237> (2.6.0) line 1237 has a do-while loop. When the poll for fetched records returns empty, the do-while loop gets executed again and again hogging the CPU. Even adding a log message in the do-while loop during debugging brought the CPU usage down because of the file IO operation during logging. I didn't try to create a reproducer but noticed that when a simple java consumer is started, I did observe high CPU usage. On Thu, Jun 30, 2022 at 5:07 PM Otavio Rodolfo Piske <angusyo...@gmail.com> wrote: > Hi, I'd like to take a closer look at this, but I need more details. > > Are you able to come up with a reproducer for this issue? Additionally, > details about your Kafka setup would be appreciated: # of partitions, any > broker setting you might have, etc. > > Kind regards > > On Sun, Jun 26, 2022 at 3:24 PM Rohan Emmanuel <rohan.emmanue...@gmail.com > > > wrote: > > > hello Claus, > > Thank you for responding. i tried with camel 3.17.0 and kafka clients > 3.2.0 > > , but I still see the CPU busy, The process is busy at 99% even i am not > > sending my messages to the topics. > > > > Looks like Kafka clients calls(polling) are doing CPU-intensive > operations > > repeatedly. Any configuration which can reduce the number of Kafka > Clients > > calls? > > > > > > > > Attached below is the Real Time process Monitor output, $sr10 & $sr11 > > running on CPUs 2 & 3 respectively, are my processes running Kafka > > endpoint. > > > > > > > > Cpus Cp hh:mm ss Busy Sec QLen Disp Disk Chit Swap MLock Pcb > PcbX > > -------- -- -------- ---% --- ---- ------ ----- ------ ---- ----% --- > ---- > > 0 18:57:29 10 998 1 > 12.88 > > 29 359 > > 1 18:57:29 10 843 12.43 > > 21 237 > > * 2 *18:57:29 *18 * 10 1 1191 326 > > 11.27 12 276 > > * 3 *18:57:29 * 17 *10 1 1168 > > 11.11 10 465 > > 4 18:57:29 10 821 6.15 > 12 > > 149 > > 5 18:57:29 10 832 6.24 > > 9 153 > > 6 18:57:29 10 805 7.79 > 9 > > 141 > > Process Cpu,Pin Pri Busy% Name RPM T0877(16SEP16) ET=10.1 Top User > > -------- ------- --- ----- -------- -------------------------- --- > ------- > > 3,612 170 *99.62* *$SR11 * /jdk110_l11/bin/java > 1 > > 201,4 > > 2,1073 170 * 98.59 $SR10* /jdk110_l11/bin/java 2 > > 201,4 > > 2,1076 170 .99 /jdk110_l11/bin/javahelper 3 > 201,4 > > 2,1077 170 .99 /jdk110_l11/bin/javahelper 4 > 201,4 > > 2,1074 170 .98 /jdk110_l11/bin/javahelper 5 > 201,4 > > > > > > > > The route looks like : > > > > <route id="kafka_endpoint-kafka"> > > <from > > > > > uri="kafka:kavi-reqs?brokers=<IP>:9092&fetchMinBytes=20480&reconnectBackoffMaxMs=10000&reconnectBackoffMs=2000&groupId=APIG"/> > > <doTry> > > <process > ref="api.gway.kafka_endpoint.kafka.request"/> > > <to uri="custom_component"/> > > > > <doCatch > > > <exception>java.lang.Exception</exception> > > > > > > <!--setBody--> > > <transform> > > <simple>Error reported: ${exception.message} - cannot process this > message. > > </simple> > > > > </transform> > > <!--/setBody--> > > > > <log message="${exception.stacktrace}" /> > > > > > > </doCatch> > > </doTry> > > > > > > > > On Sat, Jun 25, 2022 at 6:40 PM Claus Ibsen <claus.ib...@gmail.com> > wrote: > > > > > Hi > > > > > > Are you able to try with a newer Camel version as we have improved > > > camel-kafka a lot over the last couple of years. > > > > > > > > > > > > > > > On Fri, Jun 24, 2022 at 4:06 PM Rohan Emmanuel < > > rohan.emmanue...@gmail.com > > > > > > > wrote: > > > > > > > hi, > > > > i can using camel 3.7.0 with kafka consumer endpoint with default > > > > configurations. when I start my process, even though i am not sending > > any > > > > messages to the topic, i see high CPU utilization by the java > process. > > by > > > > looking at JMeter for the utilization, the KafkaConsumer poll is > > > resulting > > > > in a lot number of calls internally. > > > > Tried different configurations of fetchMinBytes ,reconnectBackoffMs > & > > > > reconnectBackoffMaxMs but these did not alter the HIGH CPU > utilization. > > > > > > > > Any help on what configuration can be done to avoid high CPU > > utilization? > > > > any pointers would be appreciated. > > > > > > > > -- > > > > Regards, > > > > Rohan Emmanuel > > > > > > > > > > > > > -- > > > Claus Ibsen > > > ----------------- > > > http://davsclaus.com @davsclaus > > > Camel in Action 2: https://www.manning.com/ibsen2 > > > > > > > > > -- > > Regards, > > Rohan Emmanuel > > > > > -- > Otavio R. Piske > http://orpiske.net > -- Regards, Rohan Emmanuel
