Mideen Abdul Gaffoor created IGNITE-16497: ---------------------------------------------
Summary: Data streaming in Apache-Ignite through Apache-Kafka consumes high CPU Key: IGNITE-16497 URL: https://issues.apache.org/jira/browse/IGNITE-16497 Project: Ignite Issue Type: Bug Reporter: Mideen Abdul Gaffoor Attachments: main idc cpu consumption issue.png I am using Ignite Source Connector (Single Kafka connector node) to export the Ignite Events from my ignite cluster(2 Nodes) to Kafka broker nodes (2 Nodes) with a single topic and 29 partitions. I have processed 0.1 Million events (PUT, DELETE) per minute with the 1k size messages(in Avg). My connector node consumes 90% of the CPU (you can see it below). !main idc cpu consumption issue.png! My Connector node machine config. RAM --> 30 GB ROM --> 1TB Configured Heap --> 15GB No of CPU Core --> 40 (20 x 2) # connector name=my-ignite-source-connector connector.class=class of my source connector tasks.max=1 topicNames=ignite-data # cache cacheName=DOCIDS cacheAllowOverwrite=true cacheEvts=put , removed evtBatchSize=100 numberOfPartitions=29 igniteCfg=/myconfig/ignite-config.xml <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:util="http://www.springframework.org/schema/util" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd"> <bean abstract="true" id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration"> <!-- Set to true to enable distributed class loading for examples, default is false. --> <property name="communicationSpi"><bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"><property name="socketWriteTimeout" value="60000"/></bean></property><property name="peerClassLoadingEnabled" value="true"/><property name="clientFailureDetectionTimeout" value="10000"/><property name="dataStorageConfiguration"><bean class="org.apache.ignite.configuration.DataStorageConfiguration"><property name="walPath" value="wal_data/"/><property name="walArchivePath" value="wal_data/"/><property name="defaultDataRegionConfiguration"><bean class="org.apache.ignite.configuration.DataRegionConfiguration"><property name="persistenceEnabled" value="true"/><property name="maxSize" value="#\{15L * 1024 * 1024 * 1024}"/></bean></property></bean></property> <!-- Enable task execution events for examples. --> <property name="includeEventTypes"> <list> <!--Task execution events--> <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED"/> <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"/> <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"/> <util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_PUT"/> <util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_REMOVED"/> <!--Cache events--> </list> </property> <!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. --> <property name="discoverySpi"> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> <property name="ipFinder"> <!-- Ignite provides several options for automatic discovery that can be used instead os static IP based discovery. For information on all options refer to our documentation: http://apacheignite.readme.io/docs/cluster-config --> <!-- Uncomment static IP finder to enable static-based discovery of initial nodes. --> <!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">--> <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> <property name="addresses"> <list> <!-- In distributed environment, replace with actual host IP address. --> <value>ignite-server-node1:47500..47509</value><value>ignite-server-node2:47500..47509</value><value>kafka-conect-node:47500..47509</value> </list> </property> </bean> </property> </bean> </property> </bean> </beans> The data transfer process is working as intended only for a few hours because of this high CPU consumption, after that the Kafka connector node will be OFFLINE in ignite server topology. Any pointers will help. Thanks in advance. -- This message was sent by Atlassian Jira (v8.20.1#820001)