Hi Ilya,
since this is production environment i could not risk to take heap dump for
now, but i will try to convince my superiors to get one and analyze it.
Queries are heavily used in our system but aren't they autoclosable objects? do
we have to close them anyway?
here are some usage examples on our system;
--insert query is like this; MERGE INTO "ProductLabel" ("productId", "label",
"language") VALUES (?, ?, ?)
igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery);
another usage example;
--sqlFieldsQuery is like this;
String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)";
SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);
sqlFieldsQuery.setLazy(true);
sqlFieldsQuery.setArgs(emails.toArray());
try (QueryCursor<List<?>> ignored = igniteCacheService.getCache(ID,
IgniteCacheType.USER).query(sqlFieldsQuery)) {...}
On 2021/09/12 20:28:09, Shishkov Ilya <shishkovi...@gmail.com> wrote:
> Hi, Ibrahim!
> Have you analyzed the heap dump of the server node JVMs?
> In case your application executes queries are their cursors closed?
>
> пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun <ibrahim.al...@segmentify.com>:
>
> > Igniters any comment on this issue, we are facing huge GC problems on
> > production environment, please advise.
> >
> > On 2021/09/07 14:11:09, Ibrahim Altun <ibrahim.al...@segmentify.com>
> > wrote:
> > > Hi,
> > >
> > > totally 400 - 600K reads/writes/updates
> > > 12core
> > > 64GB RAM
> > > no iowait
> > > 10 nodes
> > >
> > > On 2021/09/07 12:51:28, Piotr Jagielski <p...@touk.pl> wrote:
> > > > Hi,
> > > > Can you provide some information on how you use the cluster? How many
> > reads/writes/updates per second? Also CPU / RAM spec of cluster nodes?
> > > >
> > > > We observed full GC / CPU load / OOM killer when loading big amount of
> > data (15 mln records, data streamer + allowOverwrite=true). We've seen
> > 200-400k updates per sec on JMX metrics, but load up to 10 on nodes, iowait
> > to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU, 32GB
> > RAM). Ignite 2.10
> > > >
> > > > Regards,
> > > > Piotr
> > > >
> > > > On 2021/09/02 08:36:07, Ibrahim Altun <ibrahim.al...@segmentify.com>
> > wrote:
> > > > > After upgrading from 2.7.1 version to 2.10.0 version ignite nodes
> > facing
> > > > > huge full GC operations after 24-36 hours after node start.
> > > > >
> > > > > We try to increase heap size but no luck, here is the start
> > configuration
> > > > > for nodes;
> > > > >
> > > > > JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server
> > > > >
> > -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml
> > > > > -Dcom.sun.management.jmxremote
> > > > > -Dcom.sun.management.jmxremote.authenticate=false
> > > > > -Dcom.sun.management.jmxremote.port=49165
> > > > > -Dcom.sun.management.jmxremote.host=localhost
> > > > > -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g
> > > > > -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true
> > > > > -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=100000
> > > > > -Djava.net.preferIPv4Stack=true"
> > > > >
> > > > > JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC
> > > > > -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
> > > > > -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log
> > > > > -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > > > > -XX:+PrintTenuringDistribution -XX:+PrintGCCause
> > > > > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
> > > > > -XX:GCLogFileSize=100M"
> > > > >
> > > > > here is the 80 hours of GC analyize report:
> > > > >
> > https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ==&channel=WEB
> > > > >
> > > > > do we need more heap size or is there a BUG that we need to be aware?
> > > > >
> > > > > here is the node configuration:
> > > > >
> > > > > <?xml version="1.0" encoding="UTF-8"?>
> > > > > <beans xmlns="http://www.springframework.org/schema/beans";
> > > > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> > > > > xsi:schemaLocation="
> > > > > http://www.springframework.org/schema/beans
> > > > > http://www.springframework.org/schema/beans/spring-beans.xsd";>
> > > > > <bean id="ignite.cfg"
> > > > > class="org.apache.ignite.configuration.IgniteConfiguration">
> > > > > <property name="gridLogger">
> > > > > <bean class="org.apache.ignite.logger.log4j2.Log4J2Logger">
> > > > > <constructor-arg type="java.lang.String"
> > > > > value="/etc/apache-ignite/ignite-log4j2.xml"/>
> > > > > </bean>
> > > > > </property>
> > > > > <property name="communicationSpi">
> > > > > <bean
> > class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> > > > > <property name="usePairedConnections" value="true"/>
> > > > > </bean>
> > > > > </property>
> > > > > <property name="failureDetectionTimeout" value="60000"/>
> > > > > <property name="systemThreadPoolSize" value="128"/>
> > > > > <property name="publicThreadPoolSize" value="128"/>
> > > > > <property name="queryThreadPoolSize" value="128"/>
> > > > > <property name="serviceThreadPoolSize" value="128"/>
> > > > > <property name="stripedPoolSize" value="128"/>
> > > > > <property name="dataStreamerThreadPoolSize" value="4"/>
> > > > > <property name="rebalanceThreadPoolSize" value="16"/>
> > > > >
> > > > > <!-- Explicitly enable peer class loading. -->
> > > > > <property name="peerClassLoadingEnabled" value="true"/>
> > > > >
> > > > > <!-- Enable deploymentSpi,
> > > > > /usr/share/apache-ignite/libs/segmentify directory will be checked
> > > > > every 5 seconds for changed files-->
> > > > > <property name="deploymentSpi">
> > > > > <bean
> > class="org.apache.ignite.spi.deployment.uri.UriDeploymentSpi">
> > > > > <property name="temporaryDirectoryPath"
> > > > > value="/tmp/temp_ignite_libs"/>
> > > > > <property name="uriList">
> > > > > <list>
> > > > >
> > > > > <value>file://freq=5000@localhost
> > /usr/share/apache-ignite/libs/segmentify/</value>
> > > > > </list>
> > > > > </property>
> > > > > </bean>
> > > > > </property>
> > > > >
> > > > > <property name="cacheConfiguration">
> > > > > <list>
> > > > > <!-- Partitioned cache example configuration (Atomic
> > mode). -->
> > > > > <bean
> > class="org.apache.ignite.configuration.CacheConfiguration">
> > > > > <property name="name" value="default"/>
> > > > > <property name="atomicityMode" value="ATOMIC"/>
> > > > > <property name="backups" value="1"/>
> > > > > </bean>
> > > > > </list>
> > > > > </property>
> > > > >
> > > > > <!-- Explicitly configure TCP discovery SPI to provide list of
> > > > > initial nodes. -->
> > > > > <property name="discoverySpi">
> > > > > <bean
> > class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> > > > > <property name="networkTimeout" value="60000"/>
> > > > > <property name="ipFinder">
> > > > > <bean
> > > > >
> > class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> > > > > <property name="addresses">
> > > > > <list>
> > > > > <!-- THERE ARE 10 NODES -->
> > > > > </list>
> > > > > </property>
> > > > > </bean>
> > > > > </property>
> > > > > </bean>
> > > > > </property>
> > > > >
> > > > > <!-- Enabling Apache Ignite native persistence. -->
> > > > > <property name="dataStorageConfiguration">
> > > > > <bean
> > class="org.apache.ignite.configuration.DataStorageConfiguration">
> > > > > <property name="defaultDataRegionConfiguration">
> > > > > <bean
> > > > > class="org.apache.ignite.configuration.DataRegionConfiguration">
> > > > > <property name="persistenceEnabled"
> > value="true"/>
> > > > > <property name="checkpointPageBufferSize"
> > > > > value="#{ 2L * 1024 * 1024 * 1024}"/>
> > > > > <property name="maxSize" value="#{ 40L * 1024 *
> > > > > 1024 * 1024 }"/>
> > > > > </bean>
> > > > > </property>
> > > > > <property name="storagePath"
> > value="/srv/ignite/persist"/>
> > > > > <property name="walPath" value="/srv/ignite/wal"/>
> > > > > <property name="walArchivePath" value="/srv/ignite/wal"/>
> > > > > <property name="walMode" value="LOG_ONLY"/>
> > > > > <property name="walSegmentSize" value="#{ 256L * 1024 *
> > 1024 }"/>
> > > > > <property name="walFlushFrequency" value="5000"/>
> > > > > <property name="maxWalArchiveSize" value="#{ 512L * 1024
> > * 1024 }"/>
> > > > > <property name="writeThrottlingEnabled" value="true"/>
> > > > > <property name="checkpointFrequency" value="300000"/>
> > > > > <property name="checkpointWriteOrder" value="SEQUENTIAL"
> > />
> > > > > </bean>
> > > > > </property>
> > > > > </bean>
> > > > >
> > > > >
> > > > > --
> > > > > <https://www.segmentify.com/>İbrahim Halil AltunSenior Software
> > Engineer+90
> > > > > 536 3327510 • segmentify.com → <https://www.segmentify.com/>UK •
> > Germany •
> > > > > Turkey <https://www.segmentify.com/ecommerce-growth-show>
> > > > > <https://www.g2.com/products/segmentify/reviews>
> > > > >
> > > >
> > >
> >
>
