Just to add what Ibrahim mentioned, I also have a similar issue but I am
using 2.8.1 and we do have a good number of Insert/merge statements getting
executed.
We do get warnings for some of the MERGE statements, like "*The search row
by explicit key isn't supported. The primary key is always used to used to
search a row". *Does is have any impact on heap utilization
In our case, heap memory more than the off heap memory and doing a
frequent rolling-over restarts to avoid OOM.
On Mon, Sep 13, 2021 at 11:47 AM Ibrahim Altun <ibrahim.al...@segmentify.com>
wrote:
> Hi Ilya,
>
> since this is production environment i could not risk to take heap dump
> for now, but i will try to convince my superiors to get one and analyze it.
>
> Queries are heavily used in our system but aren't they autoclosable
> objects? do we have to close them anyway?
>
> here are some usage examples on our system;
> --insert query is like this; MERGE INTO "ProductLabel" ("productId",
> "label", "language") VALUES (?, ?, ?)
> igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery);
>
> another usage example;
> --sqlFieldsQuery is like this;
> String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)";
> SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);
> sqlFieldsQuery.setLazy(true);
> sqlFieldsQuery.setArgs(emails.toArray());
>
> try (QueryCursor<List<?>> ignored = igniteCacheService.getCache(ID,
> IgniteCacheType.USER).query(sqlFieldsQuery)) {...}
>
>
>
> On 2021/09/12 20:28:09, Shishkov Ilya <shishkovi...@gmail.com> wrote:
> > Hi, Ibrahim!
> > Have you analyzed the heap dump of the server node JVMs?
> > In case your application executes queries are their cursors closed?
> >
> > пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun <
> ibrahim.al...@segmentify.com>:
> >
> > > Igniters any comment on this issue, we are facing huge GC problems on
> > > production environment, please advise.
> > >
> > > On 2021/09/07 14:11:09, Ibrahim Altun <ibrahim.al...@segmentify.com>
> > > wrote:
> > > > Hi,
> > > >
> > > > totally 400 - 600K reads/writes/updates
> > > > 12core
> > > > 64GB RAM
> > > > no iowait
> > > > 10 nodes
> > > >
> > > > On 2021/09/07 12:51:28, Piotr Jagielski <p...@touk.pl> wrote:
> > > > > Hi,
> > > > > Can you provide some information on how you use the cluster? How
> many
> > > reads/writes/updates per second? Also CPU / RAM spec of cluster nodes?
> > > > >
> > > > > We observed full GC / CPU load / OOM killer when loading big
> amount of
> > > data (15 mln records, data streamer + allowOverwrite=true). We've seen
> > > 200-400k updates per sec on JMX metrics, but load up to 10 on nodes,
> iowait
> > > to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU,
> 32GB
> > > RAM). Ignite 2.10
> > > > >
> > > > > Regards,
> > > > > Piotr
> > > > >
> > > > > On 2021/09/02 08:36:07, Ibrahim Altun <
> ibrahim.al...@segmentify.com>
> > > wrote:
> > > > > > After upgrading from 2.7.1 version to 2.10.0 version ignite nodes
> > > facing
> > > > > > huge full GC operations after 24-36 hours after node start.
> > > > > >
> > > > > > We try to increase heap size but no luck, here is the start
> > > configuration
> > > > > > for nodes;
> > > > > >
> > > > > > JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server
> > > > > >
> > >
> -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml
> > > > > > -Dcom.sun.management.jmxremote
> > > > > > -Dcom.sun.management.jmxremote.authenticate=false
> > > > > > -Dcom.sun.management.jmxremote.port=49165
> > > > > > -Dcom.sun.management.jmxremote.host=localhost
> > > > > > -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g
> > > > > > -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true
> > > > > > -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=100000
> > > > > > -Djava.net.preferIPv4Stack=true"
> > > > > >
> > > > > > JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC
> > > > > > -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
> > > > > > -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log
> > > > > > -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > > > > > -XX:+PrintTenuringDistribution -XX:+PrintGCCause
> > > > > > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
> > > > > > -XX:GCLogFileSize=100M"
> > > > > >
> > > > > > here is the 80 hours of GC analyize report:
> > > > > >
> > >
> https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ==&channel=WEB
> > > > > >
> > > > > > do we need more heap size or is there a BUG that we need to be
> aware?
> > > > > >
> > > > > > here is the node configuration:
> > > > > >
> > > > > > <?xml version="1.0" encoding="UTF-8"?>
> > > > > > <beans xmlns="http://www.springframework.org/schema/beans";
> > > > > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> > > > > > xsi:schemaLocation="
> > > > > > http://www.springframework.org/schema/beans
> > > > > > http://www.springframework.org/schema/beans/spring-beans.xsd
> ">
> > > > > > <bean id="ignite.cfg"
> > > > > > class="org.apache.ignite.configuration.IgniteConfiguration">
> > > > > > <property name="gridLogger">
> > > > > > <bean
> class="org.apache.ignite.logger.log4j2.Log4J2Logger">
> > > > > > <constructor-arg type="java.lang.String"
> > > > > > value="/etc/apache-ignite/ignite-log4j2.xml"/>
> > > > > > </bean>
> > > > > > </property>
> > > > > > <property name="communicationSpi">
> > > > > > <bean
> > > class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> > > > > > <property name="usePairedConnections" value="true"/>
> > > > > > </bean>
> > > > > > </property>
> > > > > > <property name="failureDetectionTimeout" value="60000"/>
> > > > > > <property name="systemThreadPoolSize" value="128"/>
> > > > > > <property name="publicThreadPoolSize" value="128"/>
> > > > > > <property name="queryThreadPoolSize" value="128"/>
> > > > > > <property name="serviceThreadPoolSize" value="128"/>
> > > > > > <property name="stripedPoolSize" value="128"/>
> > > > > > <property name="dataStreamerThreadPoolSize" value="4"/>
> > > > > > <property name="rebalanceThreadPoolSize" value="16"/>
> > > > > >
> > > > > > <!-- Explicitly enable peer class loading. -->
> > > > > > <property name="peerClassLoadingEnabled" value="true"/>
> > > > > >
> > > > > > <!-- Enable deploymentSpi,
> > > > > > /usr/share/apache-ignite/libs/segmentify directory will be
> checked
> > > > > > every 5 seconds for changed files-->
> > > > > > <property name="deploymentSpi">
> > > > > > <bean
> > > class="org.apache.ignite.spi.deployment.uri.UriDeploymentSpi">
> > > > > > <property name="temporaryDirectoryPath"
> > > > > > value="/tmp/temp_ignite_libs"/>
> > > > > > <property name="uriList">
> > > > > > <list>
> > > > > >
> > > > > > <value>file://freq=5000@localhost
> > > /usr/share/apache-ignite/libs/segmentify/</value>
> > > > > > </list>
> > > > > > </property>
> > > > > > </bean>
> > > > > > </property>
> > > > > >
> > > > > > <property name="cacheConfiguration">
> > > > > > <list>
> > > > > > <!-- Partitioned cache example configuration (Atomic
> > > mode). -->
> > > > > > <bean
> > > class="org.apache.ignite.configuration.CacheConfiguration">
> > > > > > <property name="name" value="default"/>
> > > > > > <property name="atomicityMode" value="ATOMIC"/>
> > > > > > <property name="backups" value="1"/>
> > > > > > </bean>
> > > > > > </list>
> > > > > > </property>
> > > > > >
> > > > > > <!-- Explicitly configure TCP discovery SPI to provide list
> of
> > > > > > initial nodes. -->
> > > > > > <property name="discoverySpi">
> > > > > > <bean
> > > class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> > > > > > <property name="networkTimeout" value="60000"/>
> > > > > > <property name="ipFinder">
> > > > > > <bean
> > > > > >
> > >
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> > > > > > <property name="addresses">
> > > > > > <list>
> > > > > > <!-- THERE ARE 10 NODES -->
> > > > > > </list>
> > > > > > </property>
> > > > > > </bean>
> > > > > > </property>
> > > > > > </bean>
> > > > > > </property>
> > > > > >
> > > > > > <!-- Enabling Apache Ignite native persistence. -->
> > > > > > <property name="dataStorageConfiguration">
> > > > > > <bean
> > > class="org.apache.ignite.configuration.DataStorageConfiguration">
> > > > > > <property name="defaultDataRegionConfiguration">
> > > > > > <bean
> > > > > > class="org.apache.ignite.configuration.DataRegionConfiguration">
> > > > > > <property name="persistenceEnabled"
> > > value="true"/>
> > > > > > <property name="checkpointPageBufferSize"
> > > > > > value="#{ 2L * 1024 * 1024 * 1024}"/>
> > > > > > <property name="maxSize" value="#{ 40L *
> 1024 *
> > > > > > 1024 * 1024 }"/>
> > > > > > </bean>
> > > > > > </property>
> > > > > > <property name="storagePath"
> > > value="/srv/ignite/persist"/>
> > > > > > <property name="walPath" value="/srv/ignite/wal"/>
> > > > > > <property name="walArchivePath"
> value="/srv/ignite/wal"/>
> > > > > > <property name="walMode" value="LOG_ONLY"/>
> > > > > > <property name="walSegmentSize" value="#{ 256L *
> 1024 *
> > > 1024 }"/>
> > > > > > <property name="walFlushFrequency" value="5000"/>
> > > > > > <property name="maxWalArchiveSize" value="#{ 512L *
> 1024
> > > * 1024 }"/>
> > > > > > <property name="writeThrottlingEnabled"
> value="true"/>
> > > > > > <property name="checkpointFrequency" value="300000"/>
> > > > > > <property name="checkpointWriteOrder"
> value="SEQUENTIAL"
> > > />
> > > > > > </bean>
> > > > > > </property>
> > > > > > </bean>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > <https://www.segmentify.com/>İbrahim Halil AltunSenior Software
> > > Engineer+90
> > > > > > 536 3327510 • segmentify.com → <https://www.segmentify.com/>UK •
> > > Germany •
> > > > > > Turkey <https://www.segmentify.com/ecommerce-growth-show>
> > > > > > <https://www.g2.com/products/segmentify/reviews>
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Thanks & Regards,
Naveen Bandaru
