Hi igniters, the situation getting frustrated keeps going really huge GCs. Here is the last GC report;
https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjAvLS1nYy5sb2cuMC5jdXJyZW50IDIuemlwLS02LTMxLTg=&channel=WEB This is production environment we cannot get heap dump since the prodecure stops the whole world. please advise. On 2021/09/13 07:47:16, Naveen Kumar <naveen.band...@gmail.com> wrote: > Just to add what Ibrahim mentioned, I also have a similar issue but I am > using 2.8.1 and we do have a good number of Insert/merge statements getting > executed. > We do get warnings for some of the MERGE statements, like "*The search row > by explicit key isn't supported. The primary key is always used to used to > search a row". *Does is have any impact on heap utilization > > In our case, heap memory more than the off heap memory and doing a > frequent rolling-over restarts to avoid OOM. > > > > On Mon, Sep 13, 2021 at 11:47 AM Ibrahim Altun <ibrahim.al...@segmentify.com> > wrote: > > > Hi Ilya, > > > > since this is production environment i could not risk to take heap dump > > for now, but i will try to convince my superiors to get one and analyze it. > > > > Queries are heavily used in our system but aren't they autoclosable > > objects? do we have to close them anyway? > > > > here are some usage examples on our system; > > --insert query is like this; MERGE INTO "ProductLabel" ("productId", > > "label", "language") VALUES (?, ?, ?) > > igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery); > > > > another usage example; > > --sqlFieldsQuery is like this; > > String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)"; > > SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql); > > sqlFieldsQuery.setLazy(true); > > sqlFieldsQuery.setArgs(emails.toArray()); > > > > try (QueryCursor<List<?>> ignored = igniteCacheService.getCache(ID, > > IgniteCacheType.USER).query(sqlFieldsQuery)) {...} > > > > > > > > On 2021/09/12 20:28:09, Shishkov Ilya <shishkovi...@gmail.com> wrote: > > > Hi, Ibrahim! > > > Have you analyzed the heap dump of the server node JVMs? > > > In case your application executes queries are their cursors closed? > > > > > > пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun < > > ibrahim.al...@segmentify.com>: > > > > > > > Igniters any comment on this issue, we are facing huge GC problems on > > > > production environment, please advise. > > > > > > > > On 2021/09/07 14:11:09, Ibrahim Altun <ibrahim.al...@segmentify.com> > > > > wrote: > > > > > Hi, > > > > > > > > > > totally 400 - 600K reads/writes/updates > > > > > 12core > > > > > 64GB RAM > > > > > no iowait > > > > > 10 nodes > > > > > > > > > > On 2021/09/07 12:51:28, Piotr Jagielski <p...@touk.pl> wrote: > > > > > > Hi, > > > > > > Can you provide some information on how you use the cluster? How > > many > > > > reads/writes/updates per second? Also CPU / RAM spec of cluster nodes? > > > > > > > > > > > > We observed full GC / CPU load / OOM killer when loading big > > amount of > > > > data (15 mln records, data streamer + allowOverwrite=true). We've seen > > > > 200-400k updates per sec on JMX metrics, but load up to 10 on nodes, > > iowait > > > > to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU, > > 32GB > > > > RAM). Ignite 2.10 > > > > > > > > > > > > Regards, > > > > > > Piotr > > > > > > > > > > > > On 2021/09/02 08:36:07, Ibrahim Altun < > > ibrahim.al...@segmentify.com> > > > > wrote: > > > > > > > After upgrading from 2.7.1 version to 2.10.0 version ignite nodes > > > > facing > > > > > > > huge full GC operations after 24-36 hours after node start. > > > > > > > > > > > > > > We try to increase heap size but no luck, here is the start > > > > configuration > > > > > > > for nodes; > > > > > > > > > > > > > > JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server > > > > > > > > > > > > > -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml > > > > > > > -Dcom.sun.management.jmxremote > > > > > > > -Dcom.sun.management.jmxremote.authenticate=false > > > > > > > -Dcom.sun.management.jmxremote.port=49165 > > > > > > > -Dcom.sun.management.jmxremote.host=localhost > > > > > > > -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g > > > > > > > -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true > > > > > > > -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=100000 > > > > > > > -Djava.net.preferIPv4Stack=true" > > > > > > > > > > > > > > JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC > > > > > > > -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC > > > > > > > -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log > > > > > > > -XX:+PrintGCDetails -XX:+PrintGCDateStamps > > > > > > > -XX:+PrintTenuringDistribution -XX:+PrintGCCause > > > > > > > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 > > > > > > > -XX:GCLogFileSize=100M" > > > > > > > > > > > > > > here is the 80 hours of GC analyize report: > > > > > > > > > > > > > https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ==&channel=WEB > > > > > > > > > > > > > > do we need more heap size or is there a BUG that we need to be > > aware? > > > > > > > > > > > > > > here is the node configuration: > > > > > > > > > > > > > > <?xml version="1.0" encoding="UTF-8"?> > > > > > > > <beans xmlns="http://www.springframework.org/schema/beans" > > > > > > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > > > > > > > xsi:schemaLocation=" > > > > > > > http://www.springframework.org/schema/beans > > > > > > > http://www.springframework.org/schema/beans/spring-beans.xsd > > "> > > > > > > > <bean id="ignite.cfg" > > > > > > > class="org.apache.ignite.configuration.IgniteConfiguration"> > > > > > > > <property name="gridLogger"> > > > > > > > <bean > > class="org.apache.ignite.logger.log4j2.Log4J2Logger"> > > > > > > > <constructor-arg type="java.lang.String" > > > > > > > value="/etc/apache-ignite/ignite-log4j2.xml"/> > > > > > > > </bean> > > > > > > > </property> > > > > > > > <property name="communicationSpi"> > > > > > > > <bean > > > > class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"> > > > > > > > <property name="usePairedConnections" value="true"/> > > > > > > > </bean> > > > > > > > </property> > > > > > > > <property name="failureDetectionTimeout" value="60000"/> > > > > > > > <property name="systemThreadPoolSize" value="128"/> > > > > > > > <property name="publicThreadPoolSize" value="128"/> > > > > > > > <property name="queryThreadPoolSize" value="128"/> > > > > > > > <property name="serviceThreadPoolSize" value="128"/> > > > > > > > <property name="stripedPoolSize" value="128"/> > > > > > > > <property name="dataStreamerThreadPoolSize" value="4"/> > > > > > > > <property name="rebalanceThreadPoolSize" value="16"/> > > > > > > > > > > > > > > <!-- Explicitly enable peer class loading. --> > > > > > > > <property name="peerClassLoadingEnabled" value="true"/> > > > > > > > > > > > > > > <!-- Enable deploymentSpi, > > > > > > > /usr/share/apache-ignite/libs/segmentify directory will be > > checked > > > > > > > every 5 seconds for changed files--> > > > > > > > <property name="deploymentSpi"> > > > > > > > <bean > > > > class="org.apache.ignite.spi.deployment.uri.UriDeploymentSpi"> > > > > > > > <property name="temporaryDirectoryPath" > > > > > > > value="/tmp/temp_ignite_libs"/> > > > > > > > <property name="uriList"> > > > > > > > <list> > > > > > > > > > > > > > > <value>file://freq=5000@localhost > > > > /usr/share/apache-ignite/libs/segmentify/</value> > > > > > > > </list> > > > > > > > </property> > > > > > > > </bean> > > > > > > > </property> > > > > > > > > > > > > > > <property name="cacheConfiguration"> > > > > > > > <list> > > > > > > > <!-- Partitioned cache example configuration (Atomic > > > > mode). --> > > > > > > > <bean > > > > class="org.apache.ignite.configuration.CacheConfiguration"> > > > > > > > <property name="name" value="default"/> > > > > > > > <property name="atomicityMode" value="ATOMIC"/> > > > > > > > <property name="backups" value="1"/> > > > > > > > </bean> > > > > > > > </list> > > > > > > > </property> > > > > > > > > > > > > > > <!-- Explicitly configure TCP discovery SPI to provide list > > of > > > > > > > initial nodes. --> > > > > > > > <property name="discoverySpi"> > > > > > > > <bean > > > > class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> > > > > > > > <property name="networkTimeout" value="60000"/> > > > > > > > <property name="ipFinder"> > > > > > > > <bean > > > > > > > > > > > > > class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> > > > > > > > <property name="addresses"> > > > > > > > <list> > > > > > > > <!-- THERE ARE 10 NODES --> > > > > > > > </list> > > > > > > > </property> > > > > > > > </bean> > > > > > > > </property> > > > > > > > </bean> > > > > > > > </property> > > > > > > > > > > > > > > <!-- Enabling Apache Ignite native persistence. --> > > > > > > > <property name="dataStorageConfiguration"> > > > > > > > <bean > > > > class="org.apache.ignite.configuration.DataStorageConfiguration"> > > > > > > > <property name="defaultDataRegionConfiguration"> > > > > > > > <bean > > > > > > > class="org.apache.ignite.configuration.DataRegionConfiguration"> > > > > > > > <property name="persistenceEnabled" > > > > value="true"/> > > > > > > > <property name="checkpointPageBufferSize" > > > > > > > value="#{ 2L * 1024 * 1024 * 1024}"/> > > > > > > > <property name="maxSize" value="#{ 40L * > > 1024 * > > > > > > > 1024 * 1024 }"/> > > > > > > > </bean> > > > > > > > </property> > > > > > > > <property name="storagePath" > > > > value="/srv/ignite/persist"/> > > > > > > > <property name="walPath" value="/srv/ignite/wal"/> > > > > > > > <property name="walArchivePath" > > value="/srv/ignite/wal"/> > > > > > > > <property name="walMode" value="LOG_ONLY"/> > > > > > > > <property name="walSegmentSize" value="#{ 256L * > > 1024 * > > > > 1024 }"/> > > > > > > > <property name="walFlushFrequency" value="5000"/> > > > > > > > <property name="maxWalArchiveSize" value="#{ 512L * > > 1024 > > > > * 1024 }"/> > > > > > > > <property name="writeThrottlingEnabled" > > value="true"/> > > > > > > > <property name="checkpointFrequency" value="300000"/> > > > > > > > <property name="checkpointWriteOrder" > > value="SEQUENTIAL" > > > > /> > > > > > > > </bean> > > > > > > > </property> > > > > > > > </bean> > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > <https://www.segmentify.com/>İbrahim Halil AltunSenior Software > > > > Engineer+90 > > > > > > > 536 3327510 • segmentify.com → <https://www.segmentify.com/>UK • > > > > Germany • > > > > > > > Turkey <https://www.segmentify.com/ecommerce-growth-show> > > > > > > > <https://www.g2.com/products/segmentify/reviews> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Thanks & Regards, > Naveen Bandaru >