I can’t say whether it’s the problem but I can say that’s really not going to help. The default for most of these thread pools is going to be 12 (the number of cores). Each thread is going to have its own working set, stack, etc. If all the threads are thrashing, it’s not going to be able to garbage collect the memory.
> On 27 Sep 2021, at 16:19, Ibrahim Altun <ibrahim.al...@segmentify.com> wrote: > > <property name="systemThreadPoolSize" value="128"/> > <property name="publicThreadPoolSize" value="128"/> > <property name="queryThreadPoolSize" value="128"/> > <property name="serviceThreadPoolSize" value="128"/> > <property name="stripedPoolSize" value="128"/> > <property name="dataStreamerThreadPoolSize" value="64"/> > <property name="rebalanceThreadPoolSize" value="8"/> > > out of curiosity, how does these poolSizes affects heap size usages? we are > using physical machines with total of 12 cores. we did have these sizes > before upgrade from 2.7.6 and had no heap issues at all. we do have same data > size and no changes on any ignite usage in code-base. > > > On 2021/09/20 09:55:34, Stephen Darlington <stephen.darling...@gridgain.com> > wrote: >> I’m not sure if it’s possible to diagnose this without knowing what’s on the >> heap. >> >> I don’t think there are any known issues around heap usage. Certainly a >> MERGE is going to use a lot of heap space, but it should be recovered once >> the statement is closed. Similarly, your select could use a lot of heap >> depending on the number of records scanned, but that should also be released >> once the statement is closed. >> >>> On 20 Sep 2021, at 07:34, Ibrahim Altun <ibrahim.al...@segmentify.com> >>> wrote: >>> >>> Hi igniters, >>> >>> the situation getting frustrated keeps going really huge GCs. Here is the >>> last GC report; >>> >>> https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjAvLS1nYy5sb2cuMC5jdXJyZW50IDIuemlwLS02LTMxLTg=&channel=WEB >>> >>> This is production environment we cannot get heap dump since the prodecure >>> stops the whole world. >>> >>> please advise. >>> >>> On 2021/09/13 07:47:16, Naveen Kumar <naveen.band...@gmail.com> wrote: >>>> Just to add what Ibrahim mentioned, I also have a similar issue but I am >>>> using 2.8.1 and we do have a good number of Insert/merge statements getting >>>> executed. >>>> We do get warnings for some of the MERGE statements, like "*The search row >>>> by explicit key isn't supported. The primary key is always used to used to >>>> search a row". *Does is have any impact on heap utilization >>>> >>>> In our case, heap memory more than the off heap memory and doing a >>>> frequent rolling-over restarts to avoid OOM. >>>> >>>> >>>> >>>> On Mon, Sep 13, 2021 at 11:47 AM Ibrahim Altun >>>> <ibrahim.al...@segmentify.com> >>>> wrote: >>>> >>>>> Hi Ilya, >>>>> >>>>> since this is production environment i could not risk to take heap dump >>>>> for now, but i will try to convince my superiors to get one and analyze >>>>> it. >>>>> >>>>> Queries are heavily used in our system but aren't they autoclosable >>>>> objects? do we have to close them anyway? >>>>> >>>>> here are some usage examples on our system; >>>>> --insert query is like this; MERGE INTO "ProductLabel" ("productId", >>>>> "label", "language") VALUES (?, ?, ?) >>>>> igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery); >>>>> >>>>> another usage example; >>>>> --sqlFieldsQuery is like this; >>>>> String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)"; >>>>> SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql); >>>>> sqlFieldsQuery.setLazy(true); >>>>> sqlFieldsQuery.setArgs(emails.toArray()); >>>>> >>>>> try (QueryCursor<List<?>> ignored = igniteCacheService.getCache(ID, >>>>> IgniteCacheType.USER).query(sqlFieldsQuery)) {...} >>>>> >>>>> >>>>> >>>>> On 2021/09/12 20:28:09, Shishkov Ilya <shishkovi...@gmail.com> wrote: >>>>>> Hi, Ibrahim! >>>>>> Have you analyzed the heap dump of the server node JVMs? >>>>>> In case your application executes queries are their cursors closed? >>>>>> >>>>>> пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun < >>>>> ibrahim.al...@segmentify.com>: >>>>>> >>>>>>> Igniters any comment on this issue, we are facing huge GC problems on >>>>>>> production environment, please advise. >>>>>>> >>>>>>> On 2021/09/07 14:11:09, Ibrahim Altun <ibrahim.al...@segmentify.com> >>>>>>> wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> totally 400 - 600K reads/writes/updates >>>>>>>> 12core >>>>>>>> 64GB RAM >>>>>>>> no iowait >>>>>>>> 10 nodes >>>>>>>> >>>>>>>> On 2021/09/07 12:51:28, Piotr Jagielski <p...@touk.pl> wrote: >>>>>>>>> Hi, >>>>>>>>> Can you provide some information on how you use the cluster? How >>>>> many >>>>>>> reads/writes/updates per second? Also CPU / RAM spec of cluster nodes? >>>>>>>>> >>>>>>>>> We observed full GC / CPU load / OOM killer when loading big >>>>> amount of >>>>>>> data (15 mln records, data streamer + allowOverwrite=true). We've seen >>>>>>> 200-400k updates per sec on JMX metrics, but load up to 10 on nodes, >>>>> iowait >>>>>>> to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU, >>>>> 32GB >>>>>>> RAM). Ignite 2.10 >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Piotr >>>>>>>>> >>>>>>>>> On 2021/09/02 08:36:07, Ibrahim Altun < >>>>> ibrahim.al...@segmentify.com> >>>>>>> wrote: >>>>>>>>>> After upgrading from 2.7.1 version to 2.10.0 version ignite nodes >>>>>>> facing >>>>>>>>>> huge full GC operations after 24-36 hours after node start. >>>>>>>>>> >>>>>>>>>> We try to increase heap size but no luck, here is the start >>>>>>> configuration >>>>>>>>>> for nodes; >>>>>>>>>> >>>>>>>>>> JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server >>>>>>>>>> >>>>>>> >>>>> -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml >>>>>>>>>> -Dcom.sun.management.jmxremote >>>>>>>>>> -Dcom.sun.management.jmxremote.authenticate=false >>>>>>>>>> -Dcom.sun.management.jmxremote.port=49165 >>>>>>>>>> -Dcom.sun.management.jmxremote.host=localhost >>>>>>>>>> -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g >>>>>>>>>> -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true >>>>>>>>>> -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=100000 >>>>>>>>>> -Djava.net.preferIPv4Stack=true" >>>>>>>>>> >>>>>>>>>> JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC >>>>>>>>>> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC >>>>>>>>>> -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log >>>>>>>>>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps >>>>>>>>>> -XX:+PrintTenuringDistribution -XX:+PrintGCCause >>>>>>>>>> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 >>>>>>>>>> -XX:GCLogFileSize=100M" >>>>>>>>>> >>>>>>>>>> here is the 80 hours of GC analyize report: >>>>>>>>>> >>>>>>> >>>>> https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ==&channel=WEB >>>>>>>>>> >>>>>>>>>> do we need more heap size or is there a BUG that we need to be >>>>> aware? >>>>>>>>>> >>>>>>>>>> here is the node configuration: >>>>>>>>>> >>>>>>>>>> <?xml version="1.0" encoding="UTF-8"?> >>>>>>>>>> <beans xmlns="http://www.springframework.org/schema/beans"; >>>>>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>>>>>>>>> xsi:schemaLocation=" >>>>>>>>>> http://www.springframework.org/schema/beans >>>>>>>>>> http://www.springframework.org/schema/beans/spring-beans.xsd >>>>> "> >>>>>>>>>> <bean id="ignite.cfg" >>>>>>>>>> class="org.apache.ignite.configuration.IgniteConfiguration"> >>>>>>>>>> <property name="gridLogger"> >>>>>>>>>> <bean >>>>> class="org.apache.ignite.logger.log4j2.Log4J2Logger"> >>>>>>>>>> <constructor-arg type="java.lang.String" >>>>>>>>>> value="/etc/apache-ignite/ignite-log4j2.xml"/> >>>>>>>>>> </bean> >>>>>>>>>> </property> >>>>>>>>>> <property name="communicationSpi"> >>>>>>>>>> <bean >>>>>>> class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"> >>>>>>>>>> <property name="usePairedConnections" value="true"/> >>>>>>>>>> </bean> >>>>>>>>>> </property> >>>>>>>>>> <property name="failureDetectionTimeout" value="60000"/> >>>>>>>>>> <property name="systemThreadPoolSize" value="128"/> >>>>>>>>>> <property name="publicThreadPoolSize" value="128"/> >>>>>>>>>> <property name="queryThreadPoolSize" value="128"/> >>>>>>>>>> <property name="serviceThreadPoolSize" value="128"/> >>>>>>>>>> <property name="stripedPoolSize" value="128"/> >>>>>>>>>> <property name="dataStreamerThreadPoolSize" value="4"/> >>>>>>>>>> <property name="rebalanceThreadPoolSize" value="16"/> >>>>>>>>>> >>>>>>>>>> <!-- Explicitly enable peer class loading. --> >>>>>>>>>> <property name="peerClassLoadingEnabled" value="true"/> >>>>>>>>>> >>>>>>>>>> <!-- Enable deploymentSpi, >>>>>>>>>> /usr/share/apache-ignite/libs/segmentify directory will be >>>>> checked >>>>>>>>>> every 5 seconds for changed files--> >>>>>>>>>> <property name="deploymentSpi"> >>>>>>>>>> <bean >>>>>>> class="org.apache.ignite.spi.deployment.uri.UriDeploymentSpi"> >>>>>>>>>> <property name="temporaryDirectoryPath" >>>>>>>>>> value="/tmp/temp_ignite_libs"/> >>>>>>>>>> <property name="uriList"> >>>>>>>>>> <list> >>>>>>>>>> >>>>>>>>>> <value>file://freq=5000@localhost >>>>>>> /usr/share/apache-ignite/libs/segmentify/</value> >>>>>>>>>> </list> >>>>>>>>>> </property> >>>>>>>>>> </bean> >>>>>>>>>> </property> >>>>>>>>>> >>>>>>>>>> <property name="cacheConfiguration"> >>>>>>>>>> <list> >>>>>>>>>> <!-- Partitioned cache example configuration (Atomic >>>>>>> mode). --> >>>>>>>>>> <bean >>>>>>> class="org.apache.ignite.configuration.CacheConfiguration"> >>>>>>>>>> <property name="name" value="default"/> >>>>>>>>>> <property name="atomicityMode" value="ATOMIC"/> >>>>>>>>>> <property name="backups" value="1"/> >>>>>>>>>> </bean> >>>>>>>>>> </list> >>>>>>>>>> </property> >>>>>>>>>> >>>>>>>>>> <!-- Explicitly configure TCP discovery SPI to provide list >>>>> of >>>>>>>>>> initial nodes. --> >>>>>>>>>> <property name="discoverySpi"> >>>>>>>>>> <bean >>>>>>> class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> >>>>>>>>>> <property name="networkTimeout" value="60000"/> >>>>>>>>>> <property name="ipFinder"> >>>>>>>>>> <bean >>>>>>>>>> >>>>>>> >>>>> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> >>>>>>>>>> <property name="addresses"> >>>>>>>>>> <list> >>>>>>>>>> <!-- THERE ARE 10 NODES --> >>>>>>>>>> </list> >>>>>>>>>> </property> >>>>>>>>>> </bean> >>>>>>>>>> </property> >>>>>>>>>> </bean> >>>>>>>>>> </property> >>>>>>>>>> >>>>>>>>>> <!-- Enabling Apache Ignite native persistence. --> >>>>>>>>>> <property name="dataStorageConfiguration"> >>>>>>>>>> <bean >>>>>>> class="org.apache.ignite.configuration.DataStorageConfiguration"> >>>>>>>>>> <property name="defaultDataRegionConfiguration"> >>>>>>>>>> <bean >>>>>>>>>> class="org.apache.ignite.configuration.DataRegionConfiguration"> >>>>>>>>>> <property name="persistenceEnabled" >>>>>>> value="true"/> >>>>>>>>>> <property name="checkpointPageBufferSize" >>>>>>>>>> value="#{ 2L * 1024 * 1024 * 1024}"/> >>>>>>>>>> <property name="maxSize" value="#{ 40L * >>>>> 1024 * >>>>>>>>>> 1024 * 1024 }"/> >>>>>>>>>> </bean> >>>>>>>>>> </property> >>>>>>>>>> <property name="storagePath" >>>>>>> value="/srv/ignite/persist"/> >>>>>>>>>> <property name="walPath" value="/srv/ignite/wal"/> >>>>>>>>>> <property name="walArchivePath" >>>>> value="/srv/ignite/wal"/> >>>>>>>>>> <property name="walMode" value="LOG_ONLY"/> >>>>>>>>>> <property name="walSegmentSize" value="#{ 256L * >>>>> 1024 * >>>>>>> 1024 }"/> >>>>>>>>>> <property name="walFlushFrequency" value="5000"/> >>>>>>>>>> <property name="maxWalArchiveSize" value="#{ 512L * >>>>> 1024 >>>>>>> * 1024 }"/> >>>>>>>>>> <property name="writeThrottlingEnabled" >>>>> value="true"/> >>>>>>>>>> <property name="checkpointFrequency" value="300000"/> >>>>>>>>>> <property name="checkpointWriteOrder" >>>>> value="SEQUENTIAL" >>>>>>> /> >>>>>>>>>> </bean> >>>>>>>>>> </property> >>>>>>>>>> </bean> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> <https://www.segmentify.com/>İbrahim Halil AltunSenior Software >>>>>>> Engineer+90 >>>>>>>>>> 536 3327510 • segmentify.com → <https://www.segmentify.com/>UK • >>>>>>> Germany • >>>>>>>>>> Turkey <https://www.segmentify.com/ecommerce-growth-show> >>>>>>>>>> <https://www.g2.com/products/segmentify/reviews> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks & Regards, >>>> Naveen Bandaru >>>> >> >> >>
