@ilya We did not force system to remove any keys, just add a new node to the cluster, so my expectation was system to distribute ~120GB data across the nodes, and expect to see ~80GB of data on each node (have total ~60GB(without backup size) primary data, divided into 3 nodes ~40GB primary ~40GB backup)
@Stephen what will distribution indicate? is there a way force cluster to redistribute all data? On Tue, 9 Nov 2021 at 12:31, Stephen Darlington < stephen.darling...@gridgain.com> wrote: > You can check the distribution with the control script: > > ./control.sh —cache distribution null > > This displays the number of records per partition for all caches and > nodes. > > On 8 Nov 2021, at 20:24, Shishkov Ilya <shishkovi...@gmail.com> wrote: > > Hi! > As I know, if you remove some cache entries (keys), their corresponding > segments in the data pages remain empty in the persistent store until you > insert these keys back into Ignite. So, it looks like voids in data pages. > But the rebalanced data is put into persistent storage without any kind of > "voids", i.e. it is already compressed during the rebalance routine. > > пн, 8 нояб. 2021 г. в 17:36, Ibrahim Altun <ibrahim.al...@segmentify.com>: > >> Hi, >> >> We had a 2 node cluster with persistence enabled with backup(1) enabled, >> this morning we've added a new node to the cluster. >> Although rebalancing is finished Ignite Persistence values are not evenly >> distributed; >> >> server-1: >> [2021-11-08T14:31:29,371][INFO ][grid-timeout-worker-#13][IgniteKernal] >> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >> ^-- Node [id=d9a3fb2f, uptime=01:35:00.491] >> ^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36, >> minorTopVer=3] >> ^-- Network [addrs=[10.156.0.112, 127.0.0.1], discoPort=47500, >> commPort=47100] >> ^-- CPU [CPUs=4, curLoad=0.33%, avgLoad=3.48%, GC=0%] >> ^-- Heap [used=5418MB, free=33.85%, comm=8192MB] >> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >> ^-- Page memory [pages=1524682] >> ^-- sysMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, >> allocRam=100MB, allocTotal=0MB] >> ^-- default region [type=default, persistence=true, lazyAlloc=true, >> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >> allocRam=6144MB, allocTotal=80891MB] >> ^-- metastoreMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.81%, >> allocRam=0MB, allocTotal=0MB] >> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=100MB, allocTotal=0MB] >> ^-- volatileDsMemPlc region [type=user, persistence=false, >> lazyAlloc=true, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=0MB] >> ^-- Ignite persistence [used=80891MB] >> ^-- Outbound messages queue [size=0] >> ^-- Public thread pool [active=0, idle=0, qSize=0] >> ^-- System thread pool [active=0, idle=4, qSize=0] >> >> server-2: >> [2021-11-08T14:31:20,475][INFO ][grid-timeout-worker-#13][IgniteKernal] >> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >> ^-- Node [id=d001436d, uptime=00:46:00.231] >> ^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36, >> minorTopVer=3] >> ^-- Network [addrs=[10.156.0.113, 127.0.0.1], discoPort=47500, >> commPort=47100] >> ^-- CPU [CPUs=4, curLoad=4.43%, avgLoad=5.11%, GC=0%] >> ^-- Heap [used=6468MB, free=21.04%, comm=8192MB] >> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >> ^-- Page memory [pages=1524684] >> ^-- sysMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, >> allocRam=100MB, allocTotal=0MB] >> ^-- default region [type=default, persistence=true, lazyAlloc=true, >> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >> allocRam=6144MB, allocTotal=82852MB] >> ^-- metastoreMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.8%, >> allocRam=0MB, allocTotal=0MB] >> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=100MB, allocTotal=0MB] >> ^-- volatileDsMemPlc region [type=user, persistence=false, >> lazyAlloc=true, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=0MB] >> ^-- Ignite persistence [used=82852MB] >> ^-- Outbound messages queue [size=0] >> ^-- Public thread pool [active=0, idle=0, qSize=0] >> ^-- System thread pool [active=0, idle=4, qSize=0] >> >> server-3: >> [2021-11-08T14:31:21,364][INFO ][grid-timeout-worker-#13][IgniteKernal] >> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >> ^-- Node [id=186395d1, uptime=03:36:01.279] >> ^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36, >> minorTopVer=3] >> ^-- Network [addrs=[10.156.0.10, 127.0.0.1], discoPort=47500, >> commPort=47100] >> ^-- CPU [CPUs=4, curLoad=0.4%, avgLoad=8.55%, GC=0%] >> ^-- Heap [used=6153MB, free=24.89%, comm=8192MB] >> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >> ^-- Page memory [pages=1524749] >> ^-- sysMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.98%, >> allocRam=100MB, allocTotal=0MB] >> ^-- default region [type=default, persistence=true, lazyAlloc=true, >> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >> allocRam=6144MB, allocTotal=16164MB] >> ^-- metastoreMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.56%, >> allocRam=0MB, allocTotal=0MB] >> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=100MB, allocTotal=0MB] >> ^-- volatileDsMemPlc region [type=user, persistence=false, >> lazyAlloc=true, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=0MB] >> ^-- Ignite persistence [used=16165MB] >> ^-- Outbound messages queue [size=0] >> ^-- Public thread pool [active=0, idle=0, qSize=0] >> ^-- System thread pool [active=0, idle=4, qSize=0] >> >> Before new node added to cluster data was evenly distributed; >> >> server-1: >> [2021-11-08T06:00:52,779][INFO ][grid-timeout-worker-#13][IgniteKernal] >> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >> ^-- Node [id=cde7abcb, uptime=4 days, 03:38:05.557] >> ^-- Cluster [hosts=7, CPUs=28, servers=2, clients=7, topVer=17, >> minorTopVer=0] >> ^-- Network [addrs=[10.156.0.112, 127.0.0.1], discoPort=47500, >> commPort=47100] >> ^-- CPU [CPUs=4, curLoad=100%, avgLoad=5.78%, GC=117%] >> ^-- Heap [used=7669MB, free=6.37%, comm=8192MB] >> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >> ^-- Page memory [pages=1524669] >> ^-- sysMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, >> allocRam=100MB, allocTotal=0MB] >> ^-- default region [type=default, persistence=true, lazyAlloc=true, >> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >> allocRam=6144MB, allocTotal=121061MB] >> ^-- metastoreMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.86%, >> allocRam=0MB, allocTotal=0MB] >> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=100MB, allocTotal=0MB] >> ^-- volatileDsMemPlc region [type=user, persistence=false, >> lazyAlloc=true, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=0MB] >> ^-- Ignite persistence [used=121061MB] >> ^-- Outbound messages queue [size=1] >> ^-- Public thread pool [active=0, idle=0, qSize=0] >> ^-- System thread pool [active=1, idle=3, qSize=1] >> >> server-2: >> [2021-11-08T06:00:37,491][INFO ][grid-timeout-worker-#13][IgniteKernal] >> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >> ^-- Node [id=90f15a32, uptime=4 days, 03:37:54.391] >> ^-- Cluster [hosts=7, CPUs=28, servers=2, clients=7, topVer=17, >> minorTopVer=0] >> ^-- Network [addrs=[10.156.0.113, 127.0.0.1], discoPort=47500, >> commPort=47100] >> ^-- CPU [CPUs=4, curLoad=2.6%, avgLoad=7.38%, GC=0%] >> ^-- Heap [used=7877MB, free=3.83%, comm=8192MB] >> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >> ^-- Page memory [pages=1524670] >> ^-- sysMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, >> allocRam=100MB, allocTotal=0MB] >> ^-- default region [type=default, persistence=true, lazyAlloc=true, >> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >> allocRam=6144MB, allocTotal=121310MB] >> ^-- metastoreMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.86%, >> allocRam=0MB, allocTotal=0MB] >> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=100MB, allocTotal=0MB] >> ^-- volatileDsMemPlc region [type=user, persistence=false, >> lazyAlloc=true, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=0MB] >> ^-- Ignite persistence [used=121310MB] >> ^-- Outbound messages queue [size=0] >> ^-- Public thread pool [active=0, idle=0, qSize=0] >> ^-- System thread pool [active=0, idle=4, qSize=0] >> >> >> My expectation is data to be distributed evenly. >> >> What am I missing? >> >> Regardds. >> >> -- >> <https://www.segmentify.com/>İbrahim Halil AltunSenior Software Engineer+90 >> 536 3327510 • segmentify.com → <https://www.segmentify.com/>UK • Germany >> • Turkey <https://www.segmentify.com/ecommerce-growth-show> >> <https://www.g2.com/products/segmentify/reviews> >> > > > -- <https://www.segmentify.com/>İbrahim Halil AltunSenior Software Engineer+90 536 3327510 • segmentify.com → <https://www.segmentify.com/>UK • Germany • Turkey <https://www.segmentify.com/ecommerce-growth-show> <https://www.g2.com/products/segmentify/reviews>
