You can check the distribution with the control script: ./control.sh —cache distribution null
This displays the number of records per partition for all caches and nodes. > On 8 Nov 2021, at 20:24, Shishkov Ilya <shishkovi...@gmail.com> wrote: > > Hi! > As I know, if you remove some cache entries (keys), their corresponding > segments in the data pages remain empty in the persistent store until you > insert these keys back into Ignite. So, it looks like voids in data pages. > But the rebalanced data is put into persistent storage without any kind of > "voids", i.e. it is already compressed during the rebalance routine. > > пн, 8 нояб. 2021 г. в 17:36, Ibrahim Altun <ibrahim.al...@segmentify.com > <mailto:ibrahim.al...@segmentify.com>>: > Hi, > > We had a 2 node cluster with persistence enabled with backup(1) enabled, this > morning we've added a new node to the cluster. > Although rebalancing is finished Ignite Persistence values are not evenly > distributed; > > server-1: > [2021-11-08T14:31:29,371][INFO ][grid-timeout-worker-#13][IgniteKernal] > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=d9a3fb2f, uptime=01:35:00.491] > ^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36, > minorTopVer=3] > ^-- Network [addrs=[10.156.0.112, 127.0.0.1], discoPort=47500, > commPort=47100] > ^-- CPU [CPUs=4, curLoad=0.33%, avgLoad=3.48%, GC=0%] > ^-- Heap [used=5418MB, free=33.85%, comm=8192MB] > ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] > ^-- Page memory [pages=1524682] > ^-- sysMemPlc region [type=internal, persistence=true, lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, > allocRam=100MB, allocTotal=0MB] > ^-- default region [type=default, persistence=true, lazyAlloc=true, > ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, > allocRam=6144MB, allocTotal=80891MB] > ^-- metastoreMemPlc region [type=internal, persistence=true, > lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.81%, > allocRam=0MB, allocTotal=0MB] > ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, > allocRam=100MB, allocTotal=0MB] > ^-- volatileDsMemPlc region [type=user, persistence=false, > lazyAlloc=true, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, > allocRam=0MB] > ^-- Ignite persistence [used=80891MB] > ^-- Outbound messages queue [size=0] > ^-- Public thread pool [active=0, idle=0, qSize=0] > ^-- System thread pool [active=0, idle=4, qSize=0] > > server-2: > [2021-11-08T14:31:20,475][INFO ][grid-timeout-worker-#13][IgniteKernal] > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=d001436d, uptime=00:46:00.231] > ^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36, > minorTopVer=3] > ^-- Network [addrs=[10.156.0.113, 127.0.0.1], discoPort=47500, > commPort=47100] > ^-- CPU [CPUs=4, curLoad=4.43%, avgLoad=5.11%, GC=0%] > ^-- Heap [used=6468MB, free=21.04%, comm=8192MB] > ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] > ^-- Page memory [pages=1524684] > ^-- sysMemPlc region [type=internal, persistence=true, lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, > allocRam=100MB, allocTotal=0MB] > ^-- default region [type=default, persistence=true, lazyAlloc=true, > ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, > allocRam=6144MB, allocTotal=82852MB] > ^-- metastoreMemPlc region [type=internal, persistence=true, > lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.8%, > allocRam=0MB, allocTotal=0MB] > ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, > allocRam=100MB, allocTotal=0MB] > ^-- volatileDsMemPlc region [type=user, persistence=false, > lazyAlloc=true, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, > allocRam=0MB] > ^-- Ignite persistence [used=82852MB] > ^-- Outbound messages queue [size=0] > ^-- Public thread pool [active=0, idle=0, qSize=0] > ^-- System thread pool [active=0, idle=4, qSize=0] > > server-3: > [2021-11-08T14:31:21,364][INFO ][grid-timeout-worker-#13][IgniteKernal] > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=186395d1, uptime=03:36:01.279] > ^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36, > minorTopVer=3] > ^-- Network [addrs=[10.156.0.10, 127.0.0.1], discoPort=47500, > commPort=47100] > ^-- CPU [CPUs=4, curLoad=0.4%, avgLoad=8.55%, GC=0%] > ^-- Heap [used=6153MB, free=24.89%, comm=8192MB] > ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] > ^-- Page memory [pages=1524749] > ^-- sysMemPlc region [type=internal, persistence=true, lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.98%, > allocRam=100MB, allocTotal=0MB] > ^-- default region [type=default, persistence=true, lazyAlloc=true, > ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, > allocRam=6144MB, allocTotal=16164MB] > ^-- metastoreMemPlc region [type=internal, persistence=true, > lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.56%, > allocRam=0MB, allocTotal=0MB] > ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, > allocRam=100MB, allocTotal=0MB] > ^-- volatileDsMemPlc region [type=user, persistence=false, > lazyAlloc=true, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, > allocRam=0MB] > ^-- Ignite persistence [used=16165MB] > ^-- Outbound messages queue [size=0] > ^-- Public thread pool [active=0, idle=0, qSize=0] > ^-- System thread pool [active=0, idle=4, qSize=0] > > Before new node added to cluster data was evenly distributed; > > server-1: > [2021-11-08T06:00:52,779][INFO ][grid-timeout-worker-#13][IgniteKernal] > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=cde7abcb, uptime=4 days, 03:38:05.557] > ^-- Cluster [hosts=7, CPUs=28, servers=2, clients=7, topVer=17, > minorTopVer=0] > ^-- Network [addrs=[10.156.0.112, 127.0.0.1], discoPort=47500, > commPort=47100] > ^-- CPU [CPUs=4, curLoad=100%, avgLoad=5.78%, GC=117%] > ^-- Heap [used=7669MB, free=6.37%, comm=8192MB] > ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] > ^-- Page memory [pages=1524669] > ^-- sysMemPlc region [type=internal, persistence=true, lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, > allocRam=100MB, allocTotal=0MB] > ^-- default region [type=default, persistence=true, lazyAlloc=true, > ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, > allocRam=6144MB, allocTotal=121061MB] > ^-- metastoreMemPlc region [type=internal, persistence=true, > lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.86%, > allocRam=0MB, allocTotal=0MB] > ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, > allocRam=100MB, allocTotal=0MB] > ^-- volatileDsMemPlc region [type=user, persistence=false, > lazyAlloc=true, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, > allocRam=0MB] > ^-- Ignite persistence [used=121061MB] > ^-- Outbound messages queue [size=1] > ^-- Public thread pool [active=0, idle=0, qSize=0] > ^-- System thread pool [active=1, idle=3, qSize=1] > > server-2: > [2021-11-08T06:00:37,491][INFO ][grid-timeout-worker-#13][IgniteKernal] > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=90f15a32, uptime=4 days, 03:37:54.391] > ^-- Cluster [hosts=7, CPUs=28, servers=2, clients=7, topVer=17, > minorTopVer=0] > ^-- Network [addrs=[10.156.0.113, 127.0.0.1], discoPort=47500, > commPort=47100] > ^-- CPU [CPUs=4, curLoad=2.6%, avgLoad=7.38%, GC=0%] > ^-- Heap [used=7877MB, free=3.83%, comm=8192MB] > ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] > ^-- Page memory [pages=1524670] > ^-- sysMemPlc region [type=internal, persistence=true, lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, > allocRam=100MB, allocTotal=0MB] > ^-- default region [type=default, persistence=true, lazyAlloc=true, > ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, > allocRam=6144MB, allocTotal=121310MB] > ^-- metastoreMemPlc region [type=internal, persistence=true, > lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.86%, > allocRam=0MB, allocTotal=0MB] > ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, > allocRam=100MB, allocTotal=0MB] > ^-- volatileDsMemPlc region [type=user, persistence=false, > lazyAlloc=true, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, > allocRam=0MB] > ^-- Ignite persistence [used=121310MB] > ^-- Outbound messages queue [size=0] > ^-- Public thread pool [active=0, idle=0, qSize=0] > ^-- System thread pool [active=0, idle=4, qSize=0] > > > My expectation is data to be distributed evenly. > > What am I missing? > > Regardds. > > -- > <https://www.segmentify.com/> > İbrahim Halil Altun > Senior Software Engineer > > +90 536 3327510 • segmentify.com → <https://www.segmentify.com/> > UK • Germany • Turkey > > <https://www.segmentify.com/ecommerce-growth-show> > <https://www.g2.com/products/segmentify/reviews>
