Hi Shawn.
It looks strange that each entry costs 10k in average, but you expect 50
bytes at most.
You wrote that there are 300k entries, but I see 5000k keys in Heap.
Grid configuration and Heapdump would be helpful to understand who holds
all that data. Or It would be great to have a reproducer.
On Thu, Oct 27, 2016 at 10:29 AM, Shawn Du <shawn...@neulion.com.cn> wrote:
> Hi Andrey Mashenkov,
>
>
>
> I checkout pr/1101 pr/1037 and have a test both of them. Things maybe go
> better but not resolved.
>
>
>
> This is my cache state, I think it is full, for code like this:
>
>
>
> config.setSwapEnabled(swapEnable);
> if (swapEnable)
> {
> EvictionPolicy policy = new FifoEvictionPolicy(10000);
> config.setEvictionPolicy(policy);
> }
>
>
>
> new caches should be in memory and the old will be evicted into disk. The
> memory should stopping growing.
>
> After my test, it is not the case. It is still keep growing fast, at speed
> of 500M/30minutes.
>
> I can understand that there will need some extra memory to maintain
> something, but it leaks too fast.
>
>
>
> Thanks
>
> Shawn
>
>
>
>
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlinesessioncount_by_cityid(@c1) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlinesessioncount_by_regionid(@c2) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlineviewcount_by_cityid(@c3) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlineviewcount_by_regionid(@c4) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlinevisitorcount_by_cityid(@c5) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlinevisitorcount_by_regionid(@c6) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | videoviewcount_by_cityid(@c7) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg: 5000.00
> (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | videoviewcount_by_regionid(@c8) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-----------------------------------------------------------
> ------------------------------------------------------------
> --------------------+
>
>
>
> *发件人:* Shawn Du [mailto:shawn...@neulion.com.cn]
> *发送时间:* 2016年10月27日 9:17
> *收件人:* user@ignite.apache.org
> *主题:* 答复: 答复: ignite used too much memory
>
>
>
> Hi Andrey Mashenkov,
>
>
>
> Thanks, I will have a try and apply these locally.
>
>
>
> Do you know 1.8’s release plan?
>
>
>
> Thanks
>
> Shawn
>
>
>
> *发件人:* Andrey Mashenkov [mailto:amashen...@gridgain.com
> <amashen...@gridgain.com>]
> *发送时间:* 2016年10月26日 20:53
> *收件人:* user@ignite.apache.org
> *主题:* Re: 答复: ignite used too much memory
>
>
>
> Hi, Shawn Du
>
>
>
> It seems you faced next 2 bugs.
>
>
>
> First bug: High memory utilization using OffHeap with ExpirePolicy. Issue
> has a fix, but it is not merged to master yet. See:
> https://issues.apache.org/jira/browse/IGNITE-3840.
>
> Second bug: TTL Manager continue track evicted (and removed) entries,
> https://issues.apache.org/jira/browse/IGNITE-3948, it seems to be ok for
> merge, but still is not present in master.
>
>
>
> You can try to merge them locally or wait until they will be available in
> master.
>
>
>
> On Wed, Oct 26, 2016 at 2:35 PM, Shawn Du <shawn...@neulion.com.cn> wrote:
>
> Hi experts,
>
>
>
> Can anyone help to explain ignite memory model?
>
>
>
> Now I tried following ways, but no effect.
>
> 1) Remove all indexes.
>
> 2) enable swap. I see more than 800M’s data are stored in swapspace
> directory.
>
> 3) Don’t cache short life entries. All cache with fifo evict policy
> and max size is 10k.
>
>
>
> After running for a while, I have a check the caches, there were only 20k
> entries, but the memory still grow up?
>
>
>
> Please help.
>
>
>
> Thanks
>
> Shawn
>
>
>
>
>
> *发件人**:* Shawn Du [mailto:shawn...@neulion.com.cn]
> *发送时间**:* 2016年10月26日 17:17
> *收件人**:* user@ignite.apache.org
> *主题**:* 答复: ignite used too much memory
>
>
>
> Hi
>
>
>
> This is the output of jmap histo:live <pid>.
>
>
>
> Any useful information? There are about 300k cache entries. Also I enable
> swap.
>
>
>
> Each entry costs 10K memory in average. But each entry only has 50 bytes
> at most.
>
>
>
> Any help will be appreciated.
>
>
>
> num #instances #bytes class name
>
> ----------------------------------------------
>
> 1: 21642240 519413760 org.apache.ignite.internal.
> util.GridCircularBuffer$Item
>
> 2: 49027 405647408 [Lorg.jsr166.ConcurrentHashMap8$Node;
>
> 3: 6056069 304233144 [B
>
> 4: 5754322 279563064 [C
>
> 5: 11376526 273036624 java.util.concurrent.
> ConcurrentSkipListMap$Node
>
> 6: 5688514 227540560 org.apache.ignite.internal.
> processors.query.h2.opt.GridH2KeyValueRowOnheap
>
> 7: 5684293 181897376 org.apache.ignite.internal.
> processors.cache.GridCacheTtlManager$EntryWrapper
>
> 8: 5333102 170659264 java.util.concurrent.
> ConcurrentHashMap$Node
>
> 9: 5325117 170403744 org.apache.ignite.spi.swapspace.file.
> FileSwapSpaceSpi$SwapValue
>
> 10: 5754728 138113472 java.lang.String
>
> 11: 5689577 136549848 java.util.concurrent.
> ConcurrentSkipListMap$Index
>
> 12: 5686353 136472472 org.apache.ignite.internal.
> processors.cache.KeyCacheObjectImpl
>
> 13: 5325117 127802808 org.apache.ignite.spi.
> swapspace.SwapKey
>
> 14: 5690452 91047232 org.h2.value.ValueString
>
> 15: 85112 87930752 [Lorg.apache.ignite.internal.
> util.GridCircularBuffer$Item;
>
> 16: 1515799 48505568 java.util.HashMap$Node
>
> 17: 268818 42628512 [Ljava.lang.Object;
>
> 18: 13467 35031504 [Ljava.util.concurrent.
> ConcurrentHashMap$Node;
>
> 19: 361155 23113920 org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCacheEntry
>
> 20: 1226922 19630752 java.lang.Integer
>
> 21: 102183 15207336 [Ljava.util.HashMap$Node;
>
> 22: 363397 14535880 org.apache.ignite.internal.
> binary.BinaryObjectImpl
>
> 23: 395583 12658656 org.jsr166.ConcurrentHashMap8$Node
>
> 24: 359176 11493632 org.apache.ignite.internal.
> processors.cache.extras.GridCacheTtlEntryExtras
>
> 25: 363397 8721528 org.apache.ignite.internal.
> processors.query.h2.opt.GridH2ValueCacheObject
>
> 26: 257615 6182760 java.util.concurrent.atomic.AtomicLong
>
> 27: 170775 5464800 java.util.concurrent.locks.
> ReentrantLock$NonfairSync
>
> 28: 85112 5447168 org.apache.ignite.internal.
> processors.cache.distributed.dht.GridDhtLocalPartition
>
> 29: 85112 5447168 org.apache.ignite.internal.
> processors.cache.distributed.dht.GridDhtLocalPartition$1
>
> 30: 103188 4953024 java.util.HashMap
>
> 31: 87751 3510040 org.jsr166.ConcurrentHashMap8
>
> 32: 136382 3273168 org.jsr166.ConcurrentLinkedDeque8$Node
>
> 33: 90699 2902368 org.jsr166.LongAdder8
>
> 34: 87277 2792864 java.lang.ref.WeakReference
>
> 35: 170663 2730608 java.util.concurrent.locks.
> ReentrantLock
>
> 36: 85112 2723584 org.apache.ignite.internal.
> processors.cache.GridCacheConcurrentMapImpl
>
> 37: 85112 2723584 org.apache.ignite.internal.
> util.GridCircularBuffer
>
> 38: 134255 2148080 org.apache.ignite.internal.
> processors.cache.CacheEvictableEntryImpl
>
> 39: 85304 2047296 java.util.concurrent.
> CopyOnWriteArrayList
>
> 40: 10900 1693000 [Lorg.jsr166.ConcurrentLinkedHashMap$
> HashEntry;
>
> 41: 96594 1545504 java.util.HashSet
>
> 42: 86086 1377376 java.util.concurrent.atomic.
> AtomicInteger
>
> 43: 85435 1366960 java.util.HashMap$KeySet
>
> 44: 85002 1360032 org.apache.ignite.internal.
> processors.query.h2.opt.GridH2AbstractKeyValueRow$WeakValue
>
> 45: 44295 1063080 java.util.ArrayList
>
> 46: 13513 864832 java.util.concurrent.ConcurrentHashMap
>
> 47: 21090 843600 org.apache.ignite.internal.
> processors.cache.version.GridCacheVersion
>
> 48: 160 653824 [Lorg.apache.ignite.internal.
> processors.cache.distributed.dht.GridDhtLocalPartition;
>
> 49: 5664 637120 java.lang.Class
>
> 50: 7047 620136 java.lang.reflect.Method
>
> 51: 10900 610400 org.jsr166.ConcurrentLinkedHashMap$
> Segment
>
> 52: 12528 601344 java.util.concurrent.locks.
> ReentrantReadWriteLock$NonfairSync
>
> 53: 11566 555168 org.h2.expression.ExpressionColumn
>
> 54: 16698 400752 java.lang.Long
>
> 55: 12522 400704 javax.management.MBeanAttributeInfo
>
> 56: 7878 381504 [I
>
> 57: 1767 310992 org.h2.command.dml.Select
>
> 58: 6162 295776 org.apache.ignite.internal.
> processors.affinity.GridAffinityAssignment
>
> 59: 7223 288920 java.util.LinkedHashMap$Entry
>
> 60: 9276 222624 org.h2.expression.ValueExpression
>
> 61: 6912 221184 org.h2.expression.Comparison
>
> 62: 3042 219024 java.lang.reflect.Field
>
> 63: 12529 200464 java.util.concurrent.locks.
> ReentrantReadWriteLock$ReadLock
>
> 64: 12529 200464 java.util.concurrent.locks.
> ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
>
> 65: 12529 200464 java.util.concurrent.locks.
> ReentrantReadWriteLock$WriteLock
>
> 66: 1767 197904 org.h2.table.TableFilter
>
> 67: 2116 186208 org.apache.ignite.internal.
> processors.jobmetrics.GridJobMetricsSnapshot
>
> 68: 1767 169632 org.h2.jdbc.JdbcPreparedStatement
>
> 69: 5145 164640 org.h2.expression.ConditionAndOr
>
> 70: 7674 162472 [Ljava.lang.Class;
>
> 71: 4654 148928 org.h2.expression.Alias
>
> 72: 1726 137328 [S
>
> 73: 1767 127224 org.h2.index.IndexCursor
>
> 74: 7800 124800 java.lang.Object
>
> 75: 1360 119680 org.apache.ignite.internal.util.
> StripedCompositeReadWriteLock$ReadLock
>
> 76: 1767 113088 org.h2.jdbc.JdbcResultSet
>
> 77: 1992 111552 java.util.LinkedHashMap
>
> 78: 4228 101472 org.h2.value.ValueLong
>
> 79: 2804 89728 java.lang.ThreadLocal$
> ThreadLocalMap$Entry
>
> 80: 309 88992 org.apache.ignite.configuration.
> CacheConfiguration
>
> 81: 1767 84816 org.h2.command.CommandContainer
>
> 82: 1172 84384 org.h2.expression.Aggregate
>
> 83: 777 80808 org.h2.table.Column
>
> 84: 2502 80064 java.util.concurrent.
> ConcurrentSkipListMap$HeadIndex
>
> 85: 2939 79912 [Lorg.h2.expression.Expression;
>
> 86: 1870 73152 [Lorg.h2.table.IndexColumn;
>
> 87: 1798 71920 java.lang.ref.SoftReference
>
> 88: 17 69904 [Ljava.nio.ByteBuffer;
>
> 89: 2166 69312 org.apache.ignite.internal.
> GridLoggerProxy
>
> 90: 1715 68600 org.h2.expression.Operation
>
> 91: 64 66560 [Lorg.apache.ignite.internal.
> processors.jobmetrics.GridJobMetricsSnapshot;
>
> 92: 20 65920 [Ljava.nio.channels.SelectionKey;
>
>
>
>
>
>
>
>
>
> *发件人**:* Shawn Du [mailto:shawn...@neulion.com.cn
> <shawn...@neulion.com.cn>]
> *发送时间**:* 2016年10月26日 13:25
> *收件人**:* user@ignite.apache.org
> *主题**:* ignite used too much memory
>
>
>
> Hi,
>
>
>
> In my ignite server, I have several caches, each cache has about 10k
> entries.
>
> I build the entry using binary object. Each entry just has 3 or 4 fields,
> each field is short, less than 20 bytes. But I enable index for each field.
>
> Most entry has set expired time. The expired time is short, about 90
> seconds.
>
> After run for 2 hours, 8G memory are used, and the ignite run out of
> memory.
>
> I build the ignite from source code, and use yesterdays’ github master
> branch code.
>
>
>
> My questions:
>
> 1) Does the expired data release the memory?
>
> 2) How ignite build the index? How much memory it cost?
>
>
>
> Thanks
>
> Shawn
>
>
>
>
>
>
>
>
>
