Re: Cache size in offheap mem in bytes

[Alex Plehanov]

It's incorrect to use cache object to calculate cache data size. What you
got now is a footprint of the Ignite infrastructure used to manage your
data, but not a footprint of your data itself, since data are stored in
off-heap and this tool only calculate on-heap size of objects referenced by
cache object. As you can see in your sample 1 000 000 objects was created
with 5 longs each, but you got only  2 872 685 longs in footprint.

To roughly estimate off-heap memory needed for cache data you can enable
persistent, put large amount of entries to the cache and then measure size
of cache folder (you should also wait until checkpoint complete to get
correct results). Also page in off-heap memory have some extra header, so
final off-heap memory needed will be a little bit larger then cache folder
size.


2018-06-27 17:55 GMT+03:00 Prasad Bhalerao :

> Hi,
>
> I have written a test program to find the cache size in bytes. I am using
> this  tool
> (memory-measurer) to find the object sizes and its foot prints. I tried to
> use this tool to find the cache size in bytes. I am using on heap cache.
>
> Am I using the correct object (cache in this case) to check the cache size
> in memory? Can some please clarify?
>
> I tried to use Jprofiler in instrumentation mode to check on-heap cache
> size, but could not locate the exact class/objects.
>
> private void pushData(CacheName cacheName, List DataKey>> datas) {
>
>   final IgniteCache cache = ignite.cache(cacheName.name());
>
>   LOGGER.info("cache size in bytes : {}", 
> IpV4RangeTest.bytesToHuman(MemoryMeasurer.measureBytes(cache)));
>
>   TestDataObj data8 = null;
>   for(int ctr=0;ctr<1_000_000;ctr++){
> data8 = new TestDataObj();
> data8.setId(ctr);
> data8.setSubscriptionId(SUBSCRIPTION_ID);
> cache.put(data8.getKey(), data8);
>   }
>
>   LOGGER.info("TestDataObj size in bytes : {}",
>   IpV4RangeTest.bytesToHuman(MemoryMeasurer.measureBytes(data8)));
>
>   LOGGER.info("cache size in bytes : {}",
>   IpV4RangeTest.bytesToHuman(MemoryMeasurer.measureBytes(cache)));
>
>   Footprint footprint = ObjectGraphMeasurer.measure(cache);
>
>   LOGGER.info("{} Footprint={}", cacheName.name(), footprint.toString());
>
>   LOGGER.info("{} size={}", cacheName.name(), cache.size());
>   try {
> Thread.sleep(10);
>   } catch (InterruptedException e) {
> e.printStackTrace();
>   }
> }
>
>
>
> *  Output:*
>
>   *cache size in bytes* : 36.4 MB  *[Empty Cache]*
>
> * TestDataObj size in bytes : 64 byte  cache size in bytes* : 493.23 MB
>   *[After adding 1 million objects]*
>   *Footprint*=Footprint
>   [objects=10962325,
>   references=30951850,
>   primitives={byte=130277671, long=2872685, double=58, float=52015,
> int=11651118, boolean=2156105, char=1905788, short=10457}]
>
>
> *Inside TestDataObj  class:*
>
> public class TestDataObj implements Data {
>
>   @QuerySqlField
>   private long id;
>
>   @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = 
> "ag_domain_nb_override_data", order = 1)})
>   private long assetGroupDomainId;
>
>   @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = 
> "ag_domain_nb_override_data", order = 2)})
>   private long startIp;
>
>   @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = 
> "ag_domain_nb_override_data", order = 3)})
>   private long endIp;
>
>   private long subscriptionId;
>
>   @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = 
> "ag_domain_nb_override_data", order = 4)})
>   private int partitionId;
>
>   private boolean isUpdatedData;
>
> }
>
>
>
> Thanks,
> PRasad
>
> On Wed, Jun 27, 2018 at 3:00 PM dkarachentsev 
> wrote:
>
>> 1) This applicable to Ignite. As it grown from GridGain sometimes it may
>> appear in docs, because missed fro removal.
>> 2) Yes, and I would say overhead could be even bigger. But anyway I cannot
>> say definitely how much, because Ignite doesn't store data sequentially,
>> there a lot of nuances.
>> 3) Ignite transaction caches entries on-heap and this works only for
>> TRANSACTIONAL caches.
>>
>> Thanks!
>> -Dmitry
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Cache size in offheap mem in bytes

[Prasad Bhalerao]

Hi,

I have written a test program to find the cache size in bytes. I am using
this  tool (memory-measurer)
to find the object sizes and its foot prints. I tried to use this tool to
find the cache size in bytes. I am using on heap cache.

Am I using the correct object (cache in this case) to check the cache size
in memory? Can some please clarify?

I tried to use Jprofiler in instrumentation mode to check on-heap cache
size, but could not locate the exact class/objects.

private void pushData(CacheName cacheName, List> datas) {

  final IgniteCache cache = ignite.cache(cacheName.name());

  LOGGER.info("cache size in bytes : {}",
IpV4RangeTest.bytesToHuman(MemoryMeasurer.measureBytes(cache)));

  TestDataObj data8 = null;
  for(int ctr=0;ctr<1_000_000;ctr++){
data8 = new TestDataObj();
data8.setId(ctr);
data8.setSubscriptionId(SUBSCRIPTION_ID);
cache.put(data8.getKey(), data8);
  }

  LOGGER.info("TestDataObj size in bytes : {}",
  IpV4RangeTest.bytesToHuman(MemoryMeasurer.measureBytes(data8)));

  LOGGER.info("cache size in bytes : {}",
  IpV4RangeTest.bytesToHuman(MemoryMeasurer.measureBytes(cache)));

  Footprint footprint = ObjectGraphMeasurer.measure(cache);

  LOGGER.info("{} Footprint={}", cacheName.name(), footprint.toString());

  LOGGER.info("{} size={}", cacheName.name(), cache.size());
  try {
Thread.sleep(10);
  } catch (InterruptedException e) {
e.printStackTrace();
  }
}



*  Output:*

  *cache size in bytes* : 36.4 MB  *[Empty Cache]*

* TestDataObj size in bytes : 64 byte  cache size in bytes* : 493.23 MB
*[After adding 1 million objects]*
  *Footprint*=Footprint
  [objects=10962325,
  references=30951850,
  primitives={byte=130277671, long=2872685, double=58, float=52015,
int=11651118, boolean=2156105, char=1905788, short=10457}]


*Inside TestDataObj  class:*

public class TestDataObj implements Data {

  @QuerySqlField
  private long id;

  @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ag_domain_nb_override_data", order = 1)})
  private long assetGroupDomainId;

  @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ag_domain_nb_override_data", order = 2)})
  private long startIp;

  @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ag_domain_nb_override_data", order = 3)})
  private long endIp;

  private long subscriptionId;

  @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ag_domain_nb_override_data", order = 4)})
  private int partitionId;

  private boolean isUpdatedData;

}



Thanks,
PRasad

On Wed, Jun 27, 2018 at 3:00 PM dkarachentsev 
wrote:

> 1) This applicable to Ignite. As it grown from GridGain sometimes it may
> appear in docs, because missed fro removal.
> 2) Yes, and I would say overhead could be even bigger. But anyway I cannot
> say definitely how much, because Ignite doesn't store data sequentially,
> there a lot of nuances.
> 3) Ignite transaction caches entries on-heap and this works only for
> TRANSACTIONAL caches.
>
> Thanks!
> -Dmitry
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Cache size in offheap mem in bytes

[dkarachentsev]

1) This applicable to Ignite. As it grown from GridGain sometimes it may
appear in docs, because missed fro removal.
2) Yes, and I would say overhead could be even bigger. But anyway I cannot
say definitely how much, because Ignite doesn't store data sequentially,
there a lot of nuances.
3) Ignite transaction caches entries on-heap and this works only for
TRANSACTIONAL caches.

Thanks!
-Dmitry



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Cache size in offheap mem in bytes

[Prasad Bhalerao]

Hi,

Thank you for the explanation.

As per this link
https://apacheignite.readme.io/docs/capacity-planning#section-calculating-memory-usage
,

"GridGain will typically add around 200 bytes overhead to each entry."

1) Why does it specifically say GridGain will add around 200 bytes overhead
for each entry?
Does ignite not add this overhead?

3) Does this

example already considers 200 bytes overhead per entry?

4) When ignite transaction is started, it keeps the cache updates in temp
cache or transactional cache and it is kept in on-heap.
Does this rule apply to transactional cache as well?



Regard,
Prasad

On Wed, Jun 27, 2018 at 2:15 PM dkarachentsev 
wrote:

> Hi,
>
> It will be incorrect, because entries are not stored sequentially, there
> are
> a lot of infrastructure that requires additional space. For example, memory
> is divided into pages and each page has header, each entry has key and
> value, version and other service information. For quick access to entries
> is
> created b+tree (which consists of pages too), synchronization primitives,
> links and binary object overhead to allow access to it's fields.
>
> It's quire hard to say how much memory will be required, just approximately
> [1]. The best way will be to load some number of data and measure how much
> memory was consumed. You need to make many such measures, because only on
> large number of entries memory consumption will increase linearly.
>
> [1] https://apacheignite.readme.io/docs/capacity-planning
>
> Thanks!
> -Dmitry
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>