Re: Memory consumption in apache ignite

[dkarachentsev]

Hi,

The main point of doing that was not to give data classes to server, that it
would be unable to deserialize values on his size and force to use binary
format. But if server doesn't know classes (data classes are not available
in classpath) the only way to enable SQL queries is via QueryEntity. 

Anyway deserialized objects will be cached only when they were queried and
you can check with heap dump how many your object instances resides in
memory and who holds references on them.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9150.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[rishi007bansod]

Hi dkarachentsev,
I did loading of data from client node as suggested, but still I think
values are getting stored in both serialized and de-serialized format. I
have attached my heap dump. Which objects in this dump represents serialized
and de-serialized form. Also instead of QueryEntity, I did same indexing
configuration in my java code only, will this have any effect on memory
consumption? 

 



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9131.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[dkarachentsev]

Approximate calculation for custom object overhead on heap will look like:
entries = N
fields number = flds
header size = 24
fields id size = 4
BinaryObjectImpl = 40
byte arr header 16 + len 4 = 20 (at least, omitted padding)

N*(24 + flds*4 + 40 + 20) = 4*N*(21 + flds),

so for one entry with one field you'll get ~88 bytes overhead. But take in
account that this value is for user custom object, for primitives or
collections it will be much smaller.

Also, in CSV data stored as a strings, and the same value may take different
amount of memory:
"1" in CSV is 1 byte, when integer in memory takes 4 bytes and
"1234567" in CSV is 7 bytes, when in memory is still 4 bytes.

NOTE. Do not reference on that calculations as precise, because there a lot
of other objects and links associated with entry, result depends on actual
entry size, no optimizations are taken into account, for 32 bit system
values will differ, etc.

BinaryObjectImpl may hold deserialized value to cache and decrease number of
deserializations, and always holds serialized (byte array).



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9109.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[rishi007bansod]

Data is stored in CSV format on disk. So, I calculated CSV file size on disk.
 

Following are tables and number of fields in them
Table   Fields  Entries
warehouse 9 5
district  11   50
customer   21  15
history  83
new_order 345000
order815
order_line 10   148
item 5
10
stock17   50

are you saying for each field it will take 4 bytes overhead? So, total
overhead = total Entires*number of fields*4*2 bytes? Factor 2 is for
BinaryObjectImpl. Is this calculation correct? Also one another question,
BinaryObjectImpl are objects in deserialized format, right?  



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9107.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[dkarachentsev]

Please describe how data are stored on disk that you measured as 370 Mbytes?
Are they compressed? How many fields each entry has? The comparison may not
be correct.

Ignite stores entries in a different way than relational database, f.e. for
each field it spends 4 bytes, so if you have 10 fields, then you get at
least 40 bytes overhead, plus each BinaryObjectImpl takes additional 40
bytes, plus auxiliary objects for each entry. According to your heap dump
avg entry size is about 175 bytes, when entry in your file is about 80 bytes
(I tried approximately calculate size of byte array size), so it almost
twice larger and worth to recheck original data size.

Binary format is quite compact, but anyway it takes some additional space to
suite the cases it was designed for.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9083.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[rishi007bansod]

I did modifications as you suggested. But, still memory consumption is about
2 GB, what might be reason for this?

Memory consumption before data loading
 

Memory consumption after data loading
 



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9081.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[Dmitriy Karachentsev]

You don't have to deploy your data classes on server, because it may work
with binary types. Just configure cache as usual but use QueryEntity[1] for
that. On client side you may put and query for data as you did before.

[1]:
http://apacheignite.gridgain.org/docs/sql-queries#configuring-sql-indexes-using-queryentity

On Fri, Nov 18, 2016 at 1:23 PM, rishi007bansod 
wrote:

> I have measured database size on disk 370MB
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9073.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Memory consumption in apache ignite

[rishi007bansod]

I have measured database size on disk 370MB 



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9073.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[rishi007bansod]

But for data storage required data classes must be defined on server node.
What do you mean by "without data classes"? Can you give me example? 



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9071.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[Dmitriy Karachentsev]

For sure, if you disable indexing SQL query will not be available. It was
just recommendation if it's suitable for you.
You may try as Vlad suggested: use server node without data classes, so it
will not be able to cache deserialized values, and run your application on
client node.

On Fri, Nov 18, 2016 at 7:21 AM, rishi007bansod 
wrote:

> But now with indexing types disabled I am getting following error,
>
> Exception in thread "main" javax.cache.CacheException: Indexing is disabled
> for cache: order_line_cache. Use setIndexedTypes or setTypeMetadata methods
> on CacheConfiguration to enable.
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.validate(
> IgniteCacheProxy.java:732)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(
> IgniteCacheProxy.java:664)
> at data_test.order(data_test.java:2903)
> at data_test.main(data_test.java:2839)
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9064.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Memory consumption in apache ignite

[rishi007bansod]

But now with indexing types disabled I am getting following error,

Exception in thread "main" javax.cache.CacheException: Indexing is disabled
for cache: order_line_cache. Use setIndexedTypes or setTypeMetadata methods
on CacheConfiguration to enable.
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.validate(IgniteCacheProxy.java:732)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:664)
at data_test.order(data_test.java:2903)
at data_test.main(data_test.java:2839)




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9064.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[Dmitriy Karachentsev]

Yes, in your case Ignite stores deserialized values as well. You may try to
avoid it by switching copy on read to true, remove indexed types and
disable peer class loading if it's set.

On Thu, Nov 17, 2016 at 4:05 PM, rishi007bansod 
wrote:

> I have used single node. My cache configuration is,
> Cache configuration I have used is,
>
> CacheConfiguration ccfg_order_line = new
> CacheConfiguration<>();
> ccfg_order_line.setIndexedTypes(order_lineKey.class,
> order_line.class);
> ccfg_order_line.setName("order_line_cache");
> ccfg_order_line.setCopyOnRead(false);
> ccfg_order_line.setMemoryMode(CacheMemoryMode.ONHEAP_TIERED);
> ccfg_order_line.setSwapEnabled(false);
> ccfg_order_line.setBackups(0);
> IgniteCache cache_order_line =
> ignite.createCache(ccfg_order_line);
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9044.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Memory consumption in apache ignite

[vdpyatkov]

Hi,

How are you measure size of data (370MB)?
If you use ONHEAP_TIERED mode, entries can to store on in state (serialized
and deserialized), look at issues [1, 2], but you can avoid this behavior,
if server does not have object classes.

[1]: https://issues.apache.org/jira/browse/IGNITE-3347
[2]: https://issues.apache.org/jira/browse/IGNITE-2417



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9049.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[rishi007bansod]

I have used single node. My cache configuration is,
Cache configuration I have used is, 

CacheConfiguration ccfg_order_line = new
CacheConfiguration<>(); 
ccfg_order_line.setIndexedTypes(order_lineKey.class, order_line.class); 
ccfg_order_line.setName("order_line_cache"); 
ccfg_order_line.setCopyOnRead(false); 
ccfg_order_line.setMemoryMode(CacheMemoryMode.ONHEAP_TIERED); 
ccfg_order_line.setSwapEnabled(false); 
ccfg_order_line.setBackups(0); 
IgniteCache cache_order_line =
ignite.createCache(ccfg_order_line); 





--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9044.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[Dmitriy Karachentsev]

Hi.

How many nodes do you use? Could you provide configuration, please?

On Thu, Nov 17, 2016 at 11:50 AM, rishi007bansod 
wrote:

> For 370MB of data ignite is consuming about 3GB space in memory(ON HEAP
> MODE). Following is heap dump I got. What memory optimizations can be
> applied? Does ignite stores object by default in both serialized and
> deserialized formats?
>
> 
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>