Hi Pavel,
After doing some research, with QuerySQLField and Indexes, also with the heap
size,
it seems that with each QuerySQLField the amount of memory needed is doubled by
the size of the
field.
All test are done by 3 Windows 2012R2 Nodes with Apache.Ignite 2.2 heap size
1GB per java switch
7.5 GB Data ASCII Textfiles
64 million rows of data
Key is a guid data type
So my assumption was that I need a little more than 21GB
7.5 GB of ASCII data -> 15GB of UTF data
64 million keys of Guid -> 1 GB
Cache overhead 3 * 300MB -> 1.2 GB
Static assigned Heap 1GB per Node -> 3 GB
--------------------------------
Sum 20.2 GB
test 1
data load with No QueryEntities defined. -> 21.4 GB used memory
test 2
data load with QueryEntities defined but no [QuerySQLField] attributes. -> 22.2
GB used memory
test 3
data load with QueryEntities defined and [QuerySQLField] attributes. -> 38,7 GB
used memory
After running test 3 it seems that when I use QuerySQLField then the memory
needed is doubled
by my estimation. The difference between test 2 and test 3 is 16.5 GB of memory
needed.
src:
DataItem this is the only class which is used. It’s a simple class for only
getting the data into the db.
public class DataItem : IBinarizable
{
[QuerySqlField(IsIndexed = true)]
public DateTime DateTime;
[QuerySqlField]
public short FracSec;
[QuerySqlField(IsIndexed = true)]
public string EventType = "";
[QuerySqlField(IsIndexed = true)]
public string Category = "";
[QuerySqlField(IsIndexed = true)]
public string Area = "";
[QuerySqlField]
public string Node = "";
[QuerySqlField]
public string Unit = "";
[QuerySqlField(IsIndexed = true)]
public string Module = "";
[QuerySqlField]
public string Module_Description = "";
[QuerySqlField]
public string Attribute = "";
[QuerySqlField(IsIndexed = true)]
public string State = "";
[QuerySqlField(IsIndexed = true)]
public string Level = "";
[QuerySqlField]
public string Desc1 = "";
[QuerySqlField]
public string Desc2 = "";
[QuerySqlField]
public string Desc3 = "";
[QuerySqlField]
public string Desc4 = "";
public void ReadBinary(IBinaryReader reader)
{
Area = reader.ReadString("Area");
Attribute = reader.ReadString("Attribute");
Category =reader.ReadString("Category");
DateTime? tmp = reader.ReadTimestamp("DateTime");
if (tmp.HasValue)
DateTime = tmp.Value;
Desc1 =reader.ReadString("Desc1");
Desc2=reader.ReadString("Desc2");
Desc3 =reader.ReadString("Desc3");
Desc4=reader.ReadString("Desc4");
EventType =reader.ReadString("EventType" );
FracSec= reader.ReadShort("FracSec");
Level=reader.ReadString("Level");
Module=reader.ReadString("Module");
Module_Description=reader.ReadString("Module_Description");
Node=reader.ReadString("Node");
State=reader.ReadString("State");
Unit=reader.ReadString("Unit");
}
public void WriteBinary(IBinaryWriter writer)
{
writer.WriteString("Area", Area);
writer.WriteString("Attribute", Attribute);
writer.WriteString("Category", Category);
writer.WriteTimestamp("DateTime", DateTime);
writer.WriteString("Desc1", Desc1);
writer.WriteString("Desc2", Desc2);
writer.WriteString("Desc3", Desc3);
writer.WriteString("Desc4", Desc4);
writer.WriteString("EventType", EventType);
writer.WriteShort("FracSec", FracSec);
writer.WriteString("Level", Level);
writer.WriteString("Module", Module);
writer.WriteString("Module_Description", Module_Description);
writer.WriteString("Node", Node);
writer.WriteString("State", State);
writer.WriteString("Unit", Unit);
}
}
Apache.Ignite.exe.config addition:
<igniteConfiguration
xmlns="http://ignite.apache.org/schema/dotnet/IgniteConfigurationSection";>
<!-- Customize Ignite configuration here. -->
<memoryConfiguration defaultMemoryPolicyName="meinTest">
<memoryPolicies>
<memoryPolicyConfiguration name="meinTest" initialSize="5368709120"
maxSize="10737418240"/>
</memoryPolicies>
</memoryConfiguration>
</igniteConfiguration>
DataLoader code cache configuration:
CacheConfiguration myAECache = new CacheConfiguration() ;
myAECache.CacheMode = CacheMode.Partitioned;
myAECache.WriteThrough = false;
myAECache.ReadThrough = false;
myAECache.Backups = 0;
myAECache.Name = Name + ".Events";
myAECache.QueryEntities = new List<QueryEntity>() { new
QueryEntity(typeof(DataItem)) };
myAECache.WriteSynchronizationMode = CacheWriteSynchronizationMode.FullAsync;
myAECache.AtomicityMode = CacheAtomicityMode.Atomic;
ICache<Guid, DataItem> AECache = db.CreateCache<Guid, DataItem>(myAECache);
var ldr = db.GetDataStreamer<Guid, DataItem>(myAECache.Name);
Thanks,
Mario
From: Pavel Tupitsyn [mailto:[email protected]]
Sent: Thursday, October 19, 2017 16:59
To: [email protected]
Subject: Re: Question Ignite Memory Consumption / Object size (Apache Ignite
.NET)
Hi Mario,
See
https://apacheignite.readme.io/docs/capacity-planning<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapacheignite.readme.io%2Fdocs%2Fcapacity-planning&data=02%7C01%7CMElmers%40dow.com%7Cca42f7cc3d3140b4bbfb08d517020f02%7Cc3e32f53cb7f4809968d1cc4ccc785fe%7C0%7C0%7C636440219977766192&sdata=NyeKbTZ06Lqbi8HYZ2uxsj8UfPD22fxp29h3YCI0ACs%3D&reserved=0>
> Each field I have marked with the attribute [QuerySqlField] and some fields
> are indexed
This is most likely the case.
1) Have you tried loading data without enabling Ignite SQL (e.g. do not
configure CacheConfiguration.QueryEntities)?
2) Can you attach the class? How many fields are there?
Thanks,
Pavel
On Thu, Oct 19, 2017 at 1:57 PM, Elmers, Mario (M)
<[email protected]<mailto:[email protected]>> wrote:
Hello,
I try to estimate the needed RAM for my application. I have created 3 nodes by
starting only the Apache.Ignite.exe.
All is done with C# and Apache.Ignite 2.2
Then I created a data loader application which loads up the whole data of my
logfiles.
The size of all my logfiles are 7.5 GB. When I loaded it up to the Ignite
cluster all together need more than 32GB of RAM.
My cache is configured as partioned with 0 backups. So I hat thinked that the
cluster will need not much more than 16 GB of RAM.
Because the file are normal ASCII files which converted to UTF-8 it will twice
the amount of data needed to store.
The class file I have created has for each field of the logentry one field.
Each field I have marked with the attribute [QuerySqlField] and some fields
are indexed.
The key is of type Guid.
Can someone explain why the amount is 4x greater than the raw data ?
Thanks & regards
Mario
