Re: Ignite cache failed to read by key intermittently
Hi,
I already attached server config in the same mail thread( along with sample
code).
Regards,
Charlin
On Tue, 20 Sept 2022 at 11:50, Pavel Tupitsyn wrote:
> Now the client starts, but cannot connect to a server. How do I start a
> server, do you use a config file or something else?
>
> On Tue, Sep 20, 2022 at 9:15 AM Charlin S wrote:
>
>> Hi,
>> Sorry for inconvenience, I have removed local machine path dependency.
>>
>> Regards,
>> Charlin
>>
>>
>> On Tue, 20 Sept 2022 at 11:04, Pavel Tupitsyn
>> wrote:
>>
>>> Still can't run, sorry. It has hardcoded directories from your machine
>>> (IgniteSettings.IGNITE_HOME, IgniteSettings.WorkDirectory - commented them
>>> out), and also GlobalVariables.IgniteSettings.ConfigurationPath:
>>>
>>> > Spring XML configuration path is invalid:
>>> D:\NGSourceCode\IgniteConfig\Labs/common_dynamiccache_client_config.xml
>>>
>>> Please make sure the zip file has everything required to run the
>>> application on any machine.
>>>
>>> On Mon, Sep 19, 2022 at 5:07 PM Charlin S
>>> wrote:
>>>
Hi,
I have attached here.
Regards,
Charlin
On Mon, 19 Sept 2022 at 19:00, Pavel Tupitsyn
wrote:
> I get an exception when trying to run the code:
>
> Unhandled exception. System.NullReferenceException: Object reference
> not set to an instance of an object.
>at ConsoleApp4.IgniteInstance..ctor() in
> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/IgniteInstance.cs:line
> 47
>at ConsoleApp4.IgniteInstance.get_Instance() in
> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/IgniteInstance.cs:line
> 33
>at ConsoleApp4.UnitOfWork..ctor() in
> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/UnitOfWork.cs:line
> 15
>at ConsoleApp4.Program.Main(String[] args) in
> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/Program.cs:line
> 12
>
> On Mon, Sep 19, 2022 at 3:04 PM Charlin S
> wrote:
>
>> Hi,
>> Cache EexpiryPolicy is 60 minutes. but cache key failing in between
>> and return data after some time so its not due to expiry policy.
>> I have prepared a separate solution and attached it here. I am
>> sharing a cache data reading time screenshot for your reference .
>> [image: image.png]
>> Regards,
>> Charlin
>>
>>
>>
>>
>>
>> On Mon, 19 Sept 2022 at 11:00, Pavel Tupitsyn
>> wrote:
>>
>>> The code you provided above uses ExpiryPolicy. It might be the
>>> reason for TryGet to return false - cache entry expires.
>>> To provide a definitive answer we need a project that runs and
>>> demonstrates the problem.
>>>
>>> On Fri, Sep 16, 2022 at 5:12 PM Charlin S
>>> wrote:
>>>
Hi,
sorry for inconvenience. It's 90% actual code only. I have changed
cache name and grid name as per policy. I have to prepare a separate
solution and provide you if required.
Regards,
Charlin
On Fri, 16 Sept 2022 at 19:11, Pavel Tupitsyn
wrote:
> It does not have to be your actual code, just something that
> demonstrates the problem
>
> On Fri, Sep 16, 2022 at 4:40 PM Pavel Tupitsyn <
> ptupit...@apache.org> wrote:
>
>> Can you please share a complete project that I can run and see
>> the issue?
>>
>> On Fri, Sep 16, 2022 at 4:30 PM Charlin S
>> wrote:
>>
>>> Hi,
>>> Sorry I am not able to share a complete solution. bean xml
>>> attached and code snippet in below
>>> IgniteConfiguration igniteGridIg = new IgniteConfiguration();
>>> igniteGridIg.AutoGenerateIgniteInstanceName = true;
>>> if
>>> (!string.IsNullOrEmpty(GlobalVariables.IgniteSettings.IGNITE_HOME))
>>> {
>>> igniteGridIg.IgniteHome =
>>> GlobalVariables.IgniteSettings.IGNITE_HOME;
>>> }
>>> var gridDetails =
>>> GlobalVariables.IgniteSettings.GridDetails.Where(gd =>
>>> gd.GridType.Equals("DynamicGrid",
>>> StringComparison.OrdinalIgnoreCase)).FirstOrDefault();
>>> //setting ignite working directory if not ignite
>>> will run with default work directory
>>> if
>>> (!string.IsNullOrEmpty(GlobalVariables.IgniteSettings.WorkDirectory))
>>> {
>>> igniteGridIg.WorkDirectory =
>>> Path.Combine(GlobalVariables.IgniteSettings.WorkDirectory,
>>> gridDetails.CacheType);
>>> }
>>> if (null != gridDetails)
>>> {
>>> GridName = gridDetails.GridName;
>>> igniteGridIg.SpringCo
Re: Ignite cache failed to read by key intermittently
Now the client starts, but cannot connect to a server. How do I start a
server, do you use a config file or something else?
On Tue, Sep 20, 2022 at 9:15 AM Charlin S wrote:
> Hi,
> Sorry for inconvenience, I have removed local machine path dependency.
>
> Regards,
> Charlin
>
>
> On Tue, 20 Sept 2022 at 11:04, Pavel Tupitsyn
> wrote:
>
>> Still can't run, sorry. It has hardcoded directories from your machine
>> (IgniteSettings.IGNITE_HOME, IgniteSettings.WorkDirectory - commented them
>> out), and also GlobalVariables.IgniteSettings.ConfigurationPath:
>>
>> > Spring XML configuration path is invalid:
>> D:\NGSourceCode\IgniteConfig\Labs/common_dynamiccache_client_config.xml
>>
>> Please make sure the zip file has everything required to run the
>> application on any machine.
>>
>> On Mon, Sep 19, 2022 at 5:07 PM Charlin S wrote:
>>
>>> Hi,
>>> I have attached here.
>>> Regards,
>>> Charlin
>>>
>>>
>>> On Mon, 19 Sept 2022 at 19:00, Pavel Tupitsyn
>>> wrote:
>>>
I get an exception when trying to run the code:
Unhandled exception. System.NullReferenceException: Object reference
not set to an instance of an object.
at ConsoleApp4.IgniteInstance..ctor() in
/home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/IgniteInstance.cs:line
47
at ConsoleApp4.IgniteInstance.get_Instance() in
/home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/IgniteInstance.cs:line
33
at ConsoleApp4.UnitOfWork..ctor() in
/home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/UnitOfWork.cs:line
15
at ConsoleApp4.Program.Main(String[] args) in
/home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/Program.cs:line
12
On Mon, Sep 19, 2022 at 3:04 PM Charlin S
wrote:
> Hi,
> Cache EexpiryPolicy is 60 minutes. but cache key failing in between
> and return data after some time so its not due to expiry policy.
> I have prepared a separate solution and attached it here. I am sharing
> a cache data reading time screenshot for your reference .
> [image: image.png]
> Regards,
> Charlin
>
>
>
>
>
> On Mon, 19 Sept 2022 at 11:00, Pavel Tupitsyn
> wrote:
>
>> The code you provided above uses ExpiryPolicy. It might be the reason
>> for TryGet to return false - cache entry expires.
>> To provide a definitive answer we need a project that runs and
>> demonstrates the problem.
>>
>> On Fri, Sep 16, 2022 at 5:12 PM Charlin S
>> wrote:
>>
>>> Hi,
>>> sorry for inconvenience. It's 90% actual code only. I have changed
>>> cache name and grid name as per policy. I have to prepare a separate
>>> solution and provide you if required.
>>>
>>> Regards,
>>> Charlin
>>>
>>>
>>>
>>> On Fri, 16 Sept 2022 at 19:11, Pavel Tupitsyn
>>> wrote:
>>>
It does not have to be your actual code, just something that
demonstrates the problem
On Fri, Sep 16, 2022 at 4:40 PM Pavel Tupitsyn <
ptupit...@apache.org> wrote:
> Can you please share a complete project that I can run and see
> the issue?
>
> On Fri, Sep 16, 2022 at 4:30 PM Charlin S
> wrote:
>
>> Hi,
>> Sorry I am not able to share a complete solution. bean xml
>> attached and code snippet in below
>> IgniteConfiguration igniteGridIg = new IgniteConfiguration();
>> igniteGridIg.AutoGenerateIgniteInstanceName = true;
>> if
>> (!string.IsNullOrEmpty(GlobalVariables.IgniteSettings.IGNITE_HOME))
>> {
>> igniteGridIg.IgniteHome =
>> GlobalVariables.IgniteSettings.IGNITE_HOME;
>> }
>> var gridDetails =
>> GlobalVariables.IgniteSettings.GridDetails.Where(gd =>
>> gd.GridType.Equals("DynamicGrid",
>> StringComparison.OrdinalIgnoreCase)).FirstOrDefault();
>> //setting ignite working directory if not ignite will
>> run with default work directory
>> if
>> (!string.IsNullOrEmpty(GlobalVariables.IgniteSettings.WorkDirectory))
>> {
>> igniteGridIg.WorkDirectory =
>> Path.Combine(GlobalVariables.IgniteSettings.WorkDirectory,
>> gridDetails.CacheType);
>> }
>> if (null != gridDetails)
>> {
>> GridName = gridDetails.GridName;
>> igniteGridIg.SpringConfigUrl =
>> Path.Combine(GlobalVariables.IgniteSettings.ConfigurationPath,
>> gridDetails.SpringXMLFileName);
>> igniteGridIg.ConsistentId =
>> Guid.NewGuid().ToString().ToUpper();
>> ICollection lstJvmOptions = null;
>>
Re: Ignite cache failed to read by key intermittently
Hi,
Sorry for inconvenience, I have removed local machine path dependency.
Regards,
Charlin
On Tue, 20 Sept 2022 at 11:04, Pavel Tupitsyn wrote:
> Still can't run, sorry. It has hardcoded directories from your machine
> (IgniteSettings.IGNITE_HOME, IgniteSettings.WorkDirectory - commented them
> out), and also GlobalVariables.IgniteSettings.ConfigurationPath:
>
> > Spring XML configuration path is invalid:
> D:\NGSourceCode\IgniteConfig\Labs/common_dynamiccache_client_config.xml
>
> Please make sure the zip file has everything required to run the
> application on any machine.
>
> On Mon, Sep 19, 2022 at 5:07 PM Charlin S wrote:
>
>> Hi,
>> I have attached here.
>> Regards,
>> Charlin
>>
>>
>> On Mon, 19 Sept 2022 at 19:00, Pavel Tupitsyn
>> wrote:
>>
>>> I get an exception when trying to run the code:
>>>
>>> Unhandled exception. System.NullReferenceException: Object reference not
>>> set to an instance of an object.
>>>at ConsoleApp4.IgniteInstance..ctor() in
>>> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/IgniteInstance.cs:line
>>> 47
>>>at ConsoleApp4.IgniteInstance.get_Instance() in
>>> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/IgniteInstance.cs:line
>>> 33
>>>at ConsoleApp4.UnitOfWork..ctor() in
>>> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/UnitOfWork.cs:line
>>> 15
>>>at ConsoleApp4.Program.Main(String[] args) in
>>> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/Program.cs:line 12
>>>
>>> On Mon, Sep 19, 2022 at 3:04 PM Charlin S
>>> wrote:
>>>
Hi,
Cache EexpiryPolicy is 60 minutes. but cache key failing in between and
return data after some time so its not due to expiry policy.
I have prepared a separate solution and attached it here. I am sharing
a cache data reading time screenshot for your reference .
[image: image.png]
Regards,
Charlin
On Mon, 19 Sept 2022 at 11:00, Pavel Tupitsyn
wrote:
> The code you provided above uses ExpiryPolicy. It might be the reason
> for TryGet to return false - cache entry expires.
> To provide a definitive answer we need a project that runs and
> demonstrates the problem.
>
> On Fri, Sep 16, 2022 at 5:12 PM Charlin S
> wrote:
>
>> Hi,
>> sorry for inconvenience. It's 90% actual code only. I have changed
>> cache name and grid name as per policy. I have to prepare a separate
>> solution and provide you if required.
>>
>> Regards,
>> Charlin
>>
>>
>>
>> On Fri, 16 Sept 2022 at 19:11, Pavel Tupitsyn
>> wrote:
>>
>>> It does not have to be your actual code, just something that
>>> demonstrates the problem
>>>
>>> On Fri, Sep 16, 2022 at 4:40 PM Pavel Tupitsyn
>>> wrote:
>>>
Can you please share a complete project that I can run and see
the issue?
On Fri, Sep 16, 2022 at 4:30 PM Charlin S
wrote:
> Hi,
> Sorry I am not able to share a complete solution. bean xml
> attached and code snippet in below
> IgniteConfiguration igniteGridIg = new IgniteConfiguration();
> igniteGridIg.AutoGenerateIgniteInstanceName = true;
> if
> (!string.IsNullOrEmpty(GlobalVariables.IgniteSettings.IGNITE_HOME))
> {
> igniteGridIg.IgniteHome =
> GlobalVariables.IgniteSettings.IGNITE_HOME;
> }
> var gridDetails =
> GlobalVariables.IgniteSettings.GridDetails.Where(gd =>
> gd.GridType.Equals("DynamicGrid",
> StringComparison.OrdinalIgnoreCase)).FirstOrDefault();
> //setting ignite working directory if not ignite will
> run with default work directory
> if
> (!string.IsNullOrEmpty(GlobalVariables.IgniteSettings.WorkDirectory))
> {
> igniteGridIg.WorkDirectory =
> Path.Combine(GlobalVariables.IgniteSettings.WorkDirectory,
> gridDetails.CacheType);
> }
> if (null != gridDetails)
> {
> GridName = gridDetails.GridName;
> igniteGridIg.SpringConfigUrl =
> Path.Combine(GlobalVariables.IgniteSettings.ConfigurationPath,
> gridDetails.SpringXMLFileName);
> igniteGridIg.ConsistentId =
> Guid.NewGuid().ToString().ToUpper();
> ICollection lstJvmOptions = null;
>
> //"IgniteJVMOption":
> "-XX:+DisableExplicitGC,-XX:+UseG1GC,-Xms1g,-Xmx2g"
> if
> (string.IsNullOrEmpty(gridDetails.IgniteJVMOption))
> {
> lstJvmOptions =
> GlobalVariables.IgniteSettings.IgniteJVMOptionDefault.Split(',').Where(x
>>>
Re: Ignite cache failed to read by key intermittently
Still can't run, sorry. It has hardcoded directories from your machine
(IgniteSettings.IGNITE_HOME, IgniteSettings.WorkDirectory - commented them
out), and also GlobalVariables.IgniteSettings.ConfigurationPath:
> Spring XML configuration path is invalid:
D:\NGSourceCode\IgniteConfig\Labs/common_dynamiccache_client_config.xml
Please make sure the zip file has everything required to run the
application on any machine.
On Mon, Sep 19, 2022 at 5:07 PM Charlin S wrote:
> Hi,
> I have attached here.
> Regards,
> Charlin
>
>
> On Mon, 19 Sept 2022 at 19:00, Pavel Tupitsyn
> wrote:
>
>> I get an exception when trying to run the code:
>>
>> Unhandled exception. System.NullReferenceException: Object reference not
>> set to an instance of an object.
>>at ConsoleApp4.IgniteInstance..ctor() in
>> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/IgniteInstance.cs:line
>> 47
>>at ConsoleApp4.IgniteInstance.get_Instance() in
>> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/IgniteInstance.cs:line
>> 33
>>at ConsoleApp4.UnitOfWork..ctor() in
>> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/UnitOfWork.cs:line
>> 15
>>at ConsoleApp4.Program.Main(String[] args) in
>> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/Program.cs:line 12
>>
>> On Mon, Sep 19, 2022 at 3:04 PM Charlin S wrote:
>>
>>> Hi,
>>> Cache EexpiryPolicy is 60 minutes. but cache key failing in between and
>>> return data after some time so its not due to expiry policy.
>>> I have prepared a separate solution and attached it here. I am sharing a
>>> cache data reading time screenshot for your reference .
>>> [image: image.png]
>>> Regards,
>>> Charlin
>>>
>>>
>>>
>>>
>>>
>>> On Mon, 19 Sept 2022 at 11:00, Pavel Tupitsyn
>>> wrote:
>>>
The code you provided above uses ExpiryPolicy. It might be the reason
for TryGet to return false - cache entry expires.
To provide a definitive answer we need a project that runs and
demonstrates the problem.
On Fri, Sep 16, 2022 at 5:12 PM Charlin S
wrote:
> Hi,
> sorry for inconvenience. It's 90% actual code only. I have changed
> cache name and grid name as per policy. I have to prepare a separate
> solution and provide you if required.
>
> Regards,
> Charlin
>
>
>
> On Fri, 16 Sept 2022 at 19:11, Pavel Tupitsyn
> wrote:
>
>> It does not have to be your actual code, just something that
>> demonstrates the problem
>>
>> On Fri, Sep 16, 2022 at 4:40 PM Pavel Tupitsyn
>> wrote:
>>
>>> Can you please share a complete project that I can run and see
>>> the issue?
>>>
>>> On Fri, Sep 16, 2022 at 4:30 PM Charlin S
>>> wrote:
>>>
Hi,
Sorry I am not able to share a complete solution. bean xml
attached and code snippet in below
IgniteConfiguration igniteGridIg = new IgniteConfiguration();
igniteGridIg.AutoGenerateIgniteInstanceName = true;
if
(!string.IsNullOrEmpty(GlobalVariables.IgniteSettings.IGNITE_HOME))
{
igniteGridIg.IgniteHome =
GlobalVariables.IgniteSettings.IGNITE_HOME;
}
var gridDetails =
GlobalVariables.IgniteSettings.GridDetails.Where(gd =>
gd.GridType.Equals("DynamicGrid",
StringComparison.OrdinalIgnoreCase)).FirstOrDefault();
//setting ignite working directory if not ignite will
run with default work directory
if
(!string.IsNullOrEmpty(GlobalVariables.IgniteSettings.WorkDirectory))
{
igniteGridIg.WorkDirectory =
Path.Combine(GlobalVariables.IgniteSettings.WorkDirectory,
gridDetails.CacheType);
}
if (null != gridDetails)
{
GridName = gridDetails.GridName;
igniteGridIg.SpringConfigUrl =
Path.Combine(GlobalVariables.IgniteSettings.ConfigurationPath,
gridDetails.SpringXMLFileName);
igniteGridIg.ConsistentId =
Guid.NewGuid().ToString().ToUpper();
ICollection lstJvmOptions = null;
//"IgniteJVMOption":
"-XX:+DisableExplicitGC,-XX:+UseG1GC,-Xms1g,-Xmx2g"
if
(string.IsNullOrEmpty(gridDetails.IgniteJVMOption))
{
lstJvmOptions =
GlobalVariables.IgniteSettings.IgniteJVMOptionDefault.Split(',').Where(x
=>
!string.IsNullOrWhiteSpace(x)).ToList();
}
else
lstJvmOptions =
gridDetails.IgniteJVMOption.Split(',').Where(x =>
!string.IsNullOrWhiteSpace(x)).ToList();
Re[4]: What is data-streamer-stripe threasd?
It`s up to you, if it not annoying you leave it as it is and fill otherwise ) >Nah, it's fine just wanted to make sure what it was. Unless you think I should >log at least an issue? > >On Wed, Sep 14, 2022 at 3:13 AM Zhenya Stanilovsky via user < >user@ignite.apache.org > wrote: >>Yep, i already mention that you can`t disable this pool at all and 1 worker >>thread still be visible. >>You can fill the issue but i can`t guarantee that it would be completed soon, >>or can do it yourself and present pull request. >> >>best. >> >>>Ok so just to understand on the client side. Set the pool size for data >>>streamer to 1. >>> >>>But it will still look blocked? >>>On Mon., Sep. 12, 2022, 8:59 a.m. Zhenya Stanilovsky via user, < >>>user@ignite.apache.org > wrote: John, seems all you can here is just to set this pool size into «1» , «0» — tends to error. https://ignite.apache.org/docs/latest/data-streaming#configuring-data-streamer-thread-pool-size 1 thread will still be frozen in such a case. > >> >>>Hi I'm profiling my application through YourKit and it indicates that a >>>bunch of these threads (data-streamer-stripe) are "frozen" for 21 days. >>>This >>> >>>I'm not using data streaming, is there a way to disable it or just >>>ignore the messages? The application is configured as thick client >>>(client = true) >> >> >> >> >> >> >> >>
Re: Re[2]: What is data-streamer-stripe threasd?
Nah, it's fine just wanted to make sure what it was. Unless you think I should log at least an issue? On Wed, Sep 14, 2022 at 3:13 AM Zhenya Stanilovsky via user < user@ignite.apache.org> wrote: > Yep, i already mention that you can`t disable this pool at all and 1 > worker thread still be visible. > You can fill the issue but i can`t guarantee that it would be completed > soon, or can do it yourself and present pull request. > > best. > > > Ok so just to understand on the client side. Set the pool size for data > streamer to 1. > > But it will still look blocked? > > On Mon., Sep. 12, 2022, 8:59 a.m. Zhenya Stanilovsky via user, < > user@ignite.apache.org > > wrote: > > John, seems all you can here is just to set this pool size into «1» , «0» > — tends to error. > > > https://ignite.apache.org/docs/latest/data-streaming#configuring-data-streamer-thread-pool-size > > 1 thread will still be frozen in such a case. > > > > > > > Hi I'm profiling my application through YourKit and it indicates that a > bunch of these threads (data-streamer-stripe) are "frozen" for 21 days. > This > > I'm not using data streaming, is there a way to disable it or just ignore > the messages? The application is configured as thick client (client = true) > > > > > > > > > > >
Re: How to calculate the amount of memory used by a cache or dataregion
We know Ignite has the function of automatic memory defragmentation: https://ignite.apache.org/docs/latest/memory-architecture#memory-defragmentation I want to ask what is the threshold for triggering this function? 在 2022/9/19 22:36, Alexander Polovtcev 写道: Hm, I don't know, honestly. We need to look at the code or somebody else can give us a hint. On Mon, Sep 19, 2022 at 2:55 PM 38797715 <38797...@qq.com> wrote: if native persistence enabled, the result is the same. It seems that there is a bug in the calculation of the "PagesFillFactor" metrics? 在 2022/9/19 17:30, Alexander Polovtcev 写道: Do you have native persistence enabled? On Mon, Sep 19, 2022 at 11:57 AM 38797715 <38797...@qq.com> wrote: 2.10 and 2.13, the same result. 在 2022/9/19 16:54, Alexander Polovtcev 写道: Which version of Ignite do you use? On Mon, Sep 19, 2022 at 11:38 AM 38797715 <38797...@qq.com> wrote: for example: start a node by ignite.sh. then: CREATE TABLE City ( ID INT, Name VARCHAR, CountryCode CHAR(3), District VARCHAR, Population INT, PRIMARY KEY (ID, CountryCode) ) WITH "template=partitioned, backups=1, affinityKey=CountryCode, CACHE_NAME=City, KEY_TYPE=demo.model.CityKey, VALUE_TYPE=demo.model.City"; INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (1,'Kabul','AFG','Kabol',178); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (2,'Qandahar','AFG','Qandahar',237500); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (3,'Herat','AFG','Herat',186800); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (4,'Mazar-e-Sharif','AFG','Balkh',127800); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (5,'Amsterdam','NLD','Noord-Holland',731200); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (6,'Rotterdam','NLD','Zuid-Holland',593321); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (7,'Haag','NLD','Zuid-Holland',440900); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (8,'Utrecht','NLD','Utrecht',234323); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (9,'Eindhoven','NLD','Noord-Brabant',201843); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (10,'Tilburg','NLD','Noord-Brabant',193238); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (11,'Groningen','NLD','Groningen',172701); SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; io.dataregion.default.OffHeapSize 104857600 io.dataregion.default.PhysicalMemoryPages 2066 io.dataregion.default.EmptyDataPages 0 io.dataregion.default.UsedCheckpointBufferSize 0 io.dataregion.default.TotalThrottlingTime 0 io.dataregion.default.PagesReplaced 0 io.dataregion.default.EvictionRate 0 io.dataregion.default.InitialSize 104857600 io.dataregion.default.DirtyPages 0 io.dataregion.default.MaxSize 8589934592 io.dataregion.default.PagesWritten 0 io.dataregion.default.PagesReplaceRate 0 io.dataregion.default.PagesRead 0 io.dataregion.default.PagesFillFactor 0.9997031688690186 io.dataregion.default.TotalAllocatedPages 2066 io.dataregion.default.PhysicalMemorySize 8511920 io.dataregion.default.PagesReplaceAge 0 io.dataregion.default.AllocationRate 2066 io.dataregion.default.OffheapUsedSize 8511920 io.dataregion.default.LargeEntriesPagesCount 0 io.dataregion.default.TotalAllocatedSize 8511920 io.dataregion.default.CheckpointBufferSize 0 TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * TotalAllocatedSize) = 8511920 - (8511920 - 0.9997031688690186 * 8511920) = 8509392 delete from city; SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; io.dataregion.default.OffHeapSize 104857600 io.dataregion.default.PhysicalMemoryPages 2075 io.dataregion.default.EmptyDataPages 0 io.dataregion.default.UsedCheckpointBufferSize 0 io.dataregion.default.TotalThrottlingTime 0 io.dataregion.default.Pages
Re: How to calculate the amount of memory used by a cache or dataregion
Hm, I don't know, honestly. We need to look at the code or somebody else can give us a hint. On Mon, Sep 19, 2022 at 2:55 PM 38797715 <38797...@qq.com> wrote: > if native persistence enabled, the result is the same. > > It seems that there is a bug in the calculation of the "PagesFillFactor" > metrics? > 在 2022/9/19 17:30, Alexander Polovtcev 写道: > > Do you have native persistence enabled? > > On Mon, Sep 19, 2022 at 11:57 AM 38797715 <38797...@qq.com> wrote: > >> 2.10 and 2.13, the same result. >> 在 2022/9/19 16:54, Alexander Polovtcev 写道: >> >> Which version of Ignite do you use? >> >> On Mon, Sep 19, 2022 at 11:38 AM 38797715 <38797...@qq.com> wrote: >> >>> for example: >>> >>> start a node by ignite.sh. >>> >>> then: >>> >>> CREATE TABLE City ( >>> ID INT, >>> Name VARCHAR, >>> CountryCode CHAR(3), >>> District VARCHAR, >>> Population INT, >>> PRIMARY KEY (ID, CountryCode) >>> ) WITH "template=partitioned, backups=1, affinityKey=CountryCode, >>> CACHE_NAME=City, KEY_TYPE=demo.model.CityKey, VALUE_TYPE=demo.model.City"; >>> >>> >>> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >>> (1,'Kabul','AFG','Kabol',178); >>> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >>> (2,'Qandahar','AFG','Qandahar',237500); >>> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >>> (3,'Herat','AFG','Herat',186800); >>> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >>> (4,'Mazar-e-Sharif','AFG','Balkh',127800); >>> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >>> (5,'Amsterdam','NLD','Noord-Holland',731200); >>> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >>> (6,'Rotterdam','NLD','Zuid-Holland',593321); >>> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >>> (7,'Haag','NLD','Zuid-Holland',440900); >>> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >>> (8,'Utrecht','NLD','Utrecht',234323); >>> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >>> (9,'Eindhoven','NLD','Noord-Brabant',201843); >>> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >>> (10,'Tilburg','NLD','Noord-Brabant',193238); >>> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >>> (11,'Groningen','NLD','Groningen',172701); >>> >>> SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; >>> >>> io.dataregion.default.OffHeapSize104857600 >>> io.dataregion.default.PhysicalMemoryPages2066 >>> io.dataregion.default.EmptyDataPages0 >>> io.dataregion.default.UsedCheckpointBufferSize0 >>> io.dataregion.default.TotalThrottlingTime0 >>> io.dataregion.default.PagesReplaced0 >>> io.dataregion.default.EvictionRate0 >>> io.dataregion.default.InitialSize104857600 >>> io.dataregion.default.DirtyPages0 >>> io.dataregion.default.MaxSize8589934592 >>> io.dataregion.default.PagesWritten0 >>> io.dataregion.default.PagesReplaceRate0 >>> io.dataregion.default.PagesRead0 >>> io.dataregion.default.PagesFillFactor0.9997031688690186 >>> io.dataregion.default.TotalAllocatedPages2066 >>> io.dataregion.default.PhysicalMemorySize8511920 >>> io.dataregion.default.PagesReplaceAge0 >>> io.dataregion.default.AllocationRate2066 >>> io.dataregion.default.OffheapUsedSize8511920 >>> io.dataregion.default.LargeEntriesPagesCount0 >>> io.dataregion.default.TotalAllocatedSize8511920 >>> io.dataregion.default.CheckpointBufferSize0 >>> >>> TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * >>> TotalAllocatedSize) >>> >>> = 8511920 - (8511920 - 0.9997031688690186 * 8511920) = 8509392 >>> >>> >>> delete from city; >>> >>> SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; >>> >>> io.dataregion.default.OffHeapSize104857600 >>> io.dataregion.default.PhysicalMemoryPages2075 >>> io.dataregion.default.EmptyDataPages0 >>> io.dataregion.default.UsedCheckpointBufferSize0 >>> io.dataregion.default.TotalThrottlingTime0 >>> io.dataregion.default.PagesReplaced0 >>> io.dataregion.default.EvictionRate0 >>> io.dataregion.default.InitialSize104857600 >>> io.dataregion.default.DirtyPages0 >>> io.dataregion.default.MaxSize8589934592 >>> io.dataregion.default.PagesWritten0 >>> io.dataregion.default.PagesReplaceRate0 >>> io.dataregion.default.PagesRead0 >>> io.dataregion.default.PagesFillFactor1.0 >>> io.dataregion.default.TotalAllocatedPages2075 >>> io.dataregion.default.PhysicalMemorySize8549000 >>> io.dataregion.default.PagesReplaceAge0 >>> io.dataregion.default.AllocationRate9 >>> io.dataregion.default.OffheapUsedSize8549000 >>> io.dataregion.default.LargeEntriesPagesCount0 >>> io.dataregion.default.TotalAllocatedSize8549000 >>> io.dataregion.default.CheckpointBufferSize0 >>> >>> >>> TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * >>> TotalAllocatedSize) >
Re: Ignite cache failed to read by key intermittently
Hi,
I have attached here.
Regards,
Charlin
On Mon, 19 Sept 2022 at 19:00, Pavel Tupitsyn wrote:
> I get an exception when trying to run the code:
>
> Unhandled exception. System.NullReferenceException: Object reference not
> set to an instance of an object.
>at ConsoleApp4.IgniteInstance..ctor() in
> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/IgniteInstance.cs:line
> 47
>at ConsoleApp4.IgniteInstance.get_Instance() in
> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/IgniteInstance.cs:line
> 33
>at ConsoleApp4.UnitOfWork..ctor() in
> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/UnitOfWork.cs:line
> 15
>at ConsoleApp4.Program.Main(String[] args) in
> /home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/Program.cs:line 12
>
> On Mon, Sep 19, 2022 at 3:04 PM Charlin S wrote:
>
>> Hi,
>> Cache EexpiryPolicy is 60 minutes. but cache key failing in between and
>> return data after some time so its not due to expiry policy.
>> I have prepared a separate solution and attached it here. I am sharing a
>> cache data reading time screenshot for your reference .
>> [image: image.png]
>> Regards,
>> Charlin
>>
>>
>>
>>
>>
>> On Mon, 19 Sept 2022 at 11:00, Pavel Tupitsyn
>> wrote:
>>
>>> The code you provided above uses ExpiryPolicy. It might be the reason
>>> for TryGet to return false - cache entry expires.
>>> To provide a definitive answer we need a project that runs and
>>> demonstrates the problem.
>>>
>>> On Fri, Sep 16, 2022 at 5:12 PM Charlin S
>>> wrote:
>>>
Hi,
sorry for inconvenience. It's 90% actual code only. I have changed
cache name and grid name as per policy. I have to prepare a separate
solution and provide you if required.
Regards,
Charlin
On Fri, 16 Sept 2022 at 19:11, Pavel Tupitsyn
wrote:
> It does not have to be your actual code, just something that
> demonstrates the problem
>
> On Fri, Sep 16, 2022 at 4:40 PM Pavel Tupitsyn
> wrote:
>
>> Can you please share a complete project that I can run and see
>> the issue?
>>
>> On Fri, Sep 16, 2022 at 4:30 PM Charlin S
>> wrote:
>>
>>> Hi,
>>> Sorry I am not able to share a complete solution. bean xml attached
>>> and code snippet in below
>>> IgniteConfiguration igniteGridIg = new IgniteConfiguration();
>>> igniteGridIg.AutoGenerateIgniteInstanceName = true;
>>> if
>>> (!string.IsNullOrEmpty(GlobalVariables.IgniteSettings.IGNITE_HOME))
>>> {
>>> igniteGridIg.IgniteHome =
>>> GlobalVariables.IgniteSettings.IGNITE_HOME;
>>> }
>>> var gridDetails =
>>> GlobalVariables.IgniteSettings.GridDetails.Where(gd =>
>>> gd.GridType.Equals("DynamicGrid",
>>> StringComparison.OrdinalIgnoreCase)).FirstOrDefault();
>>> //setting ignite working directory if not ignite will
>>> run with default work directory
>>> if
>>> (!string.IsNullOrEmpty(GlobalVariables.IgniteSettings.WorkDirectory))
>>> {
>>> igniteGridIg.WorkDirectory =
>>> Path.Combine(GlobalVariables.IgniteSettings.WorkDirectory,
>>> gridDetails.CacheType);
>>> }
>>> if (null != gridDetails)
>>> {
>>> GridName = gridDetails.GridName;
>>> igniteGridIg.SpringConfigUrl =
>>> Path.Combine(GlobalVariables.IgniteSettings.ConfigurationPath,
>>> gridDetails.SpringXMLFileName);
>>> igniteGridIg.ConsistentId =
>>> Guid.NewGuid().ToString().ToUpper();
>>> ICollection lstJvmOptions = null;
>>>
>>> //"IgniteJVMOption":
>>> "-XX:+DisableExplicitGC,-XX:+UseG1GC,-Xms1g,-Xmx2g"
>>> if
>>> (string.IsNullOrEmpty(gridDetails.IgniteJVMOption))
>>> {
>>> lstJvmOptions =
>>> GlobalVariables.IgniteSettings.IgniteJVMOptionDefault.Split(',').Where(x
>>> =>
>>> !string.IsNullOrWhiteSpace(x)).ToList();
>>> }
>>> else
>>> lstJvmOptions =
>>> gridDetails.IgniteJVMOption.Split(',').Where(x =>
>>> !string.IsNullOrWhiteSpace(x)).ToList();
>>>
>>> if (lstJvmOptions.Any())
>>> igniteGridIg.JvmOptions = lstJvmOptions;
>>> int RetryCount = 0;
>>> while (InstanceObject == null)
>>> {
>>> try
>>> {
>>> RetryCount++;
>>> //Try to start Ignite
>>> if
>>> (!String.IsNullOrEmpty(ApplicationSettingKeys.Get("IgniteInstanceRetryLimit"))
>>> && RetryCount >
>>> (ApplicationSettingKeys.Get("IgniteInstanceRetryLimit")))
>>>
Re: Ignite cache failed to read by key intermittently
I get an exception when trying to run the code:
Unhandled exception. System.NullReferenceException: Object reference not
set to an instance of an object.
at ConsoleApp4.IgniteInstance..ctor() in
/home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/IgniteInstance.cs:line
47
at ConsoleApp4.IgniteInstance.get_Instance() in
/home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/IgniteInstance.cs:line
33
at ConsoleApp4.UnitOfWork..ctor() in
/home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/UnitOfWork.cs:line
15
at ConsoleApp4.Program.Main(String[] args) in
/home/pavel/Downloads/ConsoleApp4/ConsoleApp4/ConsoleApp4/Program.cs:line 12
On Mon, Sep 19, 2022 at 3:04 PM Charlin S wrote:
> Hi,
> Cache EexpiryPolicy is 60 minutes. but cache key failing in between and
> return data after some time so its not due to expiry policy.
> I have prepared a separate solution and attached it here. I am sharing a
> cache data reading time screenshot for your reference .
> [image: image.png]
> Regards,
> Charlin
>
>
>
>
>
> On Mon, 19 Sept 2022 at 11:00, Pavel Tupitsyn
> wrote:
>
>> The code you provided above uses ExpiryPolicy. It might be the reason for
>> TryGet to return false - cache entry expires.
>> To provide a definitive answer we need a project that runs and
>> demonstrates the problem.
>>
>> On Fri, Sep 16, 2022 at 5:12 PM Charlin S wrote:
>>
>>> Hi,
>>> sorry for inconvenience. It's 90% actual code only. I have changed cache
>>> name and grid name as per policy. I have to prepare a separate solution and
>>> provide you if required.
>>>
>>> Regards,
>>> Charlin
>>>
>>>
>>>
>>> On Fri, 16 Sept 2022 at 19:11, Pavel Tupitsyn
>>> wrote:
>>>
It does not have to be your actual code, just something that
demonstrates the problem
On Fri, Sep 16, 2022 at 4:40 PM Pavel Tupitsyn
wrote:
> Can you please share a complete project that I can run and see
> the issue?
>
> On Fri, Sep 16, 2022 at 4:30 PM Charlin S
> wrote:
>
>> Hi,
>> Sorry I am not able to share a complete solution. bean xml attached
>> and code snippet in below
>> IgniteConfiguration igniteGridIg = new IgniteConfiguration();
>> igniteGridIg.AutoGenerateIgniteInstanceName = true;
>> if
>> (!string.IsNullOrEmpty(GlobalVariables.IgniteSettings.IGNITE_HOME))
>> {
>> igniteGridIg.IgniteHome =
>> GlobalVariables.IgniteSettings.IGNITE_HOME;
>> }
>> var gridDetails =
>> GlobalVariables.IgniteSettings.GridDetails.Where(gd =>
>> gd.GridType.Equals("DynamicGrid",
>> StringComparison.OrdinalIgnoreCase)).FirstOrDefault();
>> //setting ignite working directory if not ignite will run
>> with default work directory
>> if
>> (!string.IsNullOrEmpty(GlobalVariables.IgniteSettings.WorkDirectory))
>> {
>> igniteGridIg.WorkDirectory =
>> Path.Combine(GlobalVariables.IgniteSettings.WorkDirectory,
>> gridDetails.CacheType);
>> }
>> if (null != gridDetails)
>> {
>> GridName = gridDetails.GridName;
>> igniteGridIg.SpringConfigUrl =
>> Path.Combine(GlobalVariables.IgniteSettings.ConfigurationPath,
>> gridDetails.SpringXMLFileName);
>> igniteGridIg.ConsistentId =
>> Guid.NewGuid().ToString().ToUpper();
>> ICollection lstJvmOptions = null;
>>
>> //"IgniteJVMOption":
>> "-XX:+DisableExplicitGC,-XX:+UseG1GC,-Xms1g,-Xmx2g"
>> if (string.IsNullOrEmpty(gridDetails.IgniteJVMOption))
>> {
>> lstJvmOptions =
>> GlobalVariables.IgniteSettings.IgniteJVMOptionDefault.Split(',').Where(x
>> =>
>> !string.IsNullOrWhiteSpace(x)).ToList();
>> }
>> else
>> lstJvmOptions =
>> gridDetails.IgniteJVMOption.Split(',').Where(x =>
>> !string.IsNullOrWhiteSpace(x)).ToList();
>>
>> if (lstJvmOptions.Any())
>> igniteGridIg.JvmOptions = lstJvmOptions;
>> int RetryCount = 0;
>> while (InstanceObject == null)
>> {
>> try
>> {
>> RetryCount++;
>> //Try to start Ignite
>> if
>> (!String.IsNullOrEmpty(ApplicationSettingKeys.Get("IgniteInstanceRetryLimit"))
>> && RetryCount >
>> (ApplicationSettingKeys.Get("IgniteInstanceRetryLimit")))
>> {
>> break;
>> }
>> //Try to start Ignite
>> if
>> (String.IsNullOrEmpty(ApplicationSettingKeys.Get("IgniteDynamicCache
Re: Ignite cache failed to read by key intermittently
Hi,
Cache EexpiryPolicy is 60 minutes. but cache key failing in between and
return data after some time so its not due to expiry policy.
I have prepared a separate solution and attached it here. I am sharing a
cache data reading time screenshot for your reference .
[image: image.png]
Regards,
Charlin
On Mon, 19 Sept 2022 at 11:00, Pavel Tupitsyn wrote:
> The code you provided above uses ExpiryPolicy. It might be the reason for
> TryGet to return false - cache entry expires.
> To provide a definitive answer we need a project that runs and
> demonstrates the problem.
>
> On Fri, Sep 16, 2022 at 5:12 PM Charlin S wrote:
>
>> Hi,
>> sorry for inconvenience. It's 90% actual code only. I have changed cache
>> name and grid name as per policy. I have to prepare a separate solution and
>> provide you if required.
>>
>> Regards,
>> Charlin
>>
>>
>>
>> On Fri, 16 Sept 2022 at 19:11, Pavel Tupitsyn
>> wrote:
>>
>>> It does not have to be your actual code, just something that
>>> demonstrates the problem
>>>
>>> On Fri, Sep 16, 2022 at 4:40 PM Pavel Tupitsyn
>>> wrote:
>>>
Can you please share a complete project that I can run and see
the issue?
On Fri, Sep 16, 2022 at 4:30 PM Charlin S
wrote:
> Hi,
> Sorry I am not able to share a complete solution. bean xml attached
> and code snippet in below
> IgniteConfiguration igniteGridIg = new IgniteConfiguration();
> igniteGridIg.AutoGenerateIgniteInstanceName = true;
> if
> (!string.IsNullOrEmpty(GlobalVariables.IgniteSettings.IGNITE_HOME))
> {
> igniteGridIg.IgniteHome =
> GlobalVariables.IgniteSettings.IGNITE_HOME;
> }
> var gridDetails =
> GlobalVariables.IgniteSettings.GridDetails.Where(gd =>
> gd.GridType.Equals("DynamicGrid",
> StringComparison.OrdinalIgnoreCase)).FirstOrDefault();
> //setting ignite working directory if not ignite will run
> with default work directory
> if
> (!string.IsNullOrEmpty(GlobalVariables.IgniteSettings.WorkDirectory))
> {
> igniteGridIg.WorkDirectory =
> Path.Combine(GlobalVariables.IgniteSettings.WorkDirectory,
> gridDetails.CacheType);
> }
> if (null != gridDetails)
> {
> GridName = gridDetails.GridName;
> igniteGridIg.SpringConfigUrl =
> Path.Combine(GlobalVariables.IgniteSettings.ConfigurationPath,
> gridDetails.SpringXMLFileName);
> igniteGridIg.ConsistentId =
> Guid.NewGuid().ToString().ToUpper();
> ICollection lstJvmOptions = null;
>
> //"IgniteJVMOption":
> "-XX:+DisableExplicitGC,-XX:+UseG1GC,-Xms1g,-Xmx2g"
> if (string.IsNullOrEmpty(gridDetails.IgniteJVMOption))
> {
> lstJvmOptions =
> GlobalVariables.IgniteSettings.IgniteJVMOptionDefault.Split(',').Where(x
> =>
> !string.IsNullOrWhiteSpace(x)).ToList();
> }
> else
> lstJvmOptions =
> gridDetails.IgniteJVMOption.Split(',').Where(x =>
> !string.IsNullOrWhiteSpace(x)).ToList();
>
> if (lstJvmOptions.Any())
> igniteGridIg.JvmOptions = lstJvmOptions;
> int RetryCount = 0;
> while (InstanceObject == null)
> {
> try
> {
> RetryCount++;
> //Try to start Ignite
> if
> (!String.IsNullOrEmpty(ApplicationSettingKeys.Get("IgniteInstanceRetryLimit"))
> && RetryCount >
> (ApplicationSettingKeys.Get("IgniteInstanceRetryLimit")))
> {
> break;
> }
> //Try to start Ignite
> if
> (String.IsNullOrEmpty(ApplicationSettingKeys.Get("IgniteDynamicCacheGrid")))
> InstanceObject =
> Ignition.Start(igniteGridIg);
> else
> InstanceObject =
> Ignition.TryGetIgnite(ApplicationSettingKeys.Get("IgniteDynamicCacheGrid"))
> ?? Ignition.Start(igniteGridIg);
> }
> catch (Exception ex) //If failing to start Ignite,
> wait a bit for the previous AppDomain to stop the Ignite running
> instance...
> {
> logContextInfo.ErrorLogger(ex);
>
> }
> }
>
> public ICache testModelICache { get; set; }
>
> using (new Log4NetTimeLogHelper(logContextInfo, "testModelModel",
> "testModelModel"))
> {
>
>
Re: How to calculate the amount of memory used by a cache or dataregion
if native persistence enabled, the result is the same. It seems that there is a bug in the calculation of the "PagesFillFactor" metrics? 在 2022/9/19 17:30, Alexander Polovtcev 写道: Do you have native persistence enabled? On Mon, Sep 19, 2022 at 11:57 AM 38797715 <38797...@qq.com> wrote: 2.10 and 2.13, the same result. 在 2022/9/19 16:54, Alexander Polovtcev 写道: Which version of Ignite do you use? On Mon, Sep 19, 2022 at 11:38 AM 38797715 <38797...@qq.com> wrote: for example: start a node by ignite.sh. then: CREATE TABLE City ( ID INT, Name VARCHAR, CountryCode CHAR(3), District VARCHAR, Population INT, PRIMARY KEY (ID, CountryCode) ) WITH "template=partitioned, backups=1, affinityKey=CountryCode, CACHE_NAME=City, KEY_TYPE=demo.model.CityKey, VALUE_TYPE=demo.model.City"; INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (1,'Kabul','AFG','Kabol',178); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (2,'Qandahar','AFG','Qandahar',237500); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (3,'Herat','AFG','Herat',186800); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (4,'Mazar-e-Sharif','AFG','Balkh',127800); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (5,'Amsterdam','NLD','Noord-Holland',731200); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (6,'Rotterdam','NLD','Zuid-Holland',593321); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (7,'Haag','NLD','Zuid-Holland',440900); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (8,'Utrecht','NLD','Utrecht',234323); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (9,'Eindhoven','NLD','Noord-Brabant',201843); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (10,'Tilburg','NLD','Noord-Brabant',193238); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (11,'Groningen','NLD','Groningen',172701); SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; io.dataregion.default.OffHeapSize 104857600 io.dataregion.default.PhysicalMemoryPages 2066 io.dataregion.default.EmptyDataPages 0 io.dataregion.default.UsedCheckpointBufferSize 0 io.dataregion.default.TotalThrottlingTime 0 io.dataregion.default.PagesReplaced 0 io.dataregion.default.EvictionRate 0 io.dataregion.default.InitialSize 104857600 io.dataregion.default.DirtyPages 0 io.dataregion.default.MaxSize 8589934592 io.dataregion.default.PagesWritten 0 io.dataregion.default.PagesReplaceRate 0 io.dataregion.default.PagesRead 0 io.dataregion.default.PagesFillFactor 0.9997031688690186 io.dataregion.default.TotalAllocatedPages 2066 io.dataregion.default.PhysicalMemorySize 8511920 io.dataregion.default.PagesReplaceAge 0 io.dataregion.default.AllocationRate 2066 io.dataregion.default.OffheapUsedSize 8511920 io.dataregion.default.LargeEntriesPagesCount 0 io.dataregion.default.TotalAllocatedSize 8511920 io.dataregion.default.CheckpointBufferSize 0 TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * TotalAllocatedSize) = 8511920 - (8511920 - 0.9997031688690186 * 8511920) = 8509392 delete from city; SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; io.dataregion.default.OffHeapSize 104857600 io.dataregion.default.PhysicalMemoryPages 2075 io.dataregion.default.EmptyDataPages 0 io.dataregion.default.UsedCheckpointBufferSize 0 io.dataregion.default.TotalThrottlingTime 0 io.dataregion.default.PagesReplaced 0 io.dataregion.default.EvictionRate 0 io.dataregion.default.InitialSize 104857600 io.dataregion.default.DirtyPages 0 io.dataregion.default.MaxSize 8589934592 io.dataregion.default.PagesWritten 0 io.dataregion.default.PagesReplaceRate 0 io.dataregion.default.PagesRead 0 io.dataregion.default.PagesFillFactor 1.0 io.dataregion.default.TotalAllocatedPages 2075 io.dataregion.default.PhysicalMemorySize 8549000 io.dataregion.default.PagesReplaceAge 0 io.dataregion.default.AllocationRate 9 io.dataregion.default.OffheapUsedSize 8549000 io.dataregion.default.LargeEntriesPagesCount 0 io.dataregion.default.TotalAllocatedSize 8549000 io.dataregion.default.Check
Re: How to calculate the amount of memory used by a cache or dataregion
no, pure memory mode. 在 2022/9/19 17:30, Alexander Polovtcev 写道: Do you have native persistence enabled? On Mon, Sep 19, 2022 at 11:57 AM 38797715 <38797...@qq.com> wrote: 2.10 and 2.13, the same result. 在 2022/9/19 16:54, Alexander Polovtcev 写道: Which version of Ignite do you use? On Mon, Sep 19, 2022 at 11:38 AM 38797715 <38797...@qq.com> wrote: for example: start a node by ignite.sh. then: CREATE TABLE City ( ID INT, Name VARCHAR, CountryCode CHAR(3), District VARCHAR, Population INT, PRIMARY KEY (ID, CountryCode) ) WITH "template=partitioned, backups=1, affinityKey=CountryCode, CACHE_NAME=City, KEY_TYPE=demo.model.CityKey, VALUE_TYPE=demo.model.City"; INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (1,'Kabul','AFG','Kabol',178); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (2,'Qandahar','AFG','Qandahar',237500); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (3,'Herat','AFG','Herat',186800); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (4,'Mazar-e-Sharif','AFG','Balkh',127800); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (5,'Amsterdam','NLD','Noord-Holland',731200); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (6,'Rotterdam','NLD','Zuid-Holland',593321); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (7,'Haag','NLD','Zuid-Holland',440900); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (8,'Utrecht','NLD','Utrecht',234323); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (9,'Eindhoven','NLD','Noord-Brabant',201843); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (10,'Tilburg','NLD','Noord-Brabant',193238); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (11,'Groningen','NLD','Groningen',172701); SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; io.dataregion.default.OffHeapSize 104857600 io.dataregion.default.PhysicalMemoryPages 2066 io.dataregion.default.EmptyDataPages 0 io.dataregion.default.UsedCheckpointBufferSize 0 io.dataregion.default.TotalThrottlingTime 0 io.dataregion.default.PagesReplaced 0 io.dataregion.default.EvictionRate 0 io.dataregion.default.InitialSize 104857600 io.dataregion.default.DirtyPages 0 io.dataregion.default.MaxSize 8589934592 io.dataregion.default.PagesWritten 0 io.dataregion.default.PagesReplaceRate 0 io.dataregion.default.PagesRead 0 io.dataregion.default.PagesFillFactor 0.9997031688690186 io.dataregion.default.TotalAllocatedPages 2066 io.dataregion.default.PhysicalMemorySize 8511920 io.dataregion.default.PagesReplaceAge 0 io.dataregion.default.AllocationRate 2066 io.dataregion.default.OffheapUsedSize 8511920 io.dataregion.default.LargeEntriesPagesCount 0 io.dataregion.default.TotalAllocatedSize 8511920 io.dataregion.default.CheckpointBufferSize 0 TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * TotalAllocatedSize) = 8511920 - (8511920 - 0.9997031688690186 * 8511920) = 8509392 delete from city; SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; io.dataregion.default.OffHeapSize 104857600 io.dataregion.default.PhysicalMemoryPages 2075 io.dataregion.default.EmptyDataPages 0 io.dataregion.default.UsedCheckpointBufferSize 0 io.dataregion.default.TotalThrottlingTime 0 io.dataregion.default.PagesReplaced 0 io.dataregion.default.EvictionRate 0 io.dataregion.default.InitialSize 104857600 io.dataregion.default.DirtyPages 0 io.dataregion.default.MaxSize 8589934592 io.dataregion.default.PagesWritten 0 io.dataregion.default.PagesReplaceRate 0 io.dataregion.default.PagesRead 0 io.dataregion.default.PagesFillFactor 1.0 io.dataregion.default.TotalAllocatedPages 2075 io.dataregion.default.PhysicalMemorySize 8549000 io.dataregion.default.PagesReplaceAge 0 io.dataregion.default.AllocationRate 9 io.dataregion.default.OffheapUsedSize 8549000 io.dataregion.default.LargeEntriesPagesCount 0 io.dataregion.default.TotalAllocatedSize 8549000 io.dataregion.default.CheckpointBufferSize 0 TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * TotalAllocatedSize)
Re: How to calculate the amount of memory used by a cache or dataregion
Do you have native persistence enabled? On Mon, Sep 19, 2022 at 11:57 AM 38797715 <38797...@qq.com> wrote: > 2.10 and 2.13, the same result. > 在 2022/9/19 16:54, Alexander Polovtcev 写道: > > Which version of Ignite do you use? > > On Mon, Sep 19, 2022 at 11:38 AM 38797715 <38797...@qq.com> wrote: > >> for example: >> >> start a node by ignite.sh. >> >> then: >> >> CREATE TABLE City ( >> ID INT, >> Name VARCHAR, >> CountryCode CHAR(3), >> District VARCHAR, >> Population INT, >> PRIMARY KEY (ID, CountryCode) >> ) WITH "template=partitioned, backups=1, affinityKey=CountryCode, >> CACHE_NAME=City, KEY_TYPE=demo.model.CityKey, VALUE_TYPE=demo.model.City"; >> >> >> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >> (1,'Kabul','AFG','Kabol',178); >> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >> (2,'Qandahar','AFG','Qandahar',237500); >> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >> (3,'Herat','AFG','Herat',186800); >> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >> (4,'Mazar-e-Sharif','AFG','Balkh',127800); >> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >> (5,'Amsterdam','NLD','Noord-Holland',731200); >> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >> (6,'Rotterdam','NLD','Zuid-Holland',593321); >> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >> (7,'Haag','NLD','Zuid-Holland',440900); >> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >> (8,'Utrecht','NLD','Utrecht',234323); >> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >> (9,'Eindhoven','NLD','Noord-Brabant',201843); >> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >> (10,'Tilburg','NLD','Noord-Brabant',193238); >> INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES >> (11,'Groningen','NLD','Groningen',172701); >> >> SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; >> >> io.dataregion.default.OffHeapSize104857600 >> io.dataregion.default.PhysicalMemoryPages2066 >> io.dataregion.default.EmptyDataPages0 >> io.dataregion.default.UsedCheckpointBufferSize0 >> io.dataregion.default.TotalThrottlingTime0 >> io.dataregion.default.PagesReplaced0 >> io.dataregion.default.EvictionRate0 >> io.dataregion.default.InitialSize104857600 >> io.dataregion.default.DirtyPages0 >> io.dataregion.default.MaxSize8589934592 >> io.dataregion.default.PagesWritten0 >> io.dataregion.default.PagesReplaceRate0 >> io.dataregion.default.PagesRead0 >> io.dataregion.default.PagesFillFactor0.9997031688690186 >> io.dataregion.default.TotalAllocatedPages2066 >> io.dataregion.default.PhysicalMemorySize8511920 >> io.dataregion.default.PagesReplaceAge0 >> io.dataregion.default.AllocationRate2066 >> io.dataregion.default.OffheapUsedSize8511920 >> io.dataregion.default.LargeEntriesPagesCount0 >> io.dataregion.default.TotalAllocatedSize8511920 >> io.dataregion.default.CheckpointBufferSize0 >> >> TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * >> TotalAllocatedSize) >> >> = 8511920 - (8511920 - 0.9997031688690186 * 8511920) = 8509392 >> >> >> delete from city; >> >> SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; >> >> io.dataregion.default.OffHeapSize104857600 >> io.dataregion.default.PhysicalMemoryPages2075 >> io.dataregion.default.EmptyDataPages0 >> io.dataregion.default.UsedCheckpointBufferSize0 >> io.dataregion.default.TotalThrottlingTime0 >> io.dataregion.default.PagesReplaced0 >> io.dataregion.default.EvictionRate0 >> io.dataregion.default.InitialSize104857600 >> io.dataregion.default.DirtyPages0 >> io.dataregion.default.MaxSize8589934592 >> io.dataregion.default.PagesWritten0 >> io.dataregion.default.PagesReplaceRate0 >> io.dataregion.default.PagesRead0 >> io.dataregion.default.PagesFillFactor1.0 >> io.dataregion.default.TotalAllocatedPages2075 >> io.dataregion.default.PhysicalMemorySize8549000 >> io.dataregion.default.PagesReplaceAge0 >> io.dataregion.default.AllocationRate9 >> io.dataregion.default.OffheapUsedSize8549000 >> io.dataregion.default.LargeEntriesPagesCount0 >> io.dataregion.default.TotalAllocatedSize8549000 >> io.dataregion.default.CheckpointBufferSize0 >> >> >> TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * >> TotalAllocatedSize) >> >> =8549000 - (8549000 - 1.0 * 8549000) = 8549000 >> >> Instead, the value becomes larger? >> >> >> 在 2022/9/19 15:13, Alexander Polovtcev 写道: >> >> Sorry, I messed up the metric names a little bit, we should use the size >> metrics, not the page metrics. So the correct formula would be: >> `TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * >> TotalAllocatedSize)` >> >> On Sun, Sep 18, 2022 at 4:29 PM 38797715 <38797...@qq.com> wrote: >> >>> I've checked
Re: How to calculate the amount of memory used by a cache or dataregion
2.10 and 2.13, the same result. 在 2022/9/19 16:54, Alexander Polovtcev 写道: Which version of Ignite do you use? On Mon, Sep 19, 2022 at 11:38 AM 38797715 <38797...@qq.com> wrote: for example: start a node by ignite.sh. then: CREATE TABLE City ( ID INT, Name VARCHAR, CountryCode CHAR(3), District VARCHAR, Population INT, PRIMARY KEY (ID, CountryCode) ) WITH "template=partitioned, backups=1, affinityKey=CountryCode, CACHE_NAME=City, KEY_TYPE=demo.model.CityKey, VALUE_TYPE=demo.model.City"; INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (1,'Kabul','AFG','Kabol',178); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (2,'Qandahar','AFG','Qandahar',237500); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (3,'Herat','AFG','Herat',186800); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (4,'Mazar-e-Sharif','AFG','Balkh',127800); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (5,'Amsterdam','NLD','Noord-Holland',731200); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (6,'Rotterdam','NLD','Zuid-Holland',593321); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (7,'Haag','NLD','Zuid-Holland',440900); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (8,'Utrecht','NLD','Utrecht',234323); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (9,'Eindhoven','NLD','Noord-Brabant',201843); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (10,'Tilburg','NLD','Noord-Brabant',193238); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (11,'Groningen','NLD','Groningen',172701); SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; io.dataregion.default.OffHeapSize 104857600 io.dataregion.default.PhysicalMemoryPages 2066 io.dataregion.default.EmptyDataPages 0 io.dataregion.default.UsedCheckpointBufferSize 0 io.dataregion.default.TotalThrottlingTime 0 io.dataregion.default.PagesReplaced 0 io.dataregion.default.EvictionRate 0 io.dataregion.default.InitialSize 104857600 io.dataregion.default.DirtyPages 0 io.dataregion.default.MaxSize 8589934592 io.dataregion.default.PagesWritten 0 io.dataregion.default.PagesReplaceRate 0 io.dataregion.default.PagesRead 0 io.dataregion.default.PagesFillFactor 0.9997031688690186 io.dataregion.default.TotalAllocatedPages 2066 io.dataregion.default.PhysicalMemorySize 8511920 io.dataregion.default.PagesReplaceAge 0 io.dataregion.default.AllocationRate 2066 io.dataregion.default.OffheapUsedSize 8511920 io.dataregion.default.LargeEntriesPagesCount 0 io.dataregion.default.TotalAllocatedSize 8511920 io.dataregion.default.CheckpointBufferSize 0 TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * TotalAllocatedSize) = 8511920 - (8511920 - 0.9997031688690186 * 8511920) = 8509392 delete from city; SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; io.dataregion.default.OffHeapSize 104857600 io.dataregion.default.PhysicalMemoryPages 2075 io.dataregion.default.EmptyDataPages 0 io.dataregion.default.UsedCheckpointBufferSize 0 io.dataregion.default.TotalThrottlingTime 0 io.dataregion.default.PagesReplaced 0 io.dataregion.default.EvictionRate 0 io.dataregion.default.InitialSize 104857600 io.dataregion.default.DirtyPages 0 io.dataregion.default.MaxSize 8589934592 io.dataregion.default.PagesWritten 0 io.dataregion.default.PagesReplaceRate 0 io.dataregion.default.PagesRead 0 io.dataregion.default.PagesFillFactor 1.0 io.dataregion.default.TotalAllocatedPages 2075 io.dataregion.default.PhysicalMemorySize 8549000 io.dataregion.default.PagesReplaceAge 0 io.dataregion.default.AllocationRate 9 io.dataregion.default.OffheapUsedSize 8549000 io.dataregion.default.LargeEntriesPagesCount 0 io.dataregion.default.TotalAllocatedSize 8549000 io.dataregion.default.CheckpointBufferSize 0 TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * TotalAllocatedSize) =8549000 - (8549000 - 1.0 * 8549000) = 8549000 Instead, the value becomes larger? 在 2022/9/19 15:13, Alexander Polovtcev 写道: Sorry, I messed up the metric names a little bit, we should use the size metrics, not the page metrics. So the correct formula would be: `TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * TotalAllocatedSize)` On Sun, Sep 18, 2022 at 4:29 PM 38797715 <38797...@qq.com> wrote: I've checked TotalUsedPages in ignite.dataRegionMetrics(), it is fou
Re: How to calculate the amount of memory used by a cache or dataregion
Which version of Ignite do you use? On Mon, Sep 19, 2022 at 11:38 AM 38797715 <38797...@qq.com> wrote: > for example: > > start a node by ignite.sh. > > then: > > CREATE TABLE City ( > ID INT, > Name VARCHAR, > CountryCode CHAR(3), > District VARCHAR, > Population INT, > PRIMARY KEY (ID, CountryCode) > ) WITH "template=partitioned, backups=1, affinityKey=CountryCode, > CACHE_NAME=City, KEY_TYPE=demo.model.CityKey, VALUE_TYPE=demo.model.City"; > > > INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES > (1,'Kabul','AFG','Kabol',178); > INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES > (2,'Qandahar','AFG','Qandahar',237500); > INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES > (3,'Herat','AFG','Herat',186800); > INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES > (4,'Mazar-e-Sharif','AFG','Balkh',127800); > INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES > (5,'Amsterdam','NLD','Noord-Holland',731200); > INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES > (6,'Rotterdam','NLD','Zuid-Holland',593321); > INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES > (7,'Haag','NLD','Zuid-Holland',440900); > INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES > (8,'Utrecht','NLD','Utrecht',234323); > INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES > (9,'Eindhoven','NLD','Noord-Brabant',201843); > INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES > (10,'Tilburg','NLD','Noord-Brabant',193238); > INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES > (11,'Groningen','NLD','Groningen',172701); > > SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; > > io.dataregion.default.OffHeapSize104857600 > io.dataregion.default.PhysicalMemoryPages2066 > io.dataregion.default.EmptyDataPages0 > io.dataregion.default.UsedCheckpointBufferSize0 > io.dataregion.default.TotalThrottlingTime0 > io.dataregion.default.PagesReplaced0 > io.dataregion.default.EvictionRate0 > io.dataregion.default.InitialSize104857600 > io.dataregion.default.DirtyPages0 > io.dataregion.default.MaxSize8589934592 > io.dataregion.default.PagesWritten0 > io.dataregion.default.PagesReplaceRate0 > io.dataregion.default.PagesRead0 > io.dataregion.default.PagesFillFactor0.9997031688690186 > io.dataregion.default.TotalAllocatedPages2066 > io.dataregion.default.PhysicalMemorySize8511920 > io.dataregion.default.PagesReplaceAge0 > io.dataregion.default.AllocationRate2066 > io.dataregion.default.OffheapUsedSize8511920 > io.dataregion.default.LargeEntriesPagesCount0 > io.dataregion.default.TotalAllocatedSize8511920 > io.dataregion.default.CheckpointBufferSize0 > > TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * TotalAllocatedSize) > > = 8511920 - (8511920 - 0.9997031688690186 * 8511920) = 8509392 > > > delete from city; > > SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; > > io.dataregion.default.OffHeapSize104857600 > io.dataregion.default.PhysicalMemoryPages2075 > io.dataregion.default.EmptyDataPages0 > io.dataregion.default.UsedCheckpointBufferSize0 > io.dataregion.default.TotalThrottlingTime0 > io.dataregion.default.PagesReplaced0 > io.dataregion.default.EvictionRate0 > io.dataregion.default.InitialSize104857600 > io.dataregion.default.DirtyPages0 > io.dataregion.default.MaxSize8589934592 > io.dataregion.default.PagesWritten0 > io.dataregion.default.PagesReplaceRate0 > io.dataregion.default.PagesRead0 > io.dataregion.default.PagesFillFactor1.0 > io.dataregion.default.TotalAllocatedPages2075 > io.dataregion.default.PhysicalMemorySize8549000 > io.dataregion.default.PagesReplaceAge0 > io.dataregion.default.AllocationRate9 > io.dataregion.default.OffheapUsedSize8549000 > io.dataregion.default.LargeEntriesPagesCount0 > io.dataregion.default.TotalAllocatedSize8549000 > io.dataregion.default.CheckpointBufferSize0 > > > TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * TotalAllocatedSize) > > =8549000 - (8549000 - 1.0 * 8549000) = 8549000 > > Instead, the value becomes larger? > > > 在 2022/9/19 15:13, Alexander Polovtcev 写道: > > Sorry, I messed up the metric names a little bit, we should use the size > metrics, not the page metrics. So the correct formula would be: > `TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * > TotalAllocatedSize)` > > On Sun, Sep 18, 2022 at 4:29 PM 38797715 <38797...@qq.com> wrote: > >> I've checked TotalUsedPages in ignite.dataRegionMetrics(), it is found >> that the logic of this metric is consistent with OffheapUsedSize, that is, >> the value will not be updated after data deletion. >> 在 2022/9/18 19:42, Alexander Polovtcev 写道: >> >> Hello, you can check out "TotalUsedPages" and "TotalUsedSize" metrics >> th
Re: How to calculate the amount of memory used by a cache or dataregion
for example: start a node by ignite.sh. then: CREATE TABLE City ( ID INT, Name VARCHAR, CountryCode CHAR(3), District VARCHAR, Population INT, PRIMARY KEY (ID, CountryCode) ) WITH "template=partitioned, backups=1, affinityKey=CountryCode, CACHE_NAME=City, KEY_TYPE=demo.model.CityKey, VALUE_TYPE=demo.model.City"; INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (1,'Kabul','AFG','Kabol',178); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (2,'Qandahar','AFG','Qandahar',237500); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (3,'Herat','AFG','Herat',186800); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (4,'Mazar-e-Sharif','AFG','Balkh',127800); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (5,'Amsterdam','NLD','Noord-Holland',731200); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (6,'Rotterdam','NLD','Zuid-Holland',593321); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (7,'Haag','NLD','Zuid-Holland',440900); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (8,'Utrecht','NLD','Utrecht',234323); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (9,'Eindhoven','NLD','Noord-Brabant',201843); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (10,'Tilburg','NLD','Noord-Brabant',193238); INSERT INTO City(ID, Name, CountryCode, District, Population) VALUES (11,'Groningen','NLD','Groningen',172701); SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; io.dataregion.default.OffHeapSize 104857600 io.dataregion.default.PhysicalMemoryPages 2066 io.dataregion.default.EmptyDataPages 0 io.dataregion.default.UsedCheckpointBufferSize 0 io.dataregion.default.TotalThrottlingTime 0 io.dataregion.default.PagesReplaced 0 io.dataregion.default.EvictionRate 0 io.dataregion.default.InitialSize 104857600 io.dataregion.default.DirtyPages 0 io.dataregion.default.MaxSize 8589934592 io.dataregion.default.PagesWritten 0 io.dataregion.default.PagesReplaceRate 0 io.dataregion.default.PagesRead 0 io.dataregion.default.PagesFillFactor 0.9997031688690186 io.dataregion.default.TotalAllocatedPages 2066 io.dataregion.default.PhysicalMemorySize 8511920 io.dataregion.default.PagesReplaceAge 0 io.dataregion.default.AllocationRate 2066 io.dataregion.default.OffheapUsedSize 8511920 io.dataregion.default.LargeEntriesPagesCount 0 io.dataregion.default.TotalAllocatedSize 8511920 io.dataregion.default.CheckpointBufferSize 0 TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * TotalAllocatedSize) = 8511920 - (8511920 - 0.9997031688690186 * 8511920) = 8509392 delete from city; SELECT * FROM sys.metrics WHERE name LIKE 'io.dataregion.default%'; io.dataregion.default.OffHeapSize 104857600 io.dataregion.default.PhysicalMemoryPages 2075 io.dataregion.default.EmptyDataPages 0 io.dataregion.default.UsedCheckpointBufferSize 0 io.dataregion.default.TotalThrottlingTime 0 io.dataregion.default.PagesReplaced 0 io.dataregion.default.EvictionRate 0 io.dataregion.default.InitialSize 104857600 io.dataregion.default.DirtyPages 0 io.dataregion.default.MaxSize 8589934592 io.dataregion.default.PagesWritten 0 io.dataregion.default.PagesReplaceRate 0 io.dataregion.default.PagesRead 0 io.dataregion.default.PagesFillFactor 1.0 io.dataregion.default.TotalAllocatedPages 2075 io.dataregion.default.PhysicalMemorySize 8549000 io.dataregion.default.PagesReplaceAge 0 io.dataregion.default.AllocationRate 9 io.dataregion.default.OffheapUsedSize 8549000 io.dataregion.default.LargeEntriesPagesCount 0 io.dataregion.default.TotalAllocatedSize 8549000 io.dataregion.default.CheckpointBufferSize 0 TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * TotalAllocatedSize) =8549000 - (8549000 - 1.0 * 8549000) = 8549000 Instead, the value becomes larger? 在 2022/9/19 15:13, Alexander Polovtcev 写道: Sorry, I messed up the metric names a little bit, we should use the size metrics, not the page metrics. So the correct formula would be: `TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * TotalAllocatedSize)` On Sun, Sep 18, 2022 at 4:29 PM 38797715 <38797...@qq.com> wrote: I've checked TotalUsedPages in ignite.dataRegionMetrics(), it is found that the logic of this metric is consistent with OffheapUsedSize, that is, the value will not be updated after data deletion. 在 2022/9/18 19:42, Alexander Polovtcev 写道: Hello, you can check out "TotalUsedPages" and "TotalUsedSize" metrics that can be found in the Data Region metrics. However, please keep in mind that this does not take page fragmentation into account. On Sat, Sep 17, 2022 at 2:35 PM 38797715 <38797...@qq.com> wrote: Hi team, We found that if the delete operation is execute
Re: How to calculate the amount of memory used by a cache or dataregion
Sorry, I messed up the metric names a little bit, we should use the size metrics, not the page metrics. So the correct formula would be: `TotalUsedSize - (TotalAllocatedSize - PagesFillFactor * TotalAllocatedSize)` On Sun, Sep 18, 2022 at 4:29 PM 38797715 <38797...@qq.com> wrote: > I've checked TotalUsedPages in ignite.dataRegionMetrics(), it is found > that the logic of this metric is consistent with OffheapUsedSize, that is, > the value will not be updated after data deletion. > 在 2022/9/18 19:42, Alexander Polovtcev 写道: > > Hello, you can check out "TotalUsedPages" and "TotalUsedSize" metrics that > can be found in the Data Region metrics. However, please keep in mind that > this does not take page fragmentation into account. > > On Sat, Sep 17, 2022 at 2:35 PM 38797715 <38797...@qq.com> wrote: > >> Hi team, >> >> We found that if the delete operation is executed, the memory space >> corresponding to the data will not be released, and this space will be >> reused later. >> >> Therefore, metrics such as OffheapUsedSize will become inaccurate. >> >> So, how to calculate the exact amount of memory occupied by data in a >> cache or a dataregion? >> > > > -- > With regards, > Aleksandr Polovtcev > > -- With regards, Aleksandr Polovtcev
Re: How to calculate the amount of memory used by a cache or dataregion
That's probably due to page fragmentation: TotalUsedPages shows pages that are not *completely* free. Looks like there's no direct metric responsible for this, but I discovered the following workaround: there exists a metric called "PagesFillFactor". It is computed as follows: (TotalAllocatedPages - FreeSpaceInUsedPages) / TotalAllocatedPages . The problem is that it does not take fully empty pages into account. So we can try to combine several metrics to do the correct calculation: TotalUsedPages - (TotalAllocatedPages - PagesFillFactor * TotalAllocatedPages) On Sun, Sep 18, 2022 at 4:29 PM 38797715 <38797...@qq.com> wrote: > I've checked TotalUsedPages in ignite.dataRegionMetrics(), it is found > that the logic of this metric is consistent with OffheapUsedSize, that is, > the value will not be updated after data deletion. > 在 2022/9/18 19:42, Alexander Polovtcev 写道: > > Hello, you can check out "TotalUsedPages" and "TotalUsedSize" metrics that > can be found in the Data Region metrics. However, please keep in mind that > this does not take page fragmentation into account. > > On Sat, Sep 17, 2022 at 2:35 PM 38797715 <38797...@qq.com> wrote: > >> Hi team, >> >> We found that if the delete operation is executed, the memory space >> corresponding to the data will not be released, and this space will be >> reused later. >> >> Therefore, metrics such as OffheapUsedSize will become inaccurate. >> >> So, how to calculate the exact amount of memory occupied by data in a >> cache or a dataregion? >> > > > -- > With regards, > Aleksandr Polovtcev > > -- With regards, Aleksandr Polovtcev
