Re: Cache expiry policy not deleting records from disk(native persistence)

[Artem Budnikov]

Hi Denis,

That's on the expiry policies page: 
https://apacheignite.readme.io/docs/expiry-policies


On 13.09.2019 19:46, Denis Magda wrote:
Artem, thanks, could you please share a reference to the updated page? 
Can't find anything here:

https://apacheignite.readme.io/docs/evictions

Shiva, I've restarted the discussion on the dev list, to get to the 
bottom of this gap and how it can be addressed:

http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html

-
Denis


On Fri, Sep 13, 2019 at 7:05 AM Artem Budnikov 
mailto:a.budnikov.ign...@gmail.com>> wrote:


Hi Denis,

I updated the page about eviction policies. Not freeing up space
on disk was not implemented for reasons explained in the dev-list
thread. I'll update the page once more if/when a solution is
implemented.

Artem

On 13.09.2019 00:34, Denis Magda wrote:

Shiva,

Hopefully, someone from the dev community will pick this ticket
up soon and solve the task. In the meantime, Artem, would you
mind documenting this limitation referring to ticket 10862?

-
Denis


On Tue, Sep 10, 2019 at 12:50 AM Shiva Kumar
mailto:shivakumar@gmail.com>> wrote:

I have filed a bug
https://issues.apache.org/jira/browse/IGNITE-12152 but this
is same as https://issues.apache.org/jira/browse/IGNITE-10862
Any idea on the timeline of these tickets?
In the documentation
https://apacheignite.readme.io/v2.7/docs/expiry-policies
it says when native persistence is enabled "*expired entries
are removed from both memory and disk tiers*" but in the disk
it just mark the pages as unwanted pages and same disk space
used by these unwanted pages will be used to store new pages
but it will not remove unwanted pages from disk and so it
will not release disk space used by these unwanted pages.

here is the developer's discussion link

http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html


On Mon, Sep 9, 2019 at 11:53 PM Shiva Kumar
mailto:shivakumar@gmail.com>>
wrote:

Hi
I have deployed ignite on kubernetes and configured two
seperate persistent volume for WAL and persistence.
The issue Iam facing is same as
https://issues.apache.org/jira/browse/IGNITE-10862

Thanks
Shiva

On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov,
mailto:aealexsand...@gmail.com>> wrote:

Hello,

I guess that generated WAL will take this disk space.
Please read about WAL here:

https://apacheignite.readme.io/docs/write-ahead-log

Please provide the size of every folder under
/opt/ignite/persistence.

BR,
Andrei

9/6/2019 9:45 PM, Shiva Kumar пишет:

Hi all,
I have set cache expiry policy like this


 
 
          
              
                
                
                
                
                
                  

                      


                      

                  
                

              
          
 


And batch inserting records to one of the table
which is created with above cache template.
Around 10 minutes, I ingested ~1.5GB of data and
after 10 minutes records started reducing(expiring)
when I monitored from sqlline.

0: jdbc:ignite:thin://192.168.*.*:10800> select
count(ID) from DIMENSIONS;


COUNT(ID)


248896

1 row selected (0.86 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select
count(ID) from DIMENSIONS;


COUNT(ID)


222174

1 row selected (0.313 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select
count(ID) from DIMENSIONS;


COUNT(

Re: Cache expiry policy not deleting records from disk(native persistence)

[Denis Magda]

Artem, thanks, could you please share a reference to the updated page?
Can't find anything here:
https://apacheignite.readme.io/docs/evictions

Shiva, I've restarted the discussion on the dev list, to get to the bottom
of this gap and how it can be addressed:
http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html

-
Denis


On Fri, Sep 13, 2019 at 7:05 AM Artem Budnikov 
wrote:

> Hi Denis,
>
> I updated the page about eviction policies. Not freeing up space on disk
> was not implemented for reasons explained in the dev-list thread. I'll
> update the page once more if/when a solution is implemented.
>
> Artem
> On 13.09.2019 00:34, Denis Magda wrote:
>
> Shiva,
>
> Hopefully, someone from the dev community will pick this ticket up soon
> and solve the task. In the meantime, Artem, would you mind documenting this
> limitation referring to ticket 10862?
>
> -
> Denis
>
>
> On Tue, Sep 10, 2019 at 12:50 AM Shiva Kumar 
> wrote:
>
>> I have filed a bug https://issues.apache.org/jira/browse/IGNITE-12152 but
>> this is same as https://issues.apache.org/jira/browse/IGNITE-10862
>> Any idea on the timeline of these tickets?
>> In the documentation
>> https://apacheignite.readme.io/v2.7/docs/expiry-policies
>> it says when native persistence is enabled "*expired entries are removed
>> from both memory and disk tiers*" but in the disk it just mark the pages
>> as unwanted pages and same disk space used by these unwanted pages will be
>> used to store new pages but it will not remove unwanted pages from disk and
>> so it will not release disk space used by these unwanted pages.
>>
>> here is the developer's discussion link
>>
>> http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html
>>
>>
>> On Mon, Sep 9, 2019 at 11:53 PM Shiva Kumar 
>> wrote:
>>
>>> Hi
>>> I have deployed ignite on kubernetes and configured two seperate
>>> persistent volume for WAL and persistence.
>>> The issue Iam facing is same as
>>> https://issues.apache.org/jira/browse/IGNITE-10862
>>>
>>> Thanks
>>> Shiva
>>>
>>> On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov, <
>>> aealexsand...@gmail.com> wrote:
>>>
 Hello,

 I guess that generated WAL will take this disk space. Please read about
 WAL here:

 https://apacheignite.readme.io/docs/write-ahead-log

 Please provide the size of every folder under /opt/ignite/persistence.

 BR,
 Andrei
 9/6/2019 9:45 PM, Shiva Kumar пишет:

 Hi all,
 I have set cache expiry policy like this




 
 >>> class="org.apache.ignite.configuration.CacheConfiguration">
   
   
   
   
   
 >>> class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
   
 
   
   
 
   
 
   

 
 



 And batch inserting records to one of the table which is created with
 above cache template.
 Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes
 records started reducing(expiring) when I monitored from sqlline.

 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
 DIMENSIONS;
 

 COUNT(ID)
 

 248896
 
 1 row selected (0.86 seconds)
 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
 DIMENSIONS;
 

 COUNT(ID)
 

 222174
 
 1 row selected (0.313 seconds)
 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
 DIMENSIONS;
 

 COUNT(ID)
 

 118154
 
 1 row selected (0.15 seconds)
 0: jdbc:ignite:thin://192.168.*.*:10800>
 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
 DIMENSIONS;
 

 COUNT(ID)
 

 76061
 
 1 row selected (0.106 seconds)
 0: jdbc:ignite:thin://192.168.*.*:10800>
 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
 DIMENSIONS;
 

 COUNT(ID)
 

 41671
 
>>

Re: Cache expiry policy not deleting records from disk(native persistence)

[Artem Budnikov]

Hi Denis,

I updated the page about eviction policies. Not freeing up space on disk 
was not implemented for reasons explained in the dev-list thread. I'll 
update the page once more if/when a solution is implemented.


Artem

On 13.09.2019 00:34, Denis Magda wrote:

Shiva,

Hopefully, someone from the dev community will pick this ticket up 
soon and solve the task. In the meantime, Artem, would you mind 
documenting this limitation referring to ticket 10862?


-
Denis


On Tue, Sep 10, 2019 at 12:50 AM Shiva Kumar > wrote:


I have filed a bug
https://issues.apache.org/jira/browse/IGNITE-12152 but this is
same as https://issues.apache.org/jira/browse/IGNITE-10862
Any idea on the timeline of these tickets?
In the documentation
https://apacheignite.readme.io/v2.7/docs/expiry-policies
it says when native persistence is enabled "*expired entries are
removed from both memory and disk tiers*" but in the disk it just
mark the pages as unwanted pages and same disk space used by these
unwanted pages will be used to store new pages but it will not
remove unwanted pages from disk and so it will not release disk
space used by these unwanted pages.

here is the developer's discussion link

http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html


On Mon, Sep 9, 2019 at 11:53 PM Shiva Kumar
mailto:shivakumar@gmail.com>> wrote:

Hi
I have deployed ignite on kubernetes and configured two
seperate persistent volume for WAL and persistence.
The issue Iam facing is same as
https://issues.apache.org/jira/browse/IGNITE-10862

Thanks
Shiva

On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov,
mailto:aealexsand...@gmail.com>> wrote:

Hello,

I guess that generated WAL will take this disk space.
Please read about WAL here:

https://apacheignite.readme.io/docs/write-ahead-log

Please provide the size of every folder under
/opt/ignite/persistence.

BR,
Andrei

9/6/2019 9:45 PM, Shiva Kumar пишет:

Hi all,
I have set cache expiry policy like this


 
 
          
              
                
                
                
                
                
                  
                    
                      
                        
                        
                      
                    
                  
                

              
          
 


And batch inserting records to one of the table which is
created with above cache template.
Around 10 minutes, I ingested ~1.5GB of data and after 10
minutes records started reducing(expiring) when I
monitored from sqlline.

0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID)
from DIMENSIONS;


COUNT(ID)


248896

1 row selected (0.86 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID)
from DIMENSIONS;


COUNT(ID)


222174

1 row selected (0.313 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID)
from DIMENSIONS;


COUNT(ID)


118154

1 row selected (0.15 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID)
from DIMENSIONS;


COUNT(ID)


76061

1 row selected (0.106 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID)
from DIMENSIONS;


COUNT(ID)


41671

1 row selected (0.063 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select coun

Re: Cache expiry policy not deleting records from disk(native persistence)

[rick_tem]

This is very troublesome.  Persistence will kill your disk and LRU doesn't
seem to work for offheap with persistence disabled (see "Data region LRU
offheap algo not working")It seems the only way to expire items is to
keep it on heap with an expiry policy

Regards,
Rick



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Cache expiry policy not deleting records from disk(native persistence)

[Denis Magda]

Shiva,

Hopefully, someone from the dev community will pick this ticket up soon and
solve the task. In the meantime, Artem, would you mind documenting this
limitation referring to ticket 10862?

-
Denis


On Tue, Sep 10, 2019 at 12:50 AM Shiva Kumar 
wrote:

> I have filed a bug https://issues.apache.org/jira/browse/IGNITE-12152 but
> this is same as https://issues.apache.org/jira/browse/IGNITE-10862
> Any idea on the timeline of these tickets?
> In the documentation
> https://apacheignite.readme.io/v2.7/docs/expiry-policies
> it says when native persistence is enabled "*expired entries are removed
> from both memory and disk tiers*" but in the disk it just mark the pages
> as unwanted pages and same disk space used by these unwanted pages will be
> used to store new pages but it will not remove unwanted pages from disk and
> so it will not release disk space used by these unwanted pages.
>
> here is the developer's discussion link
>
> http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html
>
>
> On Mon, Sep 9, 2019 at 11:53 PM Shiva Kumar 
> wrote:
>
>> Hi
>> I have deployed ignite on kubernetes and configured two seperate
>> persistent volume for WAL and persistence.
>> The issue Iam facing is same as
>> https://issues.apache.org/jira/browse/IGNITE-10862
>>
>> Thanks
>> Shiva
>>
>> On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov, <
>> aealexsand...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I guess that generated WAL will take this disk space. Please read about
>>> WAL here:
>>>
>>> https://apacheignite.readme.io/docs/write-ahead-log
>>>
>>> Please provide the size of every folder under /opt/ignite/persistence.
>>>
>>> BR,
>>> Andrei
>>> 9/6/2019 9:45 PM, Shiva Kumar пишет:
>>>
>>> Hi all,
>>> I have set cache expiry policy like this
>>>
>>>
>>>
>>>
>>> 
>>> >> class="org.apache.ignite.configuration.CacheConfiguration">
>>>   
>>>   
>>>   
>>>   
>>>   
>>> >> factory-method="factoryOf">
>>>   
>>> 
>>>   
>>>   
>>> 
>>>   
>>> 
>>>   
>>>
>>> 
>>> 
>>>
>>>
>>>
>>> And batch inserting records to one of the table which is created with
>>> above cache template.
>>> Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes
>>> records started reducing(expiring) when I monitored from sqlline.
>>>
>>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
>>> DIMENSIONS;
>>> 
>>>
>>> COUNT(ID)
>>> 
>>>
>>> 248896
>>> 
>>> 1 row selected (0.86 seconds)
>>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
>>> DIMENSIONS;
>>> 
>>>
>>> COUNT(ID)
>>> 
>>>
>>> 222174
>>> 
>>> 1 row selected (0.313 seconds)
>>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
>>> DIMENSIONS;
>>> 
>>>
>>> COUNT(ID)
>>> 
>>>
>>> 118154
>>> 
>>> 1 row selected (0.15 seconds)
>>> 0: jdbc:ignite:thin://192.168.*.*:10800>
>>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
>>> DIMENSIONS;
>>> 
>>>
>>> COUNT(ID)
>>> 
>>>
>>> 76061
>>> 
>>> 1 row selected (0.106 seconds)
>>> 0: jdbc:ignite:thin://192.168.*.*:10800>
>>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
>>> DIMENSIONS;
>>> 
>>>
>>> COUNT(ID)
>>> 
>>>
>>> 41671
>>> 
>>> 1 row selected (0.063 seconds)
>>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
>>> DIMENSIONS;
>>> 
>>>
>>> COUNT(ID)
>>> 
>>>
>>> 18455
>>> 
>>> 1 row selected (0.037 seconds)
>>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from
>>> DIMENSIONS;
>>> 
>>>
>>> COUNT(ID)
>>> 
>>>
>>> 0
>>> 
>>> 1 row selected (0.014 seconds)
>>>
>>>
>>> But in the meantime, the disk space used by the persistence store was in
>>> the same usage level instead of decreasing.
>>>
>>>
>>> [ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h
>>> /opt/ignite/persistence/; sleep 1s; done
>>> Filesystem Size Used Avail Use% Mounted on
>>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>> Filesystem Size Used Avail Use% Mounted on
>>>

Re: Cache expiry policy not deleting records from disk(native persistence)

[Shiva Kumar]

I have filed a bug https://issues.apache.org/jira/browse/IGNITE-12152 but
this is same as https://issues.apache.org/jira/browse/IGNITE-10862
Any idea on the timeline of these tickets?
In the documentation
https://apacheignite.readme.io/v2.7/docs/expiry-policies
it says when native persistence is enabled "*expired entries are removed
from both memory and disk tiers*" but in the disk it just mark the pages as
unwanted pages and same disk space used by these unwanted pages will be
used to store new pages but it will not remove unwanted pages from disk and
so it will not release disk space used by these unwanted pages.

here is the developer's discussion link
http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html


On Mon, Sep 9, 2019 at 11:53 PM Shiva Kumar 
wrote:

> Hi
> I have deployed ignite on kubernetes and configured two seperate
> persistent volume for WAL and persistence.
> The issue Iam facing is same as
> https://issues.apache.org/jira/browse/IGNITE-10862
>
> Thanks
> Shiva
>
> On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov, 
> wrote:
>
>> Hello,
>>
>> I guess that generated WAL will take this disk space. Please read about
>> WAL here:
>>
>> https://apacheignite.readme.io/docs/write-ahead-log
>>
>> Please provide the size of every folder under /opt/ignite/persistence.
>>
>> BR,
>> Andrei
>> 9/6/2019 9:45 PM, Shiva Kumar пишет:
>>
>> Hi all,
>> I have set cache expiry policy like this
>>
>>
>>
>>
>> 
>> > class="org.apache.ignite.configuration.CacheConfiguration">
>>   
>>   
>>   
>>   
>>   
>> > factory-method="factoryOf">
>>   
>> 
>>   
>>   
>> 
>>   
>> 
>>   
>>
>> 
>> 
>>
>>
>>
>> And batch inserting records to one of the table which is created with
>> above cache template.
>> Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes records
>> started reducing(expiring) when I monitored from sqlline.
>>
>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
>> 
>>
>> COUNT(ID)
>> 
>>
>> 248896
>> 
>> 1 row selected (0.86 seconds)
>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
>> 
>>
>> COUNT(ID)
>> 
>>
>> 222174
>> 
>> 1 row selected (0.313 seconds)
>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
>> 
>>
>> COUNT(ID)
>> 
>>
>> 118154
>> 
>> 1 row selected (0.15 seconds)
>> 0: jdbc:ignite:thin://192.168.*.*:10800>
>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
>> 
>>
>> COUNT(ID)
>> 
>>
>> 76061
>> 
>> 1 row selected (0.106 seconds)
>> 0: jdbc:ignite:thin://192.168.*.*:10800>
>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
>> 
>>
>> COUNT(ID)
>> 
>>
>> 41671
>> 
>> 1 row selected (0.063 seconds)
>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
>> 
>>
>> COUNT(ID)
>> 
>>
>> 18455
>> 
>> 1 row selected (0.037 seconds)
>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
>> 
>>
>> COUNT(ID)
>> 
>>
>> 0
>> 
>> 1 row selected (0.014 seconds)
>>
>>
>> But in the meantime, the disk space used by the persistence store was in
>> the same usage level instead of decreasing.
>>
>>
>> [ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h
>> /opt/ignite/persistence/; sleep 1s; done
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>> Filesystem Size Used Avail Use% Mo

Re: Cache expiry policy not deleting records from disk(native persistence)

[Shiva Kumar]

Hi
I have deployed ignite on kubernetes and configured two seperate persistent
volume for WAL and persistence.
The issue Iam facing is same as
https://issues.apache.org/jira/browse/IGNITE-10862

Thanks
Shiva

On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov, 
wrote:

> Hello,
>
> I guess that generated WAL will take this disk space. Please read about
> WAL here:
>
> https://apacheignite.readme.io/docs/write-ahead-log
>
> Please provide the size of every folder under /opt/ignite/persistence.
>
> BR,
> Andrei
> 9/6/2019 9:45 PM, Shiva Kumar пишет:
>
> Hi all,
> I have set cache expiry policy like this
>
>
>
>
> 
>  class="org.apache.ignite.configuration.CacheConfiguration">
>   
>   
>   
>   
>   
>  factory-method="factoryOf">
>   
> 
>   
>   
> 
>   
> 
>   
>
> 
> 
>
>
>
> And batch inserting records to one of the table which is created with
> above cache template.
> Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes records
> started reducing(expiring) when I monitored from sqlline.
>
> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
> 
>
> COUNT(ID)
> 
>
> 248896
> 
> 1 row selected (0.86 seconds)
> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
> 
>
> COUNT(ID)
> 
>
> 222174
> 
> 1 row selected (0.313 seconds)
> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
> 
>
> COUNT(ID)
> 
>
> 118154
> 
> 1 row selected (0.15 seconds)
> 0: jdbc:ignite:thin://192.168.*.*:10800>
> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
> 
>
> COUNT(ID)
> 
>
> 76061
> 
> 1 row selected (0.106 seconds)
> 0: jdbc:ignite:thin://192.168.*.*:10800>
> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
> 
>
> COUNT(ID)
> 
>
> 41671
> 
> 1 row selected (0.063 seconds)
> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
> 
>
> COUNT(ID)
> 
>
> 18455
> 
> 1 row selected (0.037 seconds)
> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;
> 
>
> COUNT(ID)
> 
>
> 0
> 
> 1 row selected (0.014 seconds)
>
>
> But in the meantime, the disk space used by the persistence store was in
> the same usage level instead of decreasing.
>
>
> [ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h
> /opt/ignite/persistence/; sleep 1s; done
> Filesystem Size Used Avail Use% Mounted on
> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
> Filesystem Size Used Avail Use% Mounted on
> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
> Filesystem Size Used Avail Use% Mounted on
> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
> Filesystem Size Used Avail Use% Mounted on
> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
> Filesystem Size Used Avail Use% Mounted on
> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
> Filesystem Size Used Avail Use% Mounted on
> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
> Filesystem Size Used Avail Use% Mounted on
> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>
>
>
> This means that expiry policy not deleting records from the disk, but
> ignite document says when expiry policy is set and native persistence is
> enabled then it deletes records from disk as well.
> Am I missing some configuration?
> Any help is appreciated.
>
> Shiva
>
>

Re: Cache expiry policy not deleting records from disk(native persistence)

[Andrei Aleksandrov]

Hello,

I guess that generated WAL will take this disk space. Please read about 
WAL here:


https://apacheignite.readme.io/docs/write-ahead-log

Please provide the size of every folder under /opt/ignite/persistence.

BR,
Andrei

9/6/2019 9:45 PM, Shiva Kumar пишет:

Hi all,
I have set cache expiry policy like this


 
 
          
              class="org.apache.ignite.configuration.CacheConfiguration">

                
                
                
                
                
                  factory-method="factoryOf">

                    
                      
                        
                        
                      
                    
                  
                

              
          
 


And batch inserting records to one of the table which is created with 
above cache template.
Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes 
records started reducing(expiring) when I monitored from sqlline.


0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;


COUNT(ID)


248896

1 row selected (0.86 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;


COUNT(ID)


222174

1 row selected (0.313 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;


COUNT(ID)


118154

1 row selected (0.15 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;


COUNT(ID)


76061

1 row selected (0.106 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;


COUNT(ID)


41671

1 row selected (0.063 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;


COUNT(ID)


18455

1 row selected (0.037 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;


COUNT(ID)


0

1 row selected (0.014 seconds)


But in the meantime, the disk space used by the persistence store was 
in the same usage level instead of decreasing.



[ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h 
/opt/ignite/persistence/; sleep 1s; done

Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence



This means that expiry policy not deleting records from the disk, but 
ignite document says when expiry policy is set and native persistence 
is enabled then it deletes records from disk as well.

Am I missing some configuration?
Any help is appreciated.

Shiva

Cache expiry policy not deleting records from disk(native persistence)

[Shiva Kumar]

Hi all,
I have set cache expiry policy like this


   
   


  
  
  
  
  

  

  
  

  

  



   


And batch inserting records to one of the table which is created with above
cache template.
Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes records
started reducing(expiring) when I monitored from sqlline.

0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;


COUNT(ID)


248896

1 row selected (0.86 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;


COUNT(ID)


222174

1 row selected (0.313 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;


COUNT(ID)


118154

1 row selected (0.15 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;


COUNT(ID)


76061

1 row selected (0.106 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800>
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;


COUNT(ID)


41671

1 row selected (0.063 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;


COUNT(ID)


18455

1 row selected (0.037 seconds)
0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS;


COUNT(ID)


0

1 row selected (0.014 seconds)


But in the meantime, the disk space used by the persistence store was in
the same usage level instead of decreasing.


[ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h
/opt/ignite/persistence/; sleep 1s; done
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
Filesystem Size Used Avail Use% Mounted on
/dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence



This means that expiry policy not deleting records from the disk, but
ignite document says when expiry policy is set and native persistence is
enabled then it deletes records from disk as well.
Am I missing some configuration?
Any help is appreciated.

Shiva