A rebuild of the cash reduced the size of the data dramatically.
Apparently ignite is not doing anything to rebalance or clean up pages.
I can't see how anyone using ignite native seriously will not have this
problem.
I wonder if this impacts the indexing also? And could be part of the lousy
performance we are having with ignite native.
On Wed, Jun 30, 2021, 8:27 AM Ryan Trollip wrote:
> Hey Ilya
>
> It's the data tables that keep growing not the WAL.
> We will try to rebuild the cache and see if that fixes the issue
>
> On Mon, Jun 28, 2021 at 8:46 AM Ilya Kasnacheev
> wrote:
>
>> Hello!
>>
>> Is it WAL (wal/) that is growing or checkpoint space (db/)? If latter,
>> any specific caches that are growing unbound?
>>
>> If letter, you can try creating a new cache, moving the relevant data to
>> this new cache, switch to using it, and then drop the old cache - should
>> reclaim the space.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 28 июн. 2021 г. в 17:34, Ryan Trollip :
>>
>>> Is this why the native disk storage just keeps growing and does not
>>> reduce after we delete from ignite using SQL?
>>> We are up to 80GB on disk now on some instances. We implemented a custom
>>> archiving feature to move older data out of ignite cache to a PostgresSQL
>>> database but when we delete that data from ignite instance, the disk data
>>> size ignite is using stays the same, and then keeps growing, and
>>> growing
>>>
>>> On Thu, Jun 24, 2021 at 7:10 PM Denis Magda wrote:
>>>
Ignite fellows,
I remember some of us worked on the persistence defragmentation
features. Has it been merged?
@Valentin Kulichenko probably you know
the latest state.
-
Denis
On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev <
ilya.kasnach...@gmail.com> wrote:
> Hello!
>
> You can probably drop the entire cache and then re-populate it via
> loadCache(), etc.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip :
>
>> Thanks, Ilya, we may have to consider moving back to non-native
>> storage and caching more selectively as the performance degrades when
>> there
>> is a lot of write/delete activity or tables with large amounts of rows.
>> This is with SQL with indexes and the use of query plans etc.
>>
>> Is there any easy way to rebuild the entire native database after
>> hours? e.g. with a batch run on the weeknds?
>>
>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> I don't think there's anything ready to use, but "killing
>>> performance" from fragmentation is also not something reported too
>>> often.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip >> >:
>>>
We see continual very large growth to data with ignite native. We
have a very chatty use case that's creating and deleting stuff often.
The
data on disk just keeps growing at an explosive rate. So much so we
ported
this to a DB to see the difference and the DB is much smaller. I was
searching to see if someone has the same issue. This is also killing
performance.
Founds this:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
Apparently, there is no auto-rebalancing of pages? or cleanup of
pages?
Has anyone implemented a workaround to rebuild the cache and
indexes say on a weekly basis to get it to behave reasonably?
Thanks
>>>
