A rebuild of the cash reduced the size of the data dramatically. Apparently ignite is not doing anything to rebalance or clean up pages. I can't see how anyone using ignite native seriously will not have this problem.
I wonder if this impacts the indexing also? And could be part of the lousy performance we are having with ignite native. On Wed, Jun 30, 2021, 8:27 AM Ryan Trollip <ryanonthebe...@gmail.com> wrote: > Hey Ilya > > It's the data tables that keep growing not the WAL. > We will try to rebuild the cache and see if that fixes the issue > > On Mon, Jun 28, 2021 at 8:46 AM Ilya Kasnacheev <ilya.kasnach...@gmail.com> > wrote: > >> Hello! >> >> Is it WAL (wal/) that is growing or checkpoint space (db/)? If latter, >> any specific caches that are growing unbound? >> >> If letter, you can try creating a new cache, moving the relevant data to >> this new cache, switch to using it, and then drop the old cache - should >> reclaim the space. >> >> Regards, >> -- >> Ilya Kasnacheev >> >> >> пн, 28 июн. 2021 г. в 17:34, Ryan Trollip <ryanonthebe...@gmail.com>: >> >>> Is this why the native disk storage just keeps growing and does not >>> reduce after we delete from ignite using SQL? >>> We are up to 80GB on disk now on some instances. We implemented a custom >>> archiving feature to move older data out of ignite cache to a PostgresSQL >>> database but when we delete that data from ignite instance, the disk data >>> size ignite is using stays the same, and then keeps growing, and >>> growing.... >>> >>> On Thu, Jun 24, 2021 at 7:10 PM Denis Magda <dma...@apache.org> wrote: >>> >>>> Ignite fellows, >>>> >>>> I remember some of us worked on the persistence defragmentation >>>> features. Has it been merged? >>>> >>>> @Valentin Kulichenko <valentin.kuliche...@gmail.com> probably you know >>>> the latest state. >>>> >>>> - >>>> Denis >>>> >>>> On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev < >>>> ilya.kasnach...@gmail.com> wrote: >>>> >>>>> Hello! >>>>> >>>>> You can probably drop the entire cache and then re-populate it via >>>>> loadCache(), etc. >>>>> >>>>> Regards, >>>>> -- >>>>> Ilya Kasnacheev >>>>> >>>>> >>>>> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip <ryanonthebe...@gmail.com>: >>>>> >>>>>> Thanks, Ilya, we may have to consider moving back to non-native >>>>>> storage and caching more selectively as the performance degrades when >>>>>> there >>>>>> is a lot of write/delete activity or tables with large amounts of rows. >>>>>> This is with SQL with indexes and the use of query plans etc. >>>>>> >>>>>> Is there any easy way to rebuild the entire native database after >>>>>> hours? e.g. with a batch run on the weeknds? >>>>>> >>>>>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev < >>>>>> ilya.kasnach...@gmail.com> wrote: >>>>>> >>>>>>> Hello! >>>>>>> >>>>>>> I don't think there's anything ready to use, but "killing >>>>>>> performance" from fragmentation is also not something reported too >>>>>>> often. >>>>>>> >>>>>>> Regards, >>>>>>> -- >>>>>>> Ilya Kasnacheev >>>>>>> >>>>>>> >>>>>>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip <ryanonthebe...@gmail.com >>>>>>> >: >>>>>>> >>>>>>>> We see continual very large growth to data with ignite native. We >>>>>>>> have a very chatty use case that's creating and deleting stuff often. >>>>>>>> The >>>>>>>> data on disk just keeps growing at an explosive rate. So much so we >>>>>>>> ported >>>>>>>> this to a DB to see the difference and the DB is much smaller. I was >>>>>>>> searching to see if someone has the same issue. This is also killing >>>>>>>> performance. >>>>>>>> >>>>>>>> Founds this: >>>>>>>> >>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation >>>>>>>> >>>>>>>> Apparently, there is no auto-rebalancing of pages? or cleanup of >>>>>>>> pages? >>>>>>>> >>>>>>>> Has anyone implemented a workaround to rebuild the cache and >>>>>>>> indexes say on a weekly basis to get it to behave reasonably? >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>
