Hello, Nikolay! The period of one day (24h) seems more natural, you can take more or less, I think that one day may not be enough, and it is worth getting the metric for several days (collect statistics) for example a week. Yes, the total size of the segments may not be DataStorageConfiguration#getMaxWalArchiveSize, but for capacity planning, accuracy is not so important to us, since the load can always change, it will hurt users more if we overflow the archive and it will not be able to start the node. So to say that more is better than less, it also seems that it will be more natural to operate not just bytes but multiples of a segment.
In separate threads, you can discuss the metric that you propose about page memory and indexes estimates. 14.02.2021, 11:54, "Nikolay Izhikov" <nizhi...@apache.org>: > Hello, Kirill > > Your conclusions still not clear for me. > >> It is not possible for us to estimate how much space a user will need in >> the archive so as not to overflow it under its load >> We take the maximum 44 and multiply it by a >> DataStorageConfiguration#getWalSegmentSize > > Why you take a single day (24h) for a standard period? Is there any rationale > behind this? > > 1. We have `walAutoArchiveAfterInactivity` property. So WAL segment can have > a size less than the maximum. > 2. For CDC feature I want to introduce «WAL force rollover timeout» to make > data available for a consumer in a guaranteed period [1]. > > Why does the user want to estimate those numbers in the first place? > Are we talking about some kind of capacity planning? > > If yes, then maybe it will be better to have a metric for a count of bytes > written in the WAL? > With it, we will have an exact number of space we need for WAL. > > How user should estimate capacity for a page memory and indexes? > > [1] https://issues.apache.org/jira/browse/IGNITE-13582 > >> 14 февр. 2021 г., в 09:48, ткаленко кирилл <tkalkir...@yandex.ru> >> написал(а): >> >> Hi, Nikolay! >> >> The user will be able to take the getLastArchivedSegmentIndex every day and >> remember it and do it, say, for several days. >> >> For example, when starting the application, the getLastArchivedSegmentIndex >> is 0, then at the end of the first day the value will be 30 at the end of >> the second 55 and at the end of the third 99. >> It turns out that 30 segments were used for the first day, 25 for the >> second and 44 for the third. We take the maximum 44 and multiply it by a >> DataStorageConfiguration#getWalSegmentSize, and we get the possible maximum >> that the archive overflow was the least likely. If the user uses >> compression, then it can be subtracted from the result (result * >> getMaxSizeCompressedArchivedSegment). >> >> 13.02.2021, 10:47, "Nikolay Izhikov" <nizhi...@apache.org>: >>> Hello, Kirill. >>> >>>> It is not possible for us to estimate how much space a user will need in >>>> the archive so as not to overflow it under its load >>> >>> It still not clear for me why do we need those metrics. >>> Can you please, write down specific scenario - how user will use these >>> metrics to estimate required WAL volume? >>> >>>> 12 февр. 2021 г., в 19:35, ткаленко кирилл <tkalkir...@yandex.ru> >>>> написал(а): >>>> >>>> Hi, Nikolay! >>>> >>>> It is not possible for us to estimate how much space a user will need in >>>> the archive so as not to overflow it under its load. And the proposed >>>> metrics will allow you to make a rough estimate. >>>> >>>> 12.02.2021, 17:23, "Nikolay Izhikov" <nizhi...@apache.org>: >>>>> Hello, Kirill. >>>>> >>>>> Can you, please, clarify - What question about WAL user have in mind? >>>>> And what answers he(or she) gets with these new metrics? >>>>> >>>>>> 12 февр. 2021 г., в 14:26, ткаленко кирилл <tkalkir...@yandex.ru> >>>>>> написал(а): >>>>>> >>>>>> Hi everyone! >>>>>> At the moment, I have not found an opportunity to estimate how many >>>>>> WAL segments fall into the archive, say per day. >>>>>> So I created a ticket >>>>>> https://issues.apache.org/jira/browse/IGNITE-14170 to add a couple of >>>>>> new metrics.
