Hello, Kirill. Thanks for an answers. Now, I understand your intentions.
> t also seems that it will be more natural to operate not just bytes but > multiples of a segment. Can’t agree here. From my point of view - it’s better to know exact number, not just «count of segments». > 15 февр. 2021 г., в 13:00, ткаленко кирилл <tkalkir...@yandex.ru> написал(а): > > Hello, Nikolay! > > The period of one day (24h) seems more natural, you can take more or less, I > think that one day may not be enough, and it is worth getting the metric for > several days (collect statistics) for example a week. Yes, the total size of > the segments may not be DataStorageConfiguration#getMaxWalArchiveSize, but > for capacity planning, accuracy is not so important to us, since the load can > always change, it will hurt users more if we overflow the archive and it will > not be able to start the node. So to say that more is better than less, it > also seems that it will be more natural to operate not just bytes but > multiples of a segment. > > In separate threads, you can discuss the metric that you propose about page > memory and indexes estimates. > > > 14.02.2021, 11:54, "Nikolay Izhikov" <nizhi...@apache.org>: >> Hello, Kirill >> >> Your conclusions still not clear for me. >> >>> It is not possible for us to estimate how much space a user will need in >>> the archive so as not to overflow it under its load >>> We take the maximum 44 and multiply it by a >>> DataStorageConfiguration#getWalSegmentSize >> >> Why you take a single day (24h) for a standard period? Is there any >> rationale behind this? >> >> 1. We have `walAutoArchiveAfterInactivity` property. So WAL segment can have >> a size less than the maximum. >> 2. For CDC feature I want to introduce «WAL force rollover timeout» to make >> data available for a consumer in a guaranteed period [1]. >> >> Why does the user want to estimate those numbers in the first place? >> Are we talking about some kind of capacity planning? >> >> If yes, then maybe it will be better to have a metric for a count of bytes >> written in the WAL? >> With it, we will have an exact number of space we need for WAL. >> >> How user should estimate capacity for a page memory and indexes? >> >> [1] https://issues.apache.org/jira/browse/IGNITE-13582 >> >>> 14 февр. 2021 г., в 09:48, ткаленко кирилл <tkalkir...@yandex.ru> >>> написал(а): >>> >>> Hi, Nikolay! >>> >>> The user will be able to take the getLastArchivedSegmentIndex every day >>> and remember it and do it, say, for several days. >>> >>> For example, when starting the application, the >>> getLastArchivedSegmentIndex is 0, then at the end of the first day the >>> value will be 30 at the end of the second 55 and at the end of the third 99. >>> It turns out that 30 segments were used for the first day, 25 for the >>> second and 44 for the third. We take the maximum 44 and multiply it by a >>> DataStorageConfiguration#getWalSegmentSize, and we get the possible maximum >>> that the archive overflow was the least likely. If the user uses >>> compression, then it can be subtracted from the result (result * >>> getMaxSizeCompressedArchivedSegment). >>> >>> 13.02.2021, 10:47, "Nikolay Izhikov" <nizhi...@apache.org>: >>>> Hello, Kirill. >>>> >>>>> It is not possible for us to estimate how much space a user will need >>>>> in the archive so as not to overflow it under its load >>>> >>>> It still not clear for me why do we need those metrics. >>>> Can you please, write down specific scenario - how user will use these >>>> metrics to estimate required WAL volume? >>>> >>>>> 12 февр. 2021 г., в 19:35, ткаленко кирилл <tkalkir...@yandex.ru> >>>>> написал(а): >>>>> >>>>> Hi, Nikolay! >>>>> >>>>> It is not possible for us to estimate how much space a user will need >>>>> in the archive so as not to overflow it under its load. And the proposed >>>>> metrics will allow you to make a rough estimate. >>>>> >>>>> 12.02.2021, 17:23, "Nikolay Izhikov" <nizhi...@apache.org>: >>>>>> Hello, Kirill. >>>>>> >>>>>> Can you, please, clarify - What question about WAL user have in mind? >>>>>> And what answers he(or she) gets with these new metrics? >>>>>> >>>>>>> 12 февр. 2021 г., в 14:26, ткаленко кирилл <tkalkir...@yandex.ru> >>>>>>> написал(а): >>>>>>> >>>>>>> Hi everyone! >>>>>>> At the moment, I have not found an opportunity to estimate how many >>>>>>> WAL segments fall into the archive, say per day. >>>>>>> So I created a ticket >>>>>>> https://issues.apache.org/jira/browse/IGNITE-14170 to add a couple of >>>>>>> new metrics.
