Hi, Nikolay! There may be a number (count of segments * segment size) or there may be a count of segments, whichever is more convenient for the user.
15.02.2021, 13:14, "Nikolay Izhikov" <nizhi...@apache.org>: > Hello, Kirill. > > Thanks for an answers. > Now, I understand your intentions. > >> t also seems that it will be more natural to operate not just bytes but >> multiples of a segment. > > Can’t agree here. > From my point of view - it’s better to know exact number, not just «count of > segments». > >> 15 февр. 2021 г., в 13:00, ткаленко кирилл <tkalkir...@yandex.ru> >> написал(а): >> >> Hello, Nikolay! >> >> The period of one day (24h) seems more natural, you can take more or less, >> I think that one day may not be enough, and it is worth getting the metric >> for several days (collect statistics) for example a week. Yes, the total >> size of the segments may not be >> DataStorageConfiguration#getMaxWalArchiveSize, but for capacity planning, >> accuracy is not so important to us, since the load can always change, it >> will hurt users more if we overflow the archive and it will not be able to >> start the node. So to say that more is better than less, it also seems that >> it will be more natural to operate not just bytes but multiples of a segment. >> >> In separate threads, you can discuss the metric that you propose about page >> memory and indexes estimates. >> >> 14.02.2021, 11:54, "Nikolay Izhikov" <nizhi...@apache.org>: >>> Hello, Kirill >>> >>> Your conclusions still not clear for me. >>> >>>> It is not possible for us to estimate how much space a user will need >>>> in the archive so as not to overflow it under its load >>>> We take the maximum 44 and multiply it by a >>>> DataStorageConfiguration#getWalSegmentSize >>> >>> Why you take a single day (24h) for a standard period? Is there any >>> rationale behind this? >>> >>> 1. We have `walAutoArchiveAfterInactivity` property. So WAL segment can >>> have a size less than the maximum. >>> 2. For CDC feature I want to introduce «WAL force rollover timeout» to >>> make data available for a consumer in a guaranteed period [1]. >>> >>> Why does the user want to estimate those numbers in the first place? >>> Are we talking about some kind of capacity planning? >>> >>> If yes, then maybe it will be better to have a metric for a count of bytes >>> written in the WAL? >>> With it, we will have an exact number of space we need for WAL. >>> >>> How user should estimate capacity for a page memory and indexes? >>> >>> [1] https://issues.apache.org/jira/browse/IGNITE-13582 >>> >>>> 14 февр. 2021 г., в 09:48, ткаленко кирилл <tkalkir...@yandex.ru> >>>> написал(а): >>>> >>>> Hi, Nikolay! >>>> >>>> The user will be able to take the getLastArchivedSegmentIndex every day >>>> and remember it and do it, say, for several days. >>>> >>>> For example, when starting the application, the >>>> getLastArchivedSegmentIndex is 0, then at the end of the first day the >>>> value will be 30 at the end of the second 55 and at the end of the third >>>> 99. >>>> It turns out that 30 segments were used for the first day, 25 for the >>>> second and 44 for the third. We take the maximum 44 and multiply it by a >>>> DataStorageConfiguration#getWalSegmentSize, and we get the possible >>>> maximum that the archive overflow was the least likely. If the user uses >>>> compression, then it can be subtracted from the result (result * >>>> getMaxSizeCompressedArchivedSegment). >>>> >>>> 13.02.2021, 10:47, "Nikolay Izhikov" <nizhi...@apache.org>: >>>>> Hello, Kirill. >>>>> >>>>>> It is not possible for us to estimate how much space a user will need >>>>>> in the archive so as not to overflow it under its load >>>>> >>>>> It still not clear for me why do we need those metrics. >>>>> Can you please, write down specific scenario - how user will use these >>>>> metrics to estimate required WAL volume? >>>>> >>>>>> 12 февр. 2021 г., в 19:35, ткаленко кирилл <tkalkir...@yandex.ru> >>>>>> написал(а): >>>>>> >>>>>> Hi, Nikolay! >>>>>> >>>>>> It is not possible for us to estimate how much space a user will need >>>>>> in the archive so as not to overflow it under its load. And the proposed >>>>>> metrics will allow you to make a rough estimate. >>>>>> >>>>>> 12.02.2021, 17:23, "Nikolay Izhikov" <nizhi...@apache.org>: >>>>>>> Hello, Kirill. >>>>>>> >>>>>>> Can you, please, clarify - What question about WAL user have in mind? >>>>>>> And what answers he(or she) gets with these new metrics? >>>>>>> >>>>>>>> 12 февр. 2021 г., в 14:26, ткаленко кирилл <tkalkir...@yandex.ru> >>>>>>>> написал(а): >>>>>>>> >>>>>>>> Hi everyone! >>>>>>>> At the moment, I have not found an opportunity to estimate how >>>>>>>> many WAL segments fall into the archive, say per day. >>>>>>>> So I created a ticket >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-14170 to add a couple of >>>>>>>> new metrics.
