On Thu, Jul 28 2016, gordon chung wrote: > this is probably something to discuss on ops list as well eventually but > what do you think about shrinking the max size of timeseries chunks from > 14400 to something smaller? i'm curious to understand what the length of > the typical timeseries is. my main reason for bringing this up is that > even our default 'high' policy doesn't reach 14400 limit so it at most > will only split into two, partially filled objects. as we look to make a > more efficient storage format for v3(?) seems like this may be an > opportunity to change size as well (if necessary)
1 minute granularity over a year: 525600 points, 27 splits. Even in that case, which is pretty precise, that's not a lot of split I'd say. > 14400 points roughly equals 128KB object which is cool but maybe we > should target something smaller? 7200points aka 64KB? 3600 points aka > 32KB? just for reference our biggest default series is 10080 points > (1min granularity over week). It's 128 Kb if you don't compress, but if you do, it's usually way less. > that said 128KB (at most) might not be that bad from read/write pov and > maybe it's ok to keep it at 14400? i know from the test i did earlier, > the time requirement to read/write increases linearly (7200 point object > takes roughly half time of 14400 point object)[1]. i think the main item > is we don't want it too small that we're updating multiple objects at a > time. Best way is probably to do some bench… but I think it really depends on the use cases here. The interest of having many small splits is that you can parallelize the read. Considering the compression ratio we have, I think we should split in smaller files. I'd pick 3600 and give it a try. -- Julien Danjou // Free Software hacker // https://julien.danjou.info
signature.asc
Description: PGP signature
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
