Re: rpki-client disk utilization / noting mfs in man?
Heho, Just as an update now that everything ran; I adjusted the disk-runs to rsync a full root to the disk to make things a bit more realistic; Ultimately, the results show that, indeed, softdep ends up being ~as fast as mfs (apart from reduced disk activity for mfs, of course), with noatime having limited effect. https://storage.fiebig.nl/s/H4ZHCwPN85yg4zN Will add an update accordingly. :-) With best regards, Tobais -Original Message- From: owner-m...@openbsd.org On Behalf Of Tobias Fiebig Sent: Monday, 1 August 2022 21:34 To: misc@openbsd.org Subject: Re: rpki-client disk utilization / noting mfs in man? Heho, > BTW rpki-client is one of the (relatively few) cases where softdep is likely > to give a significant improvement in performance. I took this as motivation to do some benchmarks (defaults, noatime, softdep, noatime+softdep, mfs, mfs+noatime) on a VM with eight cores and 8gb of memory using a dedicated 10gb disk for /var/cache/rpki-client. (Preliminary) results here (currently avg. over 7-8 runs, will be 11): https://storage.fiebig.nl/s/84TpQCTrQpa3S9j Bottomline: - Things are a lot faster all of a sudden. This might be related to more cores (8 vs. 2 before) or more memory (8 vs. 4 before), a background task having been running on the disks during the last tests, or the negative impact of those initial boxes having had all files in one partition. I will at least test the last case after the current benchmark has completed. - noatime does not seem to have a major effect, and seems to reduce performance upon import. - You are right; softdep is nearly as good as mfs. With best regards, Tobias
Re: rpki-client disk utilization / noting mfs in man?
Heho, > BTW rpki-client is one of the (relatively few) cases where softdep is likely > to give a significant improvement in performance. I took this as motivation to do some benchmarks (defaults, noatime, softdep, noatime+softdep, mfs, mfs+noatime) on a VM with eight cores and 8gb of memory using a dedicated 10gb disk for /var/cache/rpki-client. (Preliminary) results here (currently avg. over 7-8 runs, will be 11): https://storage.fiebig.nl/s/84TpQCTrQpa3S9j Bottomline: - Things are a lot faster all of a sudden. This might be related to more cores (8 vs. 2 before) or more memory (8 vs. 4 before), a background task having been running on the disks during the last tests, or the negative impact of those initial boxes having had all files in one partition. I will at least test the last case after the current benchmark has completed. - noatime does not seem to have a major effect, and seems to reduce performance upon import. - You are right; softdep is nearly as good as mfs. With best regards, Tobias
Re: rpki-client disk utilization / noting mfs in man?
On 2022-08-01, Kenneth Gober wrote: > On Sun, Jul 31, 2022 at 8:35 AM Tobias Fiebig < > tob...@reads-this-mailinglist.com> wrote: > >> > You could periodically rsync it to permanent storage and use mount_mfs' >> > -P option to populate at boot. >> Really good point; I will give setting that up a try later today and add >> that to the >> blogpost. >> > > In addition to a scheduled rsync job, don't forget to also put that same > rsync command > into /etc/rc.shutdown, so that a controlled reboot or shutdown will update > persistent > storage with the most up-to-date data. The scheduled rsync commands will > then just > be there to ensure you don't lose too much in case of system lockup or > other uncontrolled > reboot/powerfail/etc. I wouldn't bother with this, it's only a cache and the next rpki-client run will update it anyway. Speaking from experience the extra delay for reboot is extremely annoying. The data file written by rpki-client and read by bgpd is in a different directory so won't be affected by using MFS for /var/cache/rpki-client. (To make reboots less annoying, it might be worth using noauto and running the mount_mfs -P in the background from rc.local rather than auto-mounting, but it will need some lock mechanism to avoid running rpki-client-related cronjobs until it's done). -- Please keep replies on the mailing list.
Re: rpki-client disk utilization / noting mfs in man?
On Sun, Jul 31, 2022 at 8:35 AM Tobias Fiebig < tob...@reads-this-mailinglist.com> wrote: > > You could periodically rsync it to permanent storage and use mount_mfs' > > -P option to populate at boot. > Really good point; I will give setting that up a try later today and add > that to the > blogpost. > In addition to a scheduled rsync job, don't forget to also put that same rsync command into /etc/rc.shutdown, so that a controlled reboot or shutdown will update persistent storage with the most up-to-date data. The scheduled rsync commands will then just be there to ensure you don't lose too much in case of system lockup or other uncontrolled reboot/powerfail/etc. -ken
Re: rpki-client disk utilization / noting mfs in man?
Heho, > fwiw using a VM for a border router seems a strange choice. Agree. It is called 'doing-stupid-things' for a reason. :-| ;-) 0:-) > Also, having many routers in many networks fetch [...] Yes, and for my own systems I do just that with some added python code around it to make sure what I fetch from the cache makes sense ([1] in case anyone is Interested... still have to fix string handling -.-'); Yet, I cannot really tell people what to do on their own VMs. (Well, I can, but...) ;-) > You could periodically rsync it to permanent storage and use mount_mfs' > -P option to populate at boot. Really good point; I will give setting that up a try later today and add that to the blogpost. > BTW rpki-client is one of the (relatively few) cases where softdep is likely > to give > a significant improvement in performance. Also a good point. I think I will just bench through all the options (softdep, noatime, mfs, [...]) and write that down to have some comparison points. With best regards, Tobias [1] https://git.aperture-labs.org/AS59645/monitoring-tools/src/branch/master/other/update_rpki
Re: rpki-client disk utilization / noting mfs in man?
On 2022-07-31, Tobias Fiebig wrote: > I am running a small setup, where recently the boarder router VMs of a user > caused prolonged and consistent low bandwidth (2-3mb/s) yet high utilization > (many IOPS) disk utilization on the virtualization nodes (more writeup at > [1]). fwiw using a VM for a border router seems a strange choice. Also, having many routers in many networks fetch and validate all these certs, from many origin networks across the world, results in much duplicated work and bandwidth. The RPKI design is that fetch/validation is done by route servers or caches rather than on every individual router. The intention is to use RTR to feed routers but until that is fully handled you could e.g. run a central rpki-client box to generate the prefix list for bgpd and make it available to your own routers over sftp/http/rsync rather than fetching from origins on each router. > I ultimately resorted to giving an mfs on /var/cache/rpki-client a try. This > worked surprisingly well, (naturally) removed all disk i/o usage, and > improved the rpki-client runtime from ~30min to ~16min (CPUs aren't the > freshest, so this is fine, I guess). Of course the trade-off here is a full > sync after every reboot. You could periodically rsync it to permanent storage and use mount_mfs' -P option to populate at boot. BTW rpki-client is one of the (relatively few) cases where softdep is likely to give a significant improvement in performance. -- Please keep replies on the mailing list.
