On Thu, Feb 2, 2017 at 12:58 PM, James Page <james.p...@ubuntu.com> wrote:
> This feels like this generally needs improving, with automatic mounting > of hugepage FS for different pagesize options - so for example: > > /dev/hugepages1G > /def/hugepages > > with appropriate apparmor changes as well for libvirt to allow access to > common paths on Ubuntu systems. > I agree, I faced the same when we faced the need for a basic hugepage allocation service for DPDK and wanted to share a few of my thoughts back then. BTW by starting early the dpdk service so far had no problems, but when increasing and restarting later on that was an issue sometimes. Yet there the kernel option would not help without reboot either. It has a few fallbacks like dropping caches when trying to get more huge pages to mitigate this. But it already says in the config comments that it is a basic helper and admins need to cover more complex cases. FYI: Config: https://gerrit.fd.io/r/gitweb?p=deb_dpdk.git;a=blob;f=debian/dpdk.conf;h=a5aea865a8c43623761f9299549cbd2b25fd06a9;hb=refs/heads/16.11.x Code: https://gerrit.fd.io/r/gitweb?p=deb_dpdk.git;a=blob;f=debian/dpdk-init;h=103488edce6592a5f93e1b32cd5d476416374f81;hb=refs/heads/16.11.x Back then I noted a todo that at some point all applications will need a joint hugepage allocation strategy - just what you suggested here. In addition to your correct statement of some (configurable) default paths there also should be a way to work together in a better way. One program sets 2x1G pages, the next 4x1G and the next 200x2M - what gets to the boot parm? Also since these don't know of each other they often interfer with each other. I can't find my old notes, but at the core in addition to what you suggest it was about establishing a common shared way to be reused and not stomping on each other. In my vision it was like: /etc/hugepaged/config <- core config /etc/hugepaged/config.d/ <- every Package/Admin could drop their need (libvirt, dpdk, databases, java, users adding custom things) The service would then process the aggregate of that - ensure that it has no conflicting specifications and allocates the sum of all those configs. It could auto generate kernel commandline args to be added to make allocation more reliable and still fall back to late allocation as we do today. Note: there is also a package hugepages (2.19-0ubuntu1) which provides: This package contains a number of utilities that will help administrate the use of huge pages on your system. hugeedit modifies binaries to set default segment remapping behavior. hugectl sets environment variables for using huge pages and then execs the target program. hugeadm gives easy access to huge page pool size control. pagesize lists page sizes available on the machine. Back then I thought creating a proper service infrastructure as addon to that project might be a good way to go. Of course the systemd task suggests that this might also be a way to go. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1643675 Title: hugepages for non-default pagesize need manual setup Status in libvirt package in Ubuntu: New Status in systemd package in Ubuntu: New Status in nova-compute package in Juju Charms Collection: Triaged Bug description: This is a amd64 xenial/mitaka deploy using 1607 charms release where I need to have 1G hugepages available for openstack/kvm. nova-compute charm should provide ways to ease e.g. 1G hugepages setup, currently only exposes a "hugepages" setting that ends setting vm.nr_hugepages for the node, but even after settings kernel cmdline parameters for 1G hugepages (hugepagesz=1G hugepages=128), is not usable by libvirt-bin: #1 systemd: Installs /lib/systemd/system/dev-hugepages.mount with default pagesize => no hugetblfs mount with (e.g.) pagesize=1G #2 libvirt-bin: Installs apparmor profile ("libvirt-qemu") which only allows: # for access to hugepages owner "/run/hugepages/kvm/libvirt/qemu/**" rw, owner "/dev/hugepages/libvirt/qemu/**" rw, => not possible to have other mount points for other pagesizes. I've workaround'd #1,#2 above by overriding systemd's by creating a /etc/systemd/system/dev-hugepages.mount with below extra line at [Mount] section: Options=pagesize=1G -> https://paste.ubuntu.com/23513048/ To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1643675/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
