I'd like to better understand how hugepages are allocated at boot when vpp is not started, as well as what happens what vpp is started (ie, systemctl start vpp).
The reason I ask is that I'm running into an issue with hugepage allocation changes causing VPP to fail. Whereas 1024 2MB pages is the default, I find that as I run more vhost-user VMs, I need to back them with hugepages, and need more pages. When doing this and then starting/restarting VPP, I am seeing failures. Any tips? Has anyone else seen this? Thanks, Eric In more detail, Scenario #1: 1. Assuming VPP doesn't start by default (I had run systemctl disable vpp on a prior boot) 2. Boot system under test 3. Verify via /proc/meminfo that 1024 huge pages have been allocated and 1024 are free 4. Start VPP (systemctl start vpp), see that free pool goes down to 768, as expected and VPP runs without issue Scenario #2: 1. Assuming VPP doesn't start by default (I had run systemctl disable vpp on a prior boot) 2. Boot system under test 3. Adjust number of hugepages from 1024 to 8192 (sysctl -w vm.nr_hugepages=8192) 4. Verify via /proc/meminfo that 8192 huge pages have been allocated and are free 5. Start VPP (systemctl start vpp), see that VPP fails to start with log shown below at [1], but in summary an EAL failure to allocate memory Scenario #3: 1. Same as scenario #1 to start. After VPP is up and running,.... 2. Adjust number of huge pages from 1024 -> 8192, noting that the number of pages moves to 8192, of which some are still being used by VPP 3. You can add more vhost-user interfaces without issue, and VPP and VMs are functional. 4. Restart VPP (systemctl restart vpp) 5. Note that now VPP fails to start, though there are still _many_ pages free. [1] Snippet of Error log: Apr 13 13:43:41 eernstworkstation systemd[1]: Starting vector packet processing engine... Apr 13 13:43:41 eernstworkstation systemd[1]: Started vector packet processing engine. Apr 13 13:43:41 eernstworkstation vpp[5024]: vlib_plugin_early_init:213: plugin path /usr/lib/vpp_plugins Apr 13 13:43:41 eernstworkstation vpp[5024]: /usr/bin/vpp[5024]: vlib_pci_bind_to_uio: Skipping PCI device 0000:05:00.0 as host interface eth0 is up Apr 13 13:43:41 eernstworkstation /usr/bin/vpp[5024]: vlib_pci_bind_to_uio: Skipping PCI device 0000:05:00.0 as host interface eth0 is up Apr 13 13:43:41 eernstworkstation vpp[5024]: /usr/bin/vpp[5024]: vlib_pci_bind_to_uio: Skipping PCI device 0000:05:00.1 as host interface eth1 is up Apr 13 13:43:41 eernstworkstation /usr/bin/vpp[5024]: vlib_pci_bind_to_uio: Skipping PCI device 0000:05:00.1 as host interface eth1 is up Apr 13 13:43:41 eernstworkstation vpp[5024]: EAL: Detected 32 lcore(s) Apr 13 13:43:41 eernstworkstation vpp[5024]: EAL: No free hugepages reported in hugepages-1048576kB Apr 13 13:43:41 eernstworkstation vpp[5024]: EAL: Probing VFIO support... Apr 13 13:43:41 eernstworkstation vnet[5024]: EAL: Probing VFIO support... Apr 13 13:43:41 eernstworkstation sudo[5038]: eernst : TTY=pts/1 ; PWD=/home/eernst ; USER=root ; COMMAND=/bin/journalctl Apr 13 13:43:41 eernstworkstation sudo[5038]: pam_unix(sudo:session): session opened for user root by eernst(uid=0) Apr 13 13:43:42 eernstworkstation vpp[5024]: EAL: Cannot get a virtual area: Cannot allocate memory Apr 13 13:43:42 eernstworkstation vpp[5024]: EAL: Failed to remap 2 MB pages Apr 13 13:43:42 eernstworkstation vpp[5024]: PANIC in rte_eal_init(): Apr 13 13:43:42 eernstworkstation vpp[5024]: Cannot init memory Apr 13 13:43:42 eernstworkstation vnet[5024]: EAL: Cannot get a virtual area: Cannot allocate memory Apr 13 13:43:42 eernstworkstation vnet[5024]: EAL: Failed to remap 2 MB pages Apr 13 13:43:42 eernstworkstation vnet[5024]: PANIC in rte_eal_init(): Apr 13 13:43:42 eernstworkstation vnet[5024]: Cannot init memory Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Main process exited, code=dumped, status=6/ABRT Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Unit entered failed state. Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Failed with result 'core-dump'. Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Service hold-off time over, scheduling restart. Apr 13 13:43:43 eernstworkstation systemd[1]: Stopped vector packet processing engine. Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Start request repeated too quickly. Apr 13 13:43:43 eernstworkstation systemd[1]: Failed to start vector packet processing engine.
_______________________________________________ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
