Re: [libvirt] Problem with setting up KVM guests to use HugePages
With older qemu-1.5.3, I can start the guest by removing the following lines from the xml config file and hugepages are correctly used. I believe these lines were used to create NUMA nodes for the guest. Removing them means no NUMA nodes for the guest I guess. There is one thing I don't quite understand though. In the xml file, I kept Since no NUMA nodes were created, how come this section can pass? Then what is the nodeset here? On Thu, Jun 11, 2015 at 10:27 AM, Clarylin L wrote: > Hi Michal, > > I also tried the other option you mentioned "The other option you have is > to not use guest NUMA nodes, in which case global -mem-path can be used." > by removing from xml > > > > > > > > while keeping older qemu-1.5.3 which does not support element, it > still complained no enough memory when starting guest. > > Did I miss something? > > On Thu, Jun 11, 2015 at 12:00 AM, Michal Privoznik > wrote: > >> [please keep the list CC'ed] >> >> On 10.06.2015 20:09, Clarylin L wrote: >> > Hi Michal, >> > >> > Thanks a lot. >> > >> > If 100 hugepages are pre-allocated, the guest can start without >> decreasing >> > number of hugepages. Since the guest requires 128 hugepages, it's kind >> of >> > expected that the guest would not take memory from hugepages. >> > >> > Before guest start, >> > >> > [root@local ~]# cat /proc/meminfo | grep uge >> > >> > AnonH*uge*Pages: 0 kB >> > >> > H*uge*Pages_Total: 100 >> > >> > H*uge*Pages_Free: 100 >> > >> > H*uge*Pages_Rsvd:0 >> > >> > H*uge*Pages_Surp:0 >> > H*uge*pagesize:1048576 kB >> > >> > After: >> > >> > [root@local ~]# cat /proc/meminfo | grep uge >> > >> > AnonH*uge*Pages: 134254592 kB >> > >> > H*uge*Pages_Total: 100 >> > >> > H*uge*Pages_Free: 100 >> > >> > H*uge*Pages_Rsvd:0 >> > >> > H*uge*Pages_Surp:0 >> > H*uge*pagesize:1048576 kB >> > >> > There is no -mem-prealloc and -mem-path options in qemu command >> >> And there can't be. From the command line below, you are defining 2 NUMA >> nodes for your guest. In order to instruct qemu to back their memory by >> huge pages you need it to support memory-backend-file object which was >> introduced in qemu-2.1.0. >> The other option you have is to not use guest NUMA nodes, in which case >> global -mem-path can be used. >> >> > >> > [root@local ~]# ps -ef | grep qemu >> > >> > *qemu* 3403 1 99 17:42 ?00:36:42 >> /usr/libexec/*qemu*-kvm >> > -name qvpc-di-03-sf -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off >> -cpu >> > host -m 131072 -realtime mlock=off -smp 32,sockets=2,cores=16,threads=1 >> > -numa node,nodeid=0,cpus=0-15,mem=65536 -numa >> > node,nodeid=1,cpus=16-31,mem=65536 -uuid >> > e1b72349-4a0b-4b91-aedc-fd34e92251e4 -smbios type=1,serial=SCALE-SLOT-03 >> > -no-user-config -nodefaults -chardev >> > >> socket,id=charmonitor,path=/var/lib/libvirt/*qemu*/qvpc-di-03-sf.monitor,server,nowait >> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc >> -no-shutdown >> > -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 >> -drive >> > >> file=/var/lib/libvirt/images/asr5700/qvpc-di-03-sf-hda.img,if=none,id=drive-ide0-0-0,format=qcow2 >> > -device >> > ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 >> -drive >> > if=none,id=drive-ide0-1-0,readonly=on,format=raw -device >> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 >> > -chardev pty,id=charserial0 -device >> > isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 >> > -device isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -vga >> > cirrus -device i6300esb,id=watchdog0,bus=pci.0,addr=0x3 -watchdog-action >> > reset -device vfio-pci,host=08:00.0,id=hostdev0,bus=pci.0,addr=0x5 >> -device >> > vfio-pci,host=09:00.0,id=hostdev1,bus=pci.0,addr=0x6 -device >> > vfio-pci,host=0a:00.0,id=hostdev2,bus=pci.0,addr=0x7 -device >> > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on >> > >> > >> > If 140 hugepages are preallocated, the guest cannot start and it >> complained >> > not enough memory. >> > >> > >> > The libvirt version is shown as follows: >> > >> > virsh # version >> > >> > Compiled against library: libvirt 1.2.8 >> > >> > Using library: libvirt 1.2.8 >> > >> > Using API: QEMU 1.2.8 >> > >> > Running hypervisor: QEMU 1.5.3 >> > >> > >> >> >> >> > Also in guest configuration contains numa session. The hugepages are >> > uniformly distributed to two nodes. In this case, do I need to make >> > additional configurations to enable usage of hugepages? >> > >> > >> > >> > >> > >> > >> > >> >> This says that all the memory for you guests should be pinned onto host >> nodes 0-1. If you want to be more specific, you can explicitly wire >> guest NUMA nodes onto host NUMA nodes in 1:N relationship (where N can >> be even 1, in which case you will get 1:1), e.g.: >> >> >> >> Michal >> > > -- libvir-list mai
Re: [libvirt] Problem with setting up KVM guests to use HugePages
Hi Michal, I also tried the other option you mentioned "The other option you have is to not use guest NUMA nodes, in which case global -mem-path can be used." by removing from xml while keeping older qemu-1.5.3 which does not support element, it still complained no enough memory when starting guest. Did I miss something? On Thu, Jun 11, 2015 at 12:00 AM, Michal Privoznik wrote: > [please keep the list CC'ed] > > On 10.06.2015 20:09, Clarylin L wrote: > > Hi Michal, > > > > Thanks a lot. > > > > If 100 hugepages are pre-allocated, the guest can start without > decreasing > > number of hugepages. Since the guest requires 128 hugepages, it's kind of > > expected that the guest would not take memory from hugepages. > > > > Before guest start, > > > > [root@local ~]# cat /proc/meminfo | grep uge > > > > AnonH*uge*Pages: 0 kB > > > > H*uge*Pages_Total: 100 > > > > H*uge*Pages_Free: 100 > > > > H*uge*Pages_Rsvd:0 > > > > H*uge*Pages_Surp:0 > > H*uge*pagesize:1048576 kB > > > > After: > > > > [root@local ~]# cat /proc/meminfo | grep uge > > > > AnonH*uge*Pages: 134254592 kB > > > > H*uge*Pages_Total: 100 > > > > H*uge*Pages_Free: 100 > > > > H*uge*Pages_Rsvd:0 > > > > H*uge*Pages_Surp:0 > > H*uge*pagesize:1048576 kB > > > > There is no -mem-prealloc and -mem-path options in qemu command > > And there can't be. From the command line below, you are defining 2 NUMA > nodes for your guest. In order to instruct qemu to back their memory by > huge pages you need it to support memory-backend-file object which was > introduced in qemu-2.1.0. > The other option you have is to not use guest NUMA nodes, in which case > global -mem-path can be used. > > > > > [root@local ~]# ps -ef | grep qemu > > > > *qemu* 3403 1 99 17:42 ?00:36:42 /usr/libexec/*qemu*-kvm > > -name qvpc-di-03-sf -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off > -cpu > > host -m 131072 -realtime mlock=off -smp 32,sockets=2,cores=16,threads=1 > > -numa node,nodeid=0,cpus=0-15,mem=65536 -numa > > node,nodeid=1,cpus=16-31,mem=65536 -uuid > > e1b72349-4a0b-4b91-aedc-fd34e92251e4 -smbios type=1,serial=SCALE-SLOT-03 > > -no-user-config -nodefaults -chardev > > > socket,id=charmonitor,path=/var/lib/libvirt/*qemu*/qvpc-di-03-sf.monitor,server,nowait > > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc > -no-shutdown > > -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 > -drive > > > file=/var/lib/libvirt/images/asr5700/qvpc-di-03-sf-hda.img,if=none,id=drive-ide0-0-0,format=qcow2 > > -device > > ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 > -drive > > if=none,id=drive-ide0-1-0,readonly=on,format=raw -device > > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 > > -chardev pty,id=charserial0 -device > > isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 > > -device isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -vga > > cirrus -device i6300esb,id=watchdog0,bus=pci.0,addr=0x3 -watchdog-action > > reset -device vfio-pci,host=08:00.0,id=hostdev0,bus=pci.0,addr=0x5 > -device > > vfio-pci,host=09:00.0,id=hostdev1,bus=pci.0,addr=0x6 -device > > vfio-pci,host=0a:00.0,id=hostdev2,bus=pci.0,addr=0x7 -device > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on > > > > > > If 140 hugepages are preallocated, the guest cannot start and it > complained > > not enough memory. > > > > > > The libvirt version is shown as follows: > > > > virsh # version > > > > Compiled against library: libvirt 1.2.8 > > > > Using library: libvirt 1.2.8 > > > > Using API: QEMU 1.2.8 > > > > Running hypervisor: QEMU 1.5.3 > > > > > > > > > Also in guest configuration contains numa session. The hugepages are > > uniformly distributed to two nodes. In this case, do I need to make > > additional configurations to enable usage of hugepages? > > > > > > > > > > > > > > > > This says that all the memory for you guests should be pinned onto host > nodes 0-1. If you want to be more specific, you can explicitly wire > guest NUMA nodes onto host NUMA nodes in 1:N relationship (where N can > be even 1, in which case you will get 1:1), e.g.: > > > > Michal > -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Problem with setting up KVM guests to use HugePages
On Thu, Jun 11, 2015 at 12:00 AM, Michal Privoznik wrote: > [please keep the list CC'ed] > > On 10.06.2015 20:09, Clarylin L wrote: > > Hi Michal, > > > > Thanks a lot. > > > > If 100 hugepages are pre-allocated, the guest can start without > decreasing > > number of hugepages. Since the guest requires 128 hugepages, it's kind of > > expected that the guest would not take memory from hugepages. > > > > Before guest start, > > > > [root@local ~]# cat /proc/meminfo | grep uge > > > > AnonH*uge*Pages: 0 kB > > > > H*uge*Pages_Total: 100 > > > > H*uge*Pages_Free: 100 > > > > H*uge*Pages_Rsvd:0 > > > > H*uge*Pages_Surp:0 > > H*uge*pagesize:1048576 kB > > > > After: > > > > [root@local ~]# cat /proc/meminfo | grep uge > > > > AnonH*uge*Pages: 134254592 kB > > > > H*uge*Pages_Total: 100 > > > > H*uge*Pages_Free: 100 > > > > H*uge*Pages_Rsvd:0 > > > > H*uge*Pages_Surp:0 > > H*uge*pagesize:1048576 kB > > > > There is no -mem-prealloc and -mem-path options in qemu command > > And there can't be. From the command line below, you are defining 2 NUMA > nodes for your guest. In order to instruct qemu to back their memory by > huge pages you need it to support memory-backend-file object which was > introduced in qemu-2.1.0. > The other option you have is to not use guest NUMA nodes, in which case > global -mem-path can be used. > Michal, you were correct. My current version is qemu-1.5.3. After I updated it to 2.1.2, I was able to use element under and the problem was solved. Seemed element was required -- without using it, start guest would report an error. > > > > > [root@local ~]# ps -ef | grep qemu > > > > *qemu* 3403 1 99 17:42 ?00:36:42 /usr/libexec/*qemu*-kvm > > -name qvpc-di-03-sf -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off > -cpu > > host -m 131072 -realtime mlock=off -smp 32,sockets=2,cores=16,threads=1 > > -numa node,nodeid=0,cpus=0-15,mem=65536 -numa > > node,nodeid=1,cpus=16-31,mem=65536 -uuid > > e1b72349-4a0b-4b91-aedc-fd34e92251e4 -smbios type=1,serial=SCALE-SLOT-03 > > -no-user-config -nodefaults -chardev > > > socket,id=charmonitor,path=/var/lib/libvirt/*qemu*/qvpc-di-03-sf.monitor,server,nowait > > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc > -no-shutdown > > -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 > -drive > > > file=/var/lib/libvirt/images/asr5700/qvpc-di-03-sf-hda.img,if=none,id=drive-ide0-0-0,format=qcow2 > > -device > > ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 > -drive > > if=none,id=drive-ide0-1-0,readonly=on,format=raw -device > > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 > > -chardev pty,id=charserial0 -device > > isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 > > -device isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -vga > > cirrus -device i6300esb,id=watchdog0,bus=pci.0,addr=0x3 -watchdog-action > > reset -device vfio-pci,host=08:00.0,id=hostdev0,bus=pci.0,addr=0x5 > -device > > vfio-pci,host=09:00.0,id=hostdev1,bus=pci.0,addr=0x6 -device > > vfio-pci,host=0a:00.0,id=hostdev2,bus=pci.0,addr=0x7 -device > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on > > > > > > If 140 hugepages are preallocated, the guest cannot start and it > complained > > not enough memory. > > > > > > The libvirt version is shown as follows: > > > > virsh # version > > > > Compiled against library: libvirt 1.2.8 > > > > Using library: libvirt 1.2.8 > > > > Using API: QEMU 1.2.8 > > > > Running hypervisor: QEMU 1.5.3 > > > > > > > > > Also in guest configuration contains numa session. The hugepages are > > uniformly distributed to two nodes. In this case, do I need to make > > additional configurations to enable usage of hugepages? > > > > > > > > > > > > > > > > This says that all the memory for you guests should be pinned onto host > nodes 0-1. If you want to be more specific, you can explicitly wire > guest NUMA nodes onto host NUMA nodes in 1:N relationship (where N can > be even 1, in which case you will get 1:1), e.g.: > > > > Michal > -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Problem with setting up KVM guests to use HugePages
On 11.06.2015 10:48, Daniel P. Berrange wrote: > On Thu, Jun 11, 2015 at 10:27:05AM +0200, Michal Privoznik wrote: >> On 11.06.2015 10:13, Daniel P. Berrange wrote: >>> On Wed, Jun 10, 2015 at 09:20:40PM +, Vivi L wrote: Michal Privoznik redhat.com> writes: > > On 10.06.2015 01:05, Vivi L wrote: >> Kashyap Chamarthy redhat.com> writes: >> >> >>> You might want re-test by explicitly setting the 'page' element and >>> 'size' attribute? From my test, I had something like this: >>> >>> $ virsh dumpxml f21-vm | grep hugepages -B3 -A2 >>> 2000896 >>> 200 >>> >>> >>> >>> >>> >>> 8 >>> >>> I haven't tested this exhaustively, but some basic test notes here: >>> >>> >>> https://kashyapc.fedorapeople.org/virt/test-hugepages-with-libvirt.txt >> >> Current QEMU does not support setting element. Could it be the >> cause of my aforementioned problem? >> >> unsupported configuration: huge pages per NUMA node are not supported >> with this QEMU >> > > So this is explanation why the memory for you guest is not backed by > hugepages. I thought setting hugepages per NUMA node is a nice-to-have feature. Is it required to enable the use of hugepages for the guest? >>> >>> No, it should not be mandatory. You should be able to use >>> >>> >>> >>> >>> >>> With pretty much any KVM/QEMU version that exists. If that's >>> broken then its a libvit bug. >> >> Unless hugepages are requested for guest NUMA nodes. In that case >> memory-backend-file object is required. From my investigation, this >> seems to be the case. > > memory-backend-file should only be required if trying to setup > different hugepage configs for each guest NUMA node, or if trying > to pin each guest NUMA node to a different host node. If they just > want hugepages across the whole VM and no pinning shouldn't the > traditional setup work Vivi L, now you see why you should never ever drop the list from CC. He has sent me an additional info with this snippet in the domain: Therefore the memory object is required. We can't guarantee the placement without it. Michal -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Problem with setting up KVM guests to use HugePages
On Thu, Jun 11, 2015 at 10:27:05AM +0200, Michal Privoznik wrote: > On 11.06.2015 10:13, Daniel P. Berrange wrote: > > On Wed, Jun 10, 2015 at 09:20:40PM +, Vivi L wrote: > >> Michal Privoznik redhat.com> writes: > >> > >>> > >>> On 10.06.2015 01:05, Vivi L wrote: > Kashyap Chamarthy redhat.com> writes: > > > > You might want re-test by explicitly setting the 'page' element and > > 'size' attribute? From my test, I had something like this: > > > > $ virsh dumpxml f21-vm | grep hugepages -B3 -A2 > > 2000896 > > 200 > > > > > > > > > > > > 8 > > > > I haven't tested this exhaustively, but some basic test notes here: > > > > > > https://kashyapc.fedorapeople.org/virt/test-hugepages-with-libvirt.txt > > Current QEMU does not support setting element. Could it be the > cause of my aforementioned problem? > > unsupported configuration: huge pages per NUMA node are not supported > with this QEMU > > >>> > >>> So this is explanation why the memory for you guest is not backed by > >>> hugepages. > >> > >> I thought setting hugepages per NUMA node is a nice-to-have feature. > >> Is it required to enable the use of hugepages for the guest? > > > > No, it should not be mandatory. You should be able to use > > > > > > > > > > > > With pretty much any KVM/QEMU version that exists. If that's > > broken then its a libvit bug. > > Unless hugepages are requested for guest NUMA nodes. In that case > memory-backend-file object is required. From my investigation, this > seems to be the case. memory-backend-file should only be required if trying to setup different hugepage configs for each guest NUMA node, or if trying to pin each guest NUMA node to a different host node. If they just want hugepages across the whole VM and no pinning shouldn't the traditional setup work Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Problem with setting up KVM guests to use HugePages
On 11.06.2015 10:13, Daniel P. Berrange wrote: > On Wed, Jun 10, 2015 at 09:20:40PM +, Vivi L wrote: >> Michal Privoznik redhat.com> writes: >> >>> >>> On 10.06.2015 01:05, Vivi L wrote: Kashyap Chamarthy redhat.com> writes: > You might want re-test by explicitly setting the 'page' element and > 'size' attribute? From my test, I had something like this: > > $ virsh dumpxml f21-vm | grep hugepages -B3 -A2 > 2000896 > 200 > > > > > > 8 > > I haven't tested this exhaustively, but some basic test notes here: > > https://kashyapc.fedorapeople.org/virt/test-hugepages-with-libvirt.txt Current QEMU does not support setting element. Could it be the cause of my aforementioned problem? unsupported configuration: huge pages per NUMA node are not supported with this QEMU >>> >>> So this is explanation why the memory for you guest is not backed by >>> hugepages. >> >> I thought setting hugepages per NUMA node is a nice-to-have feature. >> Is it required to enable the use of hugepages for the guest? > > No, it should not be mandatory. You should be able to use > > > > > > With pretty much any KVM/QEMU version that exists. If that's > broken then its a libvit bug. Unless hugepages are requested for guest NUMA nodes. In that case memory-backend-file object is required. From my investigation, this seems to be the case. Michal -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Problem with setting up KVM guests to use HugePages
On Wed, Jun 10, 2015 at 09:20:40PM +, Vivi L wrote: > Michal Privoznik redhat.com> writes: > > > > > On 10.06.2015 01:05, Vivi L wrote: > > > Kashyap Chamarthy redhat.com> writes: > > > > > > > > >> You might want re-test by explicitly setting the 'page' element and > > >> 'size' attribute? From my test, I had something like this: > > >> > > >> $ virsh dumpxml f21-vm | grep hugepages -B3 -A2 > > >> 2000896 > > >> 200 > > >> > > >> > > >> > > >> > > >> > > >> 8 > > >> > > >> I haven't tested this exhaustively, but some basic test notes here: > > >> > > >> > > >> https://kashyapc.fedorapeople.org/virt/test-hugepages-with-libvirt.txt > > > > > > Current QEMU does not support setting element. Could it be the > > > cause of my aforementioned problem? > > > > > > unsupported configuration: huge pages per NUMA node are not supported > > > with this QEMU > > > > > > > So this is explanation why the memory for you guest is not backed by > > hugepages. > > I thought setting hugepages per NUMA node is a nice-to-have feature. > Is it required to enable the use of hugepages for the guest? No, it should not be mandatory. You should be able to use With pretty much any KVM/QEMU version that exists. If that's broken then its a libvit bug. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Problem with setting up KVM guests to use HugePages
[please keep the list CC'ed] On 10.06.2015 20:09, Clarylin L wrote: > Hi Michal, > > Thanks a lot. > > If 100 hugepages are pre-allocated, the guest can start without decreasing > number of hugepages. Since the guest requires 128 hugepages, it's kind of > expected that the guest would not take memory from hugepages. > > Before guest start, > > [root@local ~]# cat /proc/meminfo | grep uge > > AnonH*uge*Pages: 0 kB > > H*uge*Pages_Total: 100 > > H*uge*Pages_Free: 100 > > H*uge*Pages_Rsvd:0 > > H*uge*Pages_Surp:0 > H*uge*pagesize:1048576 kB > > After: > > [root@local ~]# cat /proc/meminfo | grep uge > > AnonH*uge*Pages: 134254592 kB > > H*uge*Pages_Total: 100 > > H*uge*Pages_Free: 100 > > H*uge*Pages_Rsvd:0 > > H*uge*Pages_Surp:0 > H*uge*pagesize:1048576 kB > > There is no -mem-prealloc and -mem-path options in qemu command And there can't be. From the command line below, you are defining 2 NUMA nodes for your guest. In order to instruct qemu to back their memory by huge pages you need it to support memory-backend-file object which was introduced in qemu-2.1.0. The other option you have is to not use guest NUMA nodes, in which case global -mem-path can be used. > > [root@local ~]# ps -ef | grep qemu > > *qemu* 3403 1 99 17:42 ?00:36:42 /usr/libexec/*qemu*-kvm > -name qvpc-di-03-sf -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu > host -m 131072 -realtime mlock=off -smp 32,sockets=2,cores=16,threads=1 > -numa node,nodeid=0,cpus=0-15,mem=65536 -numa > node,nodeid=1,cpus=16-31,mem=65536 -uuid > e1b72349-4a0b-4b91-aedc-fd34e92251e4 -smbios type=1,serial=SCALE-SLOT-03 > -no-user-config -nodefaults -chardev > socket,id=charmonitor,path=/var/lib/libvirt/*qemu*/qvpc-di-03-sf.monitor,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown > -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive > file=/var/lib/libvirt/images/asr5700/qvpc-di-03-sf-hda.img,if=none,id=drive-ide0-0-0,format=qcow2 > -device > ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 -drive > if=none,id=drive-ide0-1-0,readonly=on,format=raw -device > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 > -chardev pty,id=charserial0 -device > isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 > -device isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -vga > cirrus -device i6300esb,id=watchdog0,bus=pci.0,addr=0x3 -watchdog-action > reset -device vfio-pci,host=08:00.0,id=hostdev0,bus=pci.0,addr=0x5 -device > vfio-pci,host=09:00.0,id=hostdev1,bus=pci.0,addr=0x6 -device > vfio-pci,host=0a:00.0,id=hostdev2,bus=pci.0,addr=0x7 -device > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on > > > If 140 hugepages are preallocated, the guest cannot start and it complained > not enough memory. > > > The libvirt version is shown as follows: > > virsh # version > > Compiled against library: libvirt 1.2.8 > > Using library: libvirt 1.2.8 > > Using API: QEMU 1.2.8 > > Running hypervisor: QEMU 1.5.3 > > > Also in guest configuration contains numa session. The hugepages are > uniformly distributed to two nodes. In this case, do I need to make > additional configurations to enable usage of hugepages? > > > > > > > This says that all the memory for you guests should be pinned onto host nodes 0-1. If you want to be more specific, you can explicitly wire guest NUMA nodes onto host NUMA nodes in 1:N relationship (where N can be even 1, in which case you will get 1:1), e.g.: Michal -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Problem with setting up KVM guests to use HugePages
Michal Privoznik redhat.com> writes: > > On 10.06.2015 01:05, Vivi L wrote: > > Kashyap Chamarthy redhat.com> writes: > > > > > >> You might want re-test by explicitly setting the 'page' element and > >> 'size' attribute? From my test, I had something like this: > >> > >> $ virsh dumpxml f21-vm | grep hugepages -B3 -A2 > >> 2000896 > >> 200 > >> > >> > >> > >> > >> > >> 8 > >> > >> I haven't tested this exhaustively, but some basic test notes here: > >> > >> https://kashyapc.fedorapeople.org/virt/test-hugepages-with-libvirt.txt > > > > Current QEMU does not support setting element. Could it be the > > cause of my aforementioned problem? > > > > unsupported configuration: huge pages per NUMA node are not supported > > with this QEMU > > > > So this is explanation why the memory for you guest is not backed by > hugepages. I thought setting hugepages per NUMA node is a nice-to-have feature. Is it required to enable the use of hugepages for the guest? > Can you please share the qemu command line that libvirt generated for > your guest? If setting 100 hugepages, the guest can start while not using hugepages at all. In this case, [root@local ~]# ps -ef | grep qemu qemu 3403 1 99 17:42 ?00:36:42 /usr/libexec/qemu-kvm -name qvpc-di-03-sf -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu host -m 131072 -realtime mlock=off -smp 32,sockets=2,cores=16,threads=1 -numa node,nodeid=0,cpus=0-15,mem=65536 -numa node,nodeid=1, cpus=16-31,mem=65536 -uuid e1b72349-4a0b-4b91-aedc-fd34e92251e4 - smbios type=1,serial=SCALE-SLOT-03 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/qvpc-di-03-sf.monitor,server, nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0, addr=0x1.0x2 -drive file=/var/lib/libvirt/images/asr5700/qvpc-di-03-sf- hda.img,if=none,id=drive-ide0-0-0,format=qcow2 -device ide- hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide- cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -vga cirrus -device i6300esb,id=watchdog0,bus=pci.0, addr=0x3 -watchdog-action reset -device vfio-pci,host=08:00.0,id=hostdev0,bus=pci.0,addr=0x5 -device vfio- pci,host=09:00.0,id=hostdev1,bus=pci.0,addr=0x6 -device vfio- pci,host=0a:00.0,id=hostdev2,bus=pci.0,addr=0x7 -device virtio-balloon- pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on > Michal > > -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Problem with setting up KVM guests to use HugePages
On 10.06.2015 01:05, Vivi L wrote: > Kashyap Chamarthy redhat.com> writes: > > >> You might want re-test by explicitly setting the 'page' element and >> 'size' attribute? From my test, I had something like this: >> >> $ virsh dumpxml f21-vm | grep hugepages -B3 -A2 >> 2000896 >> 200 >> >> >> >> >> >> 8 >> >> I haven't tested this exhaustively, but some basic test notes here: >> >> https://kashyapc.fedorapeople.org/virt/test-hugepages-with-libvirt.txt > > Current QEMU does not support setting element. Could it be the > cause of my aforementioned problem? > > unsupported configuration: huge pages per NUMA node are not supported > with this QEMU > So this is explanation why the memory for you guest is not backed by hugepages. Can you please share the qemu command line that libvirt generated for your guest? Michal -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Problem with setting up KVM guests to use HugePages
Kashyap Chamarthy redhat.com> writes: > You might want re-test by explicitly setting the 'page' element and > 'size' attribute? From my test, I had something like this: > > $ virsh dumpxml f21-vm | grep hugepages -B3 -A2 > 2000896 > 200 > > > > > > 8 > > I haven't tested this exhaustively, but some basic test notes here: > > https://kashyapc.fedorapeople.org/virt/test-hugepages-with-libvirt.txt Current QEMU does not support setting element. Could it be the cause of my aforementioned problem? unsupported configuration: huge pages per NUMA node are not supported with this QEMU -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Problem with setting up KVM guests to use HugePages
On Tue, Jun 09, 2015 at 05:52:04AM +, Vivi L wrote: > Configure hugepages and then start virtual guest via "virsh start". However, > virtual guest failed to use hugepages although it's configured > > The initial usage of system memory > [root@local ~]# free > totalusedfree shared buff/cache available > Mem: 263767352 1900628 261177892 9344688832 261431872 > Swap: 4194300 0 4194300 > > After configuring hugepages, > [root@local ~]# cat /proc/meminfo | grep uge > AnonHugePages: 2048 kB > HugePages_Total: 140 > HugePages_Free: 140 > HugePages_Rsvd:0 > HugePages_Surp:0 > Hugepagesize:1048576 kB > > [root@local ~]# mount | grep huge > cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec, > relatime,hugetlb)hugetlbfs on /dev/hugepages type hugetlbfs (rw,realtime) > > On the guest side, its config xml contains > > test-01 > e1b72349-4a0b-4b91-aedc-fd34e92251e4 > test-01 > 134217728 > 134217728 > > You might want re-test by explicitly setting the 'page' element and 'size' attribute? From my test, I had something like this: $ virsh dumpxml f21-vm | grep hugepages -B3 -A2 2000896 200 8 I haven't tested this exhaustively, but some basic test notes here: https://kashyapc.fedorapeople.org/virt/test-hugepages-with-libvirt.txt > > > > > Tried to virsh start the guest, but it failed although the hugepages memory > was sufficient > virsh # start test-01 > error: Failed to start domain test-01 > error: internal error: early end of file from monitor: possible problem: > Cannot set up guest memory 'pc.ram': Cannot allocate memory > > > however, if decrease the number of hugepages to smaller(say 100) that > makes the memory not used by hugepages more than required by guest, > then guest can start. But memory is not allocated from hugepages. > [root@local ~]# cat /proc/meminfo | grep uge > AnonHugePages: 134254592 kB > HugePages_Total: 100 > HugePages_Free: 100 > HugePages_Rsvd:0 > HugePages_Surp:0 > Hugepagesize:1048576 kB > > In summary, although hugepages is configured, the guest seemed not > instructed to use hugepage, instead, the guest is allocated memory from > the memory pool leftover by hugepages. > > > > > -- > libvir-list mailing list > libvir-list@redhat.com > https://www.redhat.com/mailman/listinfo/libvir-list -- /kashyap -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
[libvirt] Problem with setting up KVM guests to use HugePages
Configure hugepages and then start virtual guest via "virsh start". However, virtual guest failed to use hugepages although it's configured The initial usage of system memory [root@local ~]# free totalusedfree shared buff/cache available Mem: 263767352 1900628 261177892 9344688832 261431872 Swap: 4194300 0 4194300 After configuring hugepages, [root@local ~]# cat /proc/meminfo | grep uge AnonHugePages: 2048 kB HugePages_Total: 140 HugePages_Free: 140 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize:1048576 kB [root@local ~]# mount | grep huge cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec, relatime,hugetlb)hugetlbfs on /dev/hugepages type hugetlbfs (rw,realtime) On the guest side, its config xml contains test-01 e1b72349-4a0b-4b91-aedc-fd34e92251e4 test-01 134217728 134217728 Tried to virsh start the guest, but it failed although the hugepages memory was sufficient virsh # start test-01 error: Failed to start domain test-01 error: internal error: early end of file from monitor: possible problem: Cannot set up guest memory 'pc.ram': Cannot allocate memory however, if decrease the number of hugepages to smaller(say 100) that makes the memory not used by hugepages more than required by guest, then guest can start. But memory is not allocated from hugepages. [root@local ~]# cat /proc/meminfo | grep uge AnonHugePages: 134254592 kB HugePages_Total: 100 HugePages_Free: 100 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize:1048576 kB In summary, although hugepages is configured, the guest seemed not instructed to use hugepage, instead, the guest is allocated memory from the memory pool leftover by hugepages. -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
