Re: [Openstack-operators] How are folks providing GPU instance types?

[Blair Bethwaite]

Finally circled back to this thread...

Joe - those are great notes!

On 12 May 2016 at 02:51, Joe Topjian  wrote:
> * I found that I didn't have to use EFI-based images. I wonder why that is?

Yeah, we've never run into this as a requirement either.
Peter - can you clarify?

We've had success with Dell R720s and R730s (also about to try C4130s)
passing through various NVIDIA cards: M2070[Q], GRID K1 and K2, K5000,
K40, K80. With the K1s and K2s we define flavors that on individual
GPUs of the board (so a K1 can support 4x GPU instances and a K2 can
support 2x GPU instances - NB: this is not vGPU, which requires a
special hypervisor and only supports windoze guests). Current
hypervisor OS is Ubuntu Trusty, using libvirt+KVM.

Further to the vGPU comment above - apparently the new AMD FirePro
S7150 cards use SRIOV for their multi-gpu support, so I hope may be
more amenable to sharing with wider hypervisor and guest support
(however the only supported hypervisor listed is VMware). Seems like
they might be good for remote visualisation use-cases (opengl and
linux support). Has anyone tried these cards?

A niggling issue we have with our OpenStack GPU infrastructure is how
to deal with special host consumable resources like GPUs in the
scheduler. In practice we have to reserve the whole host only for GPU
instances as otherwise it could be filled with non-GPU instances even
when GPU/s are available (not a good outcome when the GPUs make up
>=50% of the value of the box). But the corollary that follows is that
the GPU hypervisors are often not well utilised, so ideally we'd have
some way to limit the quantity of regular non-GPU instances on those
boxes so that there was always capacity for GPU instances.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How are folks providing GPU instance types?

[Joe Topjian]

Just wanted to add a few notes (I apologize for the brevity):

* The wiki page is indeed the best source of information to get started.
* I found that I didn't have to use EFI-based images. I wonder why that is?
* PCI devices and IDs can be found by running the following on a compute
node:

$ lspci -nn | grep -i nvidia
84:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [GRID
K1] [10de:0ff2] (rev a1)
85:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [GRID
K1] [10de:0ff2] (rev a1)
86:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [GRID
K1] [10de:0ff2] (rev a1)
87:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [GRID
K1] [10de:0ff2] (rev a1)

In which 10de becomes the vendor ID and 0ff2 becomes the product ID.

* My nova.conf looks like this:

pci_alias={"vendor_id":"10de", "product_id":"0ff2", "name":"gpu"}
scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
scheduler_available_filters=nova.scheduler.filters.all_filters
scheduler_available_filters=nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter
scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter

* My /etc/default/grub on the compute node has the following entries:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt
rd.modules-load=vfio-pci"
GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt rd.modules-load=vfio-pci"

* I use the following to create a flavor with access to a single GPU:

nova flavor-create g1.large auto 8192 20 4 --ephemeral 20 --swap 2048
nova flavor-key g1.large set "pci_passthrough:alias"="gpu:1"

For NVIDIA cards in particular, it might take a few attempts to install the
correct driver version, CUDA tools version, etc to get things working
correctly. NVIDIA has a bundle of CUDA examples, one of which is
"/usr/local/cuda-7.5/samples/1_Utilities/deviceQuery". Running this will
confirm if the instance can successfully access the GPU.

Hope this helps!
Joe


On Tue, May 10, 2016 at 8:58 AM, Tomas Vondra  wrote:

> Nordquist, Peter L  writes:
>
> > You will also have to enable iommu on your hypervisors to have libvirt
> expose the capability to Nova for PCI
> > passthrough.  I use Centos 7 and had to set 'iommu=pt intel_iommu=on' for
> my kernel parameters.  Along with
> > this, you'll have to start using EFI for your VMs by installing OVMF on
> your Hypervisors and configuring
> > your images appropriately.  I don't have a link handy for this but the
> gist is that Legacy bootloaders have a
> > much more complicated process to initialize the devices being passed to
> the VM where EFI is much easier.
>
> Hi!
> What I found out the hard way under the Xen hypervisor is that the GPU you
> are passing through must not be the primary GPU of the system. Otherwise,
> you get memory corruption as soon as something appears on the console. If
> not sooner :-). Test if your motherboards are capable of running on the
> integrated VGA even if some other graphics card is connected. Or blacklist
> it for the kernel.
> Tomas
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How are folks providing GPU instance types?

[Tomas Vondra]

Nordquist, Peter L  writes:

> You will also have to enable iommu on your hypervisors to have libvirt
expose the capability to Nova for PCI
> passthrough.  I use Centos 7 and had to set 'iommu=pt intel_iommu=on' for
my kernel parameters.  Along with
> this, you'll have to start using EFI for your VMs by installing OVMF on
your Hypervisors and configuring
> your images appropriately.  I don't have a link handy for this but the
gist is that Legacy bootloaders have a
> much more complicated process to initialize the devices being passed to
the VM where EFI is much easier.

Hi!
What I found out the hard way under the Xen hypervisor is that the GPU you
are passing through must not be the primary GPU of the system. Otherwise,
you get memory corruption as soon as something appears on the console. If
not sooner :-). Test if your motherboards are capable of running on the
integrated VGA even if some other graphics card is connected. Or blacklist
it for the kernel.
Tomas



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How are folks providing GPU instance types?

[Nordquist, Peter L]

This page [0] is not up to date but you can use it for configuration examples 
or at least that's what I've done.  I started this process in Liberty and then 
migrated to Mitaka and while I have successfully passed in a device to a VM 
from Nova, I have not tried to initialize or use that device yet since I don't 
have any EFI images yet.  In Liberty I found that Nova comes with all of the 
functionality already to do pci passthrough given you have your Hypervisor 
configured correctly.  Some of it has changed in Mitaka like needing to set 
type on the alias and including support to boot EFI images but in general it is 
close.  I think the filter is already included in the list of available filters 
so you would just have to add it to your default filter list.  I'm not sure you 
would even have to setup host aggregates, just new flavors that define what 
aliases that flavor is going to allocate.  My assumption has been that 
scheduling other VMs on a GPU node might starve the GPU flavor from bei
 ng able to launch on that node but I have not tried it yet.
 
Here's some example configuration:
pci_alias={"name": "Tesla_K80", "vendor_id": "10de", "product_id": "102d", 
"device_type": "type-PCI"}
pci_passthrough_whitelist={"vendor_id": "10de", "product_id": "102d"}

The API parts of that webpage currently seem to be integrated in the Nova 
codebase but not enabled.  You can use the Nova database itself to check for 
the pci devices in the pci_devices table.

You will also have to enable iommu on your hypervisors to have libvirt expose 
the capability to Nova for PCI passthrough.  I use Centos 7 and had to set 
'iommu=pt intel_iommu=on' for my kernel parameters.  Along with this, you'll 
have to start using EFI for your VMs by installing OVMF on your Hypervisors and 
configuring your images appropriately.  I don't have a link handy for this but 
the gist is that Legacy bootloaders have a much more complicated process to 
initialize the devices being passed to the VM where EFI is much easier.

[0]: https://wiki.openstack.org/wiki/Pci_passthrough 

Peter Nordquist

-Original Message-
From: Jonathan Proulx [mailto:j...@csail.mit.edu] 
Sent: Monday, May 09, 2016 13:13
To: openstack-operators@lists.openstack.org
Subject: [Openstack-operators] How are folks providing GPU instance types?

Hi All,

Having trouble finding any current info on best practices for providing GPU 
instances.  Most of what Google is feeding me is Grizzly or older.

I'm currently on Kilo (Mitaka upgrade planned in 60-90 days) with
Ubuntu14.04 and kvm hypervisor.  Looking to add some NVidia GPUs but haven't 
invested in hardware yet.  Presumably I'd be using host aggregates and instance 
metadata to separate these out from the general pool so not tied to kvm though 
it would be nice to have image compatibility across GPU and nonGPU instance 
types (this is currently 'raw' images in ceph rbd).

Any pointers to good info online or general advice as I travel down this path?  
I suppose particularly urgent is any hardware caveats I need ot be aware of 
before I sink cash into the wrong thing (I'm presuming that the k10,k20,k40,k80 
are all equivalent in this regard).

-Jon

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] How are folks providing GPU instance types?

[Jonathan Proulx]

Hi All,

Having trouble finding any current info on best practices for
providing GPU instances.  Most of what Google is feeding me is Grizzly
or older.

I'm currently on Kilo (Mitaka upgrade planned in 60-90 days) with
Ubuntu14.04 and kvm hypervisor.  Looking to add some NVidia GPUs but
haven't invested in hardware yet.  Presumably I'd be using host
aggregates and instance metadata to separate these out from the
general pool so not tied to kvm though it would be nice to have image
compatibility across GPU and nonGPU instance types (this is currently
'raw' images in ceph rbd).

Any pointers to good info online or general advice as I travel down
this path?  I suppose particularly urgent is any hardware caveats I
need ot be aware of before I sink cash into the wrong thing (I'm
presuming that the k10,k20,k40,k80 are all equivalent in this regard).

-Jon

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators