Re: Unable to create more than 1 guest virtio-net device using vhost-net backend
On Mon, 2010-03-22 at 20:16 +0200, Michael S. Tsirkin wrote: > On Sun, Mar 21, 2010 at 01:58:29PM +0200, Avi Kivity wrote: > > On 03/21/2010 01:34 PM, Michael S. Tsirkin wrote: > >> On Sun, Mar 21, 2010 at 12:29:31PM +0200, Avi Kivity wrote: > >> > >>> On 03/21/2010 12:15 PM, Michael S. Tsirkin wrote: > >>> > >> Nothing easy that I can see. Each device needs 2 of these. Avi, Gleb, > >> any objections to increasing the limit to say 16? That would give us > >> 5 more devices to the limit of 6 per guest. > >> > >> > >> > > Increase it to 200, then. > > > > > OK. I think we'll also need a smarter allocator > than bus->dev_count++ than we now have. Right? > > > >>> No, why? > >>> > >> We'll run into problems if devices are created/removed in random order, > >> won't we? > >> > > > > unregister_dev() takes care of it. > > > >>> Eventually we'll want faster scanning than the linear search we employ > >>> now, though. > >>> > >> Yes I suspect with 200 entries we will :). Let's just make it 16 for > >> now? > >> > > > > Let's make it 200 and fix the performance problems later. Making it 16 > > is just asking for trouble. > > I did this and performance with vhost seems to become much more noisy, > and drop by about 10% on average, even though in practice only > a single device is created. Still trying to figure it out ... > Any idea? I am not sure if this 10% variation is due to the increase of NR_IO_BUS_DEVS. In our testing, we do see variations of 10% or higher between multiple netperf instances with the same setup/configuration when using virtio/vhost. Thanks Sridhar -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Unable to create more than 1 guest virtio-net device using vhost-net backend
On Sun, Mar 21, 2010 at 01:58:29PM +0200, Avi Kivity wrote: > On 03/21/2010 01:34 PM, Michael S. Tsirkin wrote: >> On Sun, Mar 21, 2010 at 12:29:31PM +0200, Avi Kivity wrote: >> >>> On 03/21/2010 12:15 PM, Michael S. Tsirkin wrote: >>> >> Nothing easy that I can see. Each device needs 2 of these. Avi, Gleb, >> any objections to increasing the limit to say 16? That would give us >> 5 more devices to the limit of 6 per guest. >> >> >> > Increase it to 200, then. > > OK. I think we'll also need a smarter allocator than bus->dev_count++ than we now have. Right? >>> No, why? >>> >> We'll run into problems if devices are created/removed in random order, >> won't we? >> > > unregister_dev() takes care of it. > >>> Eventually we'll want faster scanning than the linear search we employ >>> now, though. >>> >> Yes I suspect with 200 entries we will :). Let's just make it 16 for >> now? >> > > Let's make it 200 and fix the performance problems later. Making it 16 > is just asking for trouble. I did this and performance with vhost seems to become much more noisy, and drop by about 10% on average, even though in practice only a single device is created. Still trying to figure it out ... Any idea? > -- > error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Unable to create more than 1 guest virtio-net device using vhost-net backend
On 03/21/2010 01:34 PM, Michael S. Tsirkin wrote: On Sun, Mar 21, 2010 at 12:29:31PM +0200, Avi Kivity wrote: On 03/21/2010 12:15 PM, Michael S. Tsirkin wrote: Nothing easy that I can see. Each device needs 2 of these. Avi, Gleb, any objections to increasing the limit to say 16? That would give us 5 more devices to the limit of 6 per guest. Increase it to 200, then. OK. I think we'll also need a smarter allocator than bus->dev_count++ than we now have. Right? No, why? We'll run into problems if devices are created/removed in random order, won't we? unregister_dev() takes care of it. Eventually we'll want faster scanning than the linear search we employ now, though. Yes I suspect with 200 entries we will :). Let's just make it 16 for now? Let's make it 200 and fix the performance problems later. Making it 16 is just asking for trouble. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Unable to create more than 1 guest virtio-net device using vhost-net backend
On Sun, Mar 21, 2010 at 12:29:31PM +0200, Avi Kivity wrote: > On 03/21/2010 12:15 PM, Michael S. Tsirkin wrote: Nothing easy that I can see. Each device needs 2 of these. Avi, Gleb, any objections to increasing the limit to say 16? That would give us 5 more devices to the limit of 6 per guest. >>> Increase it to 200, then. >>> >> OK. I think we'll also need a smarter allocator >> than bus->dev_count++ than we now have. Right? >> > > No, why? We'll run into problems if devices are created/removed in random order, won't we? > Eventually we'll want faster scanning than the linear search we employ > now, though. Yes I suspect with 200 entries we will :). Let's just make it 16 for now? >>> Is the limit visible to userspace? If not, we need to expose it. >>> >> I don't think it's visible: it seems to be used in a single >> place in kvm. Let's add an ioctl? Note that qemu doesn't >> need it now ... >> > > We usually expose limits via KVM_CHECK_EXTENSION(KVM_CAP_BLAH). We can > expose it via KVM_CAP_IOEVENTFD (and need to reserve iodev entries for > those). > > -- > error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Unable to create more than 1 guest virtio-net device using vhost-net backend
On 03/21/2010 12:21 PM, Gleb Natapov wrote: On Sun, Mar 21, 2010 at 12:11:33PM +0200, Avi Kivity wrote: On 03/21/2010 11:55 AM, Michael S. Tsirkin wrote: On Fri, Mar 19, 2010 at 03:19:27PM -0700, Sridhar Samudrala wrote: When creating a guest with 2 virtio-net interfaces, i am running into a issue causing the 2nd i/f falling back to userpace virtio even when vhost is enabled. After some debugging, it turned out that KVM_IOEVENTFD ioctl() call in qemu is failing with ENOSPC. This is because of the NR_IOBUS_DEVS(6) limit in kvm_io_bus_register_dev() routine in the host kernel. I think we need to increase this limit if we want to support multiple network interfaces using vhost-net. Is there an alternate solution? Thanks Sridhar Nothing easy that I can see. Each device needs 2 of these. Avi, Gleb, any objections to increasing the limit to say 16? That would give us 5 more devices to the limit of 6 per guest. Increase it to 200, then. Currently on each device read/write we iterate over all registered devices. This is not scalable. Yeah. We need first to drop the callback based matching and replace it with explicit ranges, then to replace the search with a hash table for small ranges (keeping a linear search for large ranges, can happen for coalesced mmio). -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Unable to create more than 1 guest virtio-net device using vhost-net backend
On 03/21/2010 12:15 PM, Michael S. Tsirkin wrote: Nothing easy that I can see. Each device needs 2 of these. Avi, Gleb, any objections to increasing the limit to say 16? That would give us 5 more devices to the limit of 6 per guest. Increase it to 200, then. OK. I think we'll also need a smarter allocator than bus->dev_count++ than we now have. Right? No, why? Eventually we'll want faster scanning than the linear search we employ now, though. Is the limit visible to userspace? If not, we need to expose it. I don't think it's visible: it seems to be used in a single place in kvm. Let's add an ioctl? Note that qemu doesn't need it now ... We usually expose limits via KVM_CHECK_EXTENSION(KVM_CAP_BLAH). We can expose it via KVM_CAP_IOEVENTFD (and need to reserve iodev entries for those). -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Unable to create more than 1 guest virtio-net device using vhost-net backend
On Sun, Mar 21, 2010 at 12:11:33PM +0200, Avi Kivity wrote: > On 03/21/2010 11:55 AM, Michael S. Tsirkin wrote: > >On Fri, Mar 19, 2010 at 03:19:27PM -0700, Sridhar Samudrala wrote: > >>When creating a guest with 2 virtio-net interfaces, i am running > >>into a issue causing the 2nd i/f falling back to userpace virtio > >>even when vhost is enabled. > >> > >>After some debugging, it turned out that KVM_IOEVENTFD ioctl() > >>call in qemu is failing with ENOSPC. > >>This is because of the NR_IOBUS_DEVS(6) limit in kvm_io_bus_register_dev() > >>routine in the host kernel. > >> > >>I think we need to increase this limit if we want to support multiple > >>network interfaces using vhost-net. > >>Is there an alternate solution? > >> > >>Thanks > >>Sridhar > >Nothing easy that I can see. Each device needs 2 of these. Avi, Gleb, > >any objections to increasing the limit to say 16? That would give us > >5 more devices to the limit of 6 per guest. > > Increase it to 200, then. > Currently on each device read/write we iterate over all registered devices. This is not scalable. > Is the limit visible to userspace? If not, we need to expose it. > > -- > error compiling committee.c: too many arguments to function -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Unable to create more than 1 guest virtio-net device using vhost-net backend
On Sun, Mar 21, 2010 at 12:11:33PM +0200, Avi Kivity wrote: > On 03/21/2010 11:55 AM, Michael S. Tsirkin wrote: >> On Fri, Mar 19, 2010 at 03:19:27PM -0700, Sridhar Samudrala wrote: >> >>> When creating a guest with 2 virtio-net interfaces, i am running >>> into a issue causing the 2nd i/f falling back to userpace virtio >>> even when vhost is enabled. >>> >>> After some debugging, it turned out that KVM_IOEVENTFD ioctl() >>> call in qemu is failing with ENOSPC. >>> This is because of the NR_IOBUS_DEVS(6) limit in kvm_io_bus_register_dev() >>> routine in the host kernel. >>> >>> I think we need to increase this limit if we want to support multiple >>> network interfaces using vhost-net. >>> Is there an alternate solution? >>> >>> Thanks >>> Sridhar >>> >> Nothing easy that I can see. Each device needs 2 of these. Avi, Gleb, >> any objections to increasing the limit to say 16? That would give us >> 5 more devices to the limit of 6 per guest. >> > > Increase it to 200, then. OK. I think we'll also need a smarter allocator than bus->dev_count++ than we now have. Right? > Is the limit visible to userspace? If not, we need to expose it. I don't think it's visible: it seems to be used in a single place in kvm. Let's add an ioctl? Note that qemu doesn't need it now ... > -- > error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Unable to create more than 1 guest virtio-net device using vhost-net backend
On 03/21/2010 11:55 AM, Michael S. Tsirkin wrote: On Fri, Mar 19, 2010 at 03:19:27PM -0700, Sridhar Samudrala wrote: When creating a guest with 2 virtio-net interfaces, i am running into a issue causing the 2nd i/f falling back to userpace virtio even when vhost is enabled. After some debugging, it turned out that KVM_IOEVENTFD ioctl() call in qemu is failing with ENOSPC. This is because of the NR_IOBUS_DEVS(6) limit in kvm_io_bus_register_dev() routine in the host kernel. I think we need to increase this limit if we want to support multiple network interfaces using vhost-net. Is there an alternate solution? Thanks Sridhar Nothing easy that I can see. Each device needs 2 of these. Avi, Gleb, any objections to increasing the limit to say 16? That would give us 5 more devices to the limit of 6 per guest. Increase it to 200, then. Is the limit visible to userspace? If not, we need to expose it. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Unable to create more than 1 guest virtio-net device using vhost-net backend
On Fri, Mar 19, 2010 at 03:19:27PM -0700, Sridhar Samudrala wrote: > When creating a guest with 2 virtio-net interfaces, i am running > into a issue causing the 2nd i/f falling back to userpace virtio > even when vhost is enabled. > > After some debugging, it turned out that KVM_IOEVENTFD ioctl() > call in qemu is failing with ENOSPC. > This is because of the NR_IOBUS_DEVS(6) limit in kvm_io_bus_register_dev() > routine in the host kernel. > > I think we need to increase this limit if we want to support multiple > network interfaces using vhost-net. > Is there an alternate solution? > > Thanks > Sridhar Nothing easy that I can see. Each device needs 2 of these. Avi, Gleb, any objections to increasing the limit to say 16? That would give us 5 more devices to the limit of 6 per guest. -- MST -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
