Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu
Hi, AFAIU this works only when the guest allocates a continuous range of physical pages. This is a large requirement from the guest, which I'd like to drop. Is it? The world is moving to huge pages, with all the stuff needed for it like moving around userspace pages to compact memory and make huge page allocation easier. I think these days it is alot easier to allocate 2M of continuous physical memory than it used to be a few years ago. At least on linux, dunno about windows. When allocating stuff at boot time (say qxl kms driver) allocating even larger chunks shouldn't be a big issue. And having a single big guest memory chunk, then register that as qxl memory slot is what works best with the existing interfaces I guess. Another option we can think about is a 64bit PCI bar for the surfaces which can be moved out of the low 4G. So I would like to have the guest use a regular allocator, generating for instance two sequential pages in virtual memory that are scattered in physical memory. Those two physical guest page addresses (gp1 and gp2) correspond to two host virtual memory addresses (hv1, hv2). I would now like to provide to spice-server a single virtual address p that maps to those two pages in sequence. Playing mapping tricks like this doesn't come for free. When doing it this way we probaby still want to register a big chunk of memory as qxl memory slot so we have the mapping cost only once, not for each and every surface we create and destroy. cheers, Gerd
Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu
On Tue, Oct 11, 2011 at 01:28:12PM +0200, Gerd Hoffmann wrote: Hi, AFAIU this works only when the guest allocates a continuous range of physical pages. This is a large requirement from the guest, which I'd like to drop. Is it? The world is moving to huge pages, with all the stuff needed for it like moving around userspace pages to compact memory and make huge page allocation easier. I think these days it is alot easier to allocate 2M of continuous physical memory than it used to be a few years ago. At least on linux, dunno about windows. When allocating stuff at boot time (say qxl kms driver) allocating even larger chunks shouldn't be a big issue. And having a single big guest memory chunk, then register that as qxl memory slot is what works best with the existing interfaces I guess. Right, this would work. I was trying to avoid claiming a large chunk up front. I was also trying to avoid having our own allocator, although I think that's not really a problem (can be replaced with an in kernel allocator probably). Another option we can think about is a 64bit PCI bar for the surfaces which can be moved out of the low 4G. I heard this suggested by Avi, so this would allow us to allocate a large chunk without requiring any memory hole? So I would like to have the guest use a regular allocator, generating for instance two sequential pages in virtual memory that are scattered in physical memory. Those two physical guest page addresses (gp1 and gp2) correspond to two host virtual memory addresses (hv1, hv2). I would now like to provide to spice-server a single virtual address p that maps to those two pages in sequence. Playing mapping tricks like this doesn't come for free. When doing it this way we probaby still want to register a big chunk of memory as qxl memory slot so we have the mapping cost only once, not for each and every surface we create and destroy. cheers, Gerd ___ Spice-devel mailing list spice-de...@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/spice-devel
Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu
Hi, Another option we can think about is a 64bit PCI bar for the surfaces which can be moved out of the low 4G. I heard this suggested by Avi, so this would allow us to allocate a large chunk without requiring any memory hole? You still need some address space for it, but as it isn't limited to 32bit addresses it can be mapped somewhere above 4G, i.e. we don't have to squeeze it into the 512MB pci hole at 0xe000. We can easily create a qxl gfx card with 1G of vram (aka surface) memory. Only question is how to handle that in a backward compatible way. We have regions 4+5 in pci config space still unused. 64bit bars need two entries, so adding a 64bit bar would use up both. The new 64bit vram bar would cover all vram memory. The old 32bit vram bar would be a window to the first 32 or 64 MB of vram memory. We could probably even create some register to make the 32bit bar window offset configurable, so you can access all vram memory using the 32bit bar, just not all at the same time. Not sure the latter is worth the effort with the world moving to 64bit. cheers, Gerd
Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu
On Sun, Oct 02, 2011 at 07:12:07PM +0200, Avi Kivity wrote: On 10/02/2011 04:31 PM, Alon Levy wrote: On Sun, Oct 02, 2011 at 03:24:36PM +0200, Alon Levy wrote: Hi, Converting qemu's ram allocation to a mmap and using remap_file_pages seems like it could work. Any ideas why it wouldn't? It's linux-specific. Also, does it work on anonymous memory? Anonymous: I don't think so. I've done shm_open and mmap'ed it twice, one for the ram and the other to do remap_file_pages on it. Couldn't figure out how to get two mmaps pointing to the same physical memory without shm_open. I suggest using scatter-gather, though it's annoying. -- error compiling committee.c: too many arguments to function ___ Spice-devel mailing list spice-de...@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/spice-devel
Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu
On 10/02/2011 03:24 PM, Alon Levy wrote: Hi, I'm trying to acheive the $subject. Some background: currently spice relies on a preallocated pci bar for both surfaces and for VGA framebuffer + commands. I have been trying to get rid of the surfaces bar. To do that I allocate memory in the guest and then translate it for spice-server consumption using cpu_physical_memory_map. AFAIU this works only when the guest allocates a continuous range of physical pages. This is a large requirement from the guest, which I'd like to drop. So I would like to have the guest use a regular allocator, generating for instance two sequential pages in virtual memory that are scattered in physical memory. Those two physical guest page addresses (gp1 and gp2) correspond to two host virtual memory addresses (hv1, hv2). I would now like to provide to spice-server a single virtual address p that maps to those two pages in sequence. I don't want to handle my own scatter-gather list, I would like to have this mapping done once so I can use an existing library that requires a single pointer (for instance pixman or libGL) to do the rendering. Is there any way to acheive that without host kernel support, in user space, i.e. in qemu? or with an existing host kernel device? I'd appreciate any help, Alon ___ Spice-devel mailing list spice-de...@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/spice-devel Hi, won't there be an overhead for rendering on a non continuous surface? Will it be worthwhile comparing to not creating the surface? BTW. We should test if the split to vram (surfaces) and devram (commands and others) is more efficient than having one section. Even if it is more efficient, we can remove the split and give to the surfaces higher allocation priority on a part of the pci bar. Anyway, by default, we can try allocating surfaces on the guest RAM. If it fails, we can try to allocate on the pci-bar. Cheers, Yonit.
Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu
On Mon, Oct 03, 2011 at 10:17:59AM +0200, Yonit Halperin wrote: On 10/02/2011 03:24 PM, Alon Levy wrote: Hi, I'm trying to acheive the $subject. Some background: currently spice relies on a preallocated pci bar for both surfaces and for VGA framebuffer + commands. I have been trying to get rid of the surfaces bar. To do that I allocate memory in the guest and then translate it for spice-server consumption using cpu_physical_memory_map. AFAIU this works only when the guest allocates a continuous range of physical pages. This is a large requirement from the guest, which I'd like to drop. So I would like to have the guest use a regular allocator, generating for instance two sequential pages in virtual memory that are scattered in physical memory. Those two physical guest page addresses (gp1 and gp2) correspond to two host virtual memory addresses (hv1, hv2). I would now like to provide to spice-server a single virtual address p that maps to those two pages in sequence. I don't want to handle my own scatter-gather list, I would like to have this mapping done once so I can use an existing library that requires a single pointer (for instance pixman or libGL) to do the rendering. Is there any way to acheive that without host kernel support, in user space, i.e. in qemu? or with an existing host kernel device? I'd appreciate any help, Alon ___ Spice-devel mailing list spice-de...@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/spice-devel Hi, won't there be an overhead for rendering on a non continuous surface? Will it be worthwhile comparing to not creating the surface? If I use a scatter-gather list there is overhead of allocating and copying the surface whenever I want to synchronize. Minimally once to copy from guest to host, and another copy from host to guest for any update_area. (we can only copy the required area. If I use page remapping like remap_file_pages does, I don't think there is any overhead for rendering. There is overhead for doing the remap_file_pages calls, but they are minimal (or so the man page says). I should benchmark this. The additional cost is not large - I suppose rendering should be more costly then a memcpy. But the question is true regardless of this - some surfaces should be punted probably, if we had an oracle to know they would be immediately update_area'ed and destroyed. BTW. We should test if the split to vram (surfaces) and devram (commands and others) is more efficient than having one section. Even if it is more efficient, we can remove the split and give to the surfaces higher allocation priority on a part of the pci bar. Anyway, by default, we can try allocating surfaces on the guest RAM. If it fails, we can try to allocate on the pci-bar. Right. What I was aiming at is removing the BAR all together. This reduces per vm allocation, and we can still ensure a maximum via the driver. It also reduces PCI requirements, which are a problem with more then one card. Actually the more productive thing for reducing PCI memory would be to change to a single card for multiple monitor support. Another reason for allocating on guest RAM is to make migration simpler (but I'm not sure it really is). Cheers, Yonit.
Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu
On Mon, Oct 03, 2011 at 10:37:55AM +0200, Alon Levy wrote: On Mon, Oct 03, 2011 at 10:17:59AM +0200, Yonit Halperin wrote: On 10/02/2011 03:24 PM, Alon Levy wrote: Hi, I'm trying to acheive the $subject. Some background: currently spice relies on a preallocated pci bar for both surfaces and for VGA framebuffer + commands. I have been trying to get rid of the surfaces bar. To do that I allocate memory in the guest and then translate it for spice-server consumption using cpu_physical_memory_map. AFAIU this works only when the guest allocates a continuous range of physical pages. This is a large requirement from the guest, which I'd like to drop. So I would like to have the guest use a regular allocator, generating for instance two sequential pages in virtual memory that are scattered in physical memory. Those two physical guest page addresses (gp1 and gp2) correspond to two host virtual memory addresses (hv1, hv2). I would now like to provide to spice-server a single virtual address p that maps to those two pages in sequence. I don't want to handle my own scatter-gather list, I would like to have this mapping done once so I can use an existing library that requires a single pointer (for instance pixman or libGL) to do the rendering. Is there any way to acheive that without host kernel support, in user space, i.e. in qemu? or with an existing host kernel device? I'd appreciate any help, Alon ___ Spice-devel mailing list spice-de...@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/spice-devel Hi, won't there be an overhead for rendering on a non continuous surface? Will it be worthwhile comparing to not creating the surface? If I use a scatter-gather list there is overhead of allocating and copying the surface whenever I want to synchronize. Minimally once to copy from guest to host, and another copy from host to guest for any update_area. (we can only copy the required area. If I use page remapping like remap_file_pages does, I don't think there is any overhead for rendering. There is overhead for doing the remap_file_pages calls, but they are minimal (or so the man page says). I should benchmark this. The additional cost is not large - I suppose rendering should be more costly then a memcpy. But the question is true regardless of this - some surfaces should be punted probably, if we had an oracle to know they would be immediately update_area'ed and destroyed. Actually we could delay pushing the commands to the server until there is a command that relies on this surface, or an update_area occours. If update_area happens first, the surface has not been created yet (we only need to store the commands), and we can do the whole thing on the guest - we can't punt, since it is too late, but we can create the gdi surface ourselves and replay all the commands. If a command relying on this surface happens first (BitBlt to another surface), then we push all the commands to the server. This would play well with making the command ring hold bunches like the release ring already does, instead of individual commands. BTW. We should test if the split to vram (surfaces) and devram (commands and others) is more efficient than having one section. Even if it is more efficient, we can remove the split and give to the surfaces higher allocation priority on a part of the pci bar. Anyway, by default, we can try allocating surfaces on the guest RAM. If it fails, we can try to allocate on the pci-bar. Right. What I was aiming at is removing the BAR all together. This reduces per vm allocation, and we can still ensure a maximum via the driver. It also reduces PCI requirements, which are a problem with more then one card. Actually the more productive thing for reducing PCI memory would be to change to a single card for multiple monitor support. Another reason for allocating on guest RAM is to make migration simpler (but I'm not sure it really is). Cheers, Yonit. ___ Spice-devel mailing list spice-de...@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/spice-devel
Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu
On 10/03/2011 10:37 AM, Alon Levy wrote: Hi, won't there be an overhead for rendering on a non continuous surface? Will it be worthwhile comparing to not creating the surface? If I use a scatter-gather list there is overhead of allocating and copying the surface whenever I want to synchronize. Minimally once to copy from guest to host, and another copy from host to guest for any update_area. (we can only copy the required area. If I use page remapping like remap_file_pages does, I don't think there is any overhead for rendering. There is overhead for doing the remap_file_pages calls, but they are minimal (or so the man page says). I should benchmark this. It's not trivial; and kvm and smp magnify the cost. It shouldn't be done as part of normal rendering (setup is okay). -- error compiling committee.c: too many arguments to function
Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu
On Sun, Oct 02, 2011 at 03:24:36PM +0200, Alon Levy wrote: Hi, Converting qemu's ram allocation to a mmap and using remap_file_pages seems like it could work. Any ideas why it wouldn't? Alon I'm trying to acheive the $subject. Some background: currently spice relies on a preallocated pci bar for both surfaces and for VGA framebuffer + commands. I have been trying to get rid of the surfaces bar. To do that I allocate memory in the guest and then translate it for spice-server consumption using cpu_physical_memory_map. AFAIU this works only when the guest allocates a continuous range of physical pages. This is a large requirement from the guest, which I'd like to drop. So I would like to have the guest use a regular allocator, generating for instance two sequential pages in virtual memory that are scattered in physical memory. Those two physical guest page addresses (gp1 and gp2) correspond to two host virtual memory addresses (hv1, hv2). I would now like to provide to spice-server a single virtual address p that maps to those two pages in sequence. I don't want to handle my own scatter-gather list, I would like to have this mapping done once so I can use an existing library that requires a single pointer (for instance pixman or libGL) to do the rendering. Is there any way to acheive that without host kernel support, in user space, i.e. in qemu? or with an existing host kernel device? I'd appreciate any help, Alon ___ Spice-devel mailing list spice-de...@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/spice-devel
Re: [Qemu-devel] [Spice-devel] viewing continuous guest virtual memory as continuous in qemu
On 10/02/2011 04:31 PM, Alon Levy wrote: On Sun, Oct 02, 2011 at 03:24:36PM +0200, Alon Levy wrote: Hi, Converting qemu's ram allocation to a mmap and using remap_file_pages seems like it could work. Any ideas why it wouldn't? It's linux-specific. Also, does it work on anonymous memory? I suggest using scatter-gather, though it's annoying. -- error compiling committee.c: too many arguments to function
