[resending as plain text, sorry about that]

Thanks Daniel, more below.

From: Daniel Vetter <mailto:dan...@ffwll.ch> 
Sent: Wednesday, May 20, 2020 12:41 AM
To: Steve Pronovost <mailto:spron...@microsoft.com>
Cc: Dave Airlie <mailto:airl...@gmail.com>; Sasha Levin 
<mailto:sas...@kernel.org>; mailto:linux-hyp...@vger.kernel.org; Stephen 
Hemminger <mailto:sthem...@microsoft.com>; Ursulin, Tvrtko 
<mailto:tvrtko.ursu...@intel.com>; Greg Kroah-Hartman 
<mailto:gre...@linuxfoundation.org>; Haiyang Zhang 
<mailto:haiya...@microsoft.com>; LKML <mailto:linux-ker...@vger.kernel.org>; 
dri-devel <mailto:dri-devel@lists.freedesktop.org>; Chris Wilson 
<mailto:ch...@chris-wilson.co.uk>; Linux Fbdev development list 
<mailto:linux-fb...@vger.kernel.org>; Iouri Tarassov 
<mailto:iou...@microsoft.com>; Deucher, Alexander 
<mailto:alexander.deuc...@amd.com>; KY Srinivasan <mailto:k...@microsoft.com>; 
Wei Liu <mailto:wei....@kernel.org>; Hawking Zhang 
<mailto:hawking.zh...@amd.com>
Subject: Re: [EXTERNAL] Re: [RFC PATCH 0/4] DirectX on Linux

Hi Steve,

Sounds all good, some more comments and details below.

On Wed, May 20, 2020 at 5:47 AM Steve Pronovost <mailto:spron...@microsoft.com> 
wrote:
Hey guys,

Thanks for the discussion. I may not be able to immediately answer all of your 
questions, but I'll do my best 😊.

drivers/hyperv sounds like it could be a better location. We weren't too sure 
where to put this, we though /drivers/gpu would be appropriate given this deal 
with GPUs, but I get your point... this is a vGPU driver that really only works 
when being run under Hyper-V, so drivers/hyperv is likely more appropriate.

I think "it's a virtual gpu" is the wrong sales pitch, as is "only runs on 
$platform". We have lots of drm drivers in drivers/gpu that fit that bill. The 
better pitch I think is "it's a not a gpu, it's a dx12 protocol pipe" and "we 
actually do not want to integrate with the linux gpu ecosystem and primitives, 
we want to integrate with dx12 ecosystem and primitives to make the seamless 
rdp/rail/vail stuff work nicely". Below some more thoughts on the technical 
said.

[spronovo] Agreed. As I mentioned in another reply, that protocol isn’t tied to 
DX… but the point you are making is still valid. This is really a projection of 
the Windows native abstraction of a GPU that windows user mode driver (dx, gl, 
cl, vulkan, cuda, etc…) are familiar with and use to communicate with the GPU… 
This effectively enable porting of these user mode driver to Linux inside of 
WSL and allow them to share the GPU with the host. Our goal is to offer 
CL/EGL/GLX/CUDA/… API support for applications running inside of WSL and 
integrate their output on the Windows desktop through the Wayland compositor we 
are building. The fact that we are using layer to implement some of these APIs 
(to reduce our partners work among other thing) is just an implementation 
details that most application shouldn’t have to worry about… “it just works” 😊. 
From that perspective we’re fine moving the driver under a different node than 
/driver/gpu 😊.

In term of presentation, I need to clarify a few things. We announced today 
that we're also adding support for Linux GUI applications. The way this will 
work is roughly as follow. We're writing a Wayland compositor that will 
essentially bridge over RDP-RAIL (RAIL=Remote Application Integrated Locally). 
We're starting from a Weston base. Weston already has an RDP Backend, but 
that's for a full desktop remoting scheme. Weston draws a desktop and remote it 
over RDP... and then you can peek at that desktop using an rdp client on the 
Windows side. RAIL works differently. In that case our wayland compositor no 
longer paint a desktop... instead it simply forward individual visual / 
wl_surface over the RDP RAIL channel such that these visual can be displayed on 
the Windows desktop. The RDP client create proxy window for each of these top 
level visual and their content is filled with the data coming over the RDP 
channel. All pixels are owned by the RDP server/WSL... so these windows looks 
different than native window are they are painted and themed by WSL. The proxy 
window on the host gather input and inject back over RDP... This is essentially 
how application remoting works on windows and this is all publicly documented 
as part of the various RDP protocol specification. As a matter of fact, for the 
RDP server on the Weston side we are looking at continue to leverage FreeRDP 
(and provide fixes/enhancement as needed to the public project). Further, we're 
looking at further improvement down this path to avoid having to copy the 
content over the RAIL channel and instead just share/swap buffer between the 
guest and the host. We have extension to the RDP protocol, called VAIL 
(Virtualized Application Integrated Locally) which does that today. Today this 
is only use in Windows on Windows for very specific scenario. We're looking at 
extending the public RDP protocol with these VAIL extension to make this an 
official Microsoft supported protocol which would allow us to target this in 
WSL. We have finished designing this part in details. Our goal would be to 
leverage something along the line of wl_drm, dma-buf, dma-fence, etc... This 
compositor and all our contribution to FreeRDP will be fully open source, 
including our design doc. We're not quite sure yet whether this will be offered 
as a separate project entirely distinct from it's Weston root... or if we'll 
propose an extension to Weston to operate in this mode. We would like to build 
it such that in theory any Wayland compositor could add support for this mode 
of operation if they want to remote application to a Windows host (over the 
network, or on the same box).

Sounds like a solid plan for presentation. I think this is all up to 
wayland/weston folks to figure out with you, from the kernel side I have only 
one concern (and I discussed that with a few folks already on irc, I think 
they're at least on this thread involved within microsoft too in some form): If 
we do integrate with linux concepts like wl_drm/dma-buf/fence and so on then we 
end up with a normal gpu driver, with with lots of blobby components all around 
that can't be opened (since large chunks written by hw vendors, so nothing 
microsoft can do about them). That's the awkward exception (why microsoft but 
not other gpu hw vendors/plaforms/whatever?) that we need to avoid.

But wayland doesn't really need dma-buf and the wl_drm protocols afaiui, as 
long as the egl extensions work you can have whatever private wayland protocol 
in your winsys code you want to shovel the buffers and syncobj from client to 
the wayland-rdp-rail compositor. If that uses dx12 native handles for these 
things we side-step the awkward exception question for linux gpu stack since it 
all stays 100% contained in drivers/hv. Just try to avoid the nvidia fail of 
insisting that you need your own set of egl extensions (egl_streams and 
egl_drm_kms_reinvented_but_badly and a few others iirc) for everything, that 
makes it very painful for all the compositor projects since they need 
duplicated code for no good reason :-)

[spronovo] Agreed. I haven’t fully dig into the details for the VAIL 
enhancement to our Wayland compositor that enables buffer sharing/swapping. 
This is something we do on window and are quite familiar with the 
mechanic/concept around this… but new to doing this under Wayland. I’m sharing 
early thinking so apology if I get some stuff wrong 😊… I fundamentally agree 
with the spirit of what you are saying. Fundamentally our goal here is that we 
want to avoid having Wayland client do anything non-standard to interop with 
our projected graphics stack. We want Wayland client to work unmodified. It’s 
not entirely clear to me at the moment whether this buffer handshake is 
entirely hidden inside of Mesa (where we could do whatever we want as you say, 
and just reuse what we already have) or visible to some Wayland client, in 
which case we would want to ensure we present a public interface that is 
compatible. I gather from your suggestion that it is the former. In a nutshell 
we want to be as standard as possible so things just work and avoid having to 
chase a never ending set of app compat problem 😊.

Also looking at the fun the virtio folks have right now trying to get virgl 
dma-buf handles in the guest shared with other virtio devices or virgl instance 
in some other guest and getting it working ... If you avoid the dma-buf 
integration that might also sidestep a lot of technical headaches and be the 
simpler solution. At least as long as we're talking about wsl2 only.

[spronovo] Yeah, will definitely look into this.


We see /dev/dxg really as a projection of the GPU when running in WSL such that 
the GPU can be shared between WSL and the host... not something that would 
coexist "at the same time" with a real DRM GPU.

We have consider the possibility of bringing DX to Linux with no Windows cord 
attached. I'm not ready to discuss this at this time 😊... but in the 
hypothetical that we were do this, DX would be running on top of DRI/DRM on 
native Linux. We likely would be contributing some changes to DRM to address 
area of divergence and get better mapping for our user mode driver, but we 
wouldn't try to shoehorn /dev/dxg into the picture. In that hypothetical world, 
we would essentially have DX target DRM on native Linux and DX continue to 
target DXG in WSL to share the GPU with the host. I think this further 
reinforce the point you guys were making that the right place for our current 
dxgkrnl driver to live in would be /drivers/hyperv/dxgkrnl. In insight, I 
totally agree 😊.
 
We had a pile of discussions on irc about some of the ideas floating around for 
extending drm with some of the newer memory/sync concepts. So very much 
interested, but that's indeed a different fish and most likely will look a lot 
different to the dxgkrnl interface, while ofc still aiming to give the same 
power to apps. At least that's the goal, but aside from some prototype in some 
place this really doesn't exist yet anywhere.

[spronovo] There are indeed some very interesting and hard challenges if we 
wanted to do this 😊. But yeah, this would be totally separate from the dxgkrnl 
interface. Dxgkrnl is really meant for WSL (or Linux in a Windows hosted VM)... 
which we should have made clearer and avoid some of the confusion.


I think this cover all questions, let me know if I missed anything.

I think it's all good.

Cheers, Daniel


Thanks,
Steve

-----Original Message-----
From: Daniel Vetter <mailto:dan...@ffwll.ch> 
Sent: Tuesday, May 19, 2020 4:01 PM
To: Dave Airlie <mailto:airl...@gmail.com>
Cc: Sasha Levin <mailto:sas...@kernel.org>; 
mailto:linux-hyp...@vger.kernel.org; Stephen Hemminger 
<mailto:sthem...@microsoft.com>; Ursulin, Tvrtko 
<mailto:tvrtko.ursu...@intel.com>; Greg Kroah-Hartman 
<mailto:gre...@linuxfoundation.org>; Haiyang Zhang 
<mailto:haiya...@microsoft.com>; LKML <mailto:linux-ker...@vger.kernel.org>; 
dri-devel <mailto:dri-devel@lists.freedesktop.org>; Chris Wilson 
<mailto:ch...@chris-wilson.co.uk>; Steve Pronovost 
<mailto:spron...@microsoft.com>; Linux Fbdev development list 
<mailto:linux-fb...@vger.kernel.org>; Iouri Tarassov 
<mailto:iou...@microsoft.com>; Deucher, Alexander 
<mailto:alexander.deuc...@amd.com>; KY Srinivasan <mailto:k...@microsoft.com>; 
Wei Liu <mailto:wei....@kernel.org>; Hawking Zhang 
<mailto:hawking.zh...@amd.com>
Subject: [EXTERNAL] Re: [RFC PATCH 0/4] DirectX on Linux

On Wed, May 20, 2020 at 12:42 AM Dave Airlie <mailto:airl...@gmail.com> wrote:
>
> On Wed, 20 May 2020 at 02:33, Sasha Levin <mailto:sas...@kernel.org> wrote:
> >
> > There is a blog post that goes into more detail about the bigger 
> > picture, and walks through all the required pieces to make this 
> > work. It is available here:
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fde
> > https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fvblogs.microsoft.com%2F&data=02%7C01%7Cspronovo%40microsoft.com%7C5e9c6fb0890844aef87c08d7fc91265e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637255572659005449&sdata=gVJvYw%2BRkrdugZw5LzWbYXzoAoT2KhVjRZLs%2Ft%2BG3aY%3D&reserved=0%2Fdirectx%2Fdirectx-heart-linux&amp;data=02%7C01%7Cspronovo%https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2F40microsoft.com%2F&data=02%7C01%7Cspronovo%40microsoft.com%7C5e9c6fb0890844aef87c08d7fc91265e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637255572659015442&sdata=0TfOBiUpDLctMq7EZmK7N3xdPFeqauzsXJPODRCErXU%3D&reserved=0%7C3f18e46192b24cccf6a008d7fc489063%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637255260910730243&amp;sdata=IRRknzg%2F6Myzj3JXESN7GgmN6AcUV3DxhL95P%2ButtCw%3D&amp;reserved=0
> >  . The rest of this cover letter will focus on the Linux Kernel bits.
> >
> > Overview
> > ========
> >
> > This is the first draft of the Microsoft Virtual GPU (vGPU) driver. 
> > The driver exposes a paravirtualized GPU to user mode applications 
> > running in a virtual machine on a Windows host. This enables 
> > hardware acceleration in environment such as WSL (Windows Subsystem 
> > for Linux) where the Linux virtual machine is able to share the GPU 
> > with the Windows host.
> >
> > The projection is accomplished by exposing the WDDM (Windows Display 
> > Driver Model) interface as a set of IOCTL. This allows APIs and user 
> > mode driver written against the WDDM GPU abstraction on Windows to 
> > be ported to run within a Linux environment. This enables the port 
> > of the
> > D3D12 and DirectML APIs as well as their associated user mode driver 
> > to Linux. This also enables third party APIs, such as the popular 
> > NVIDIA Cuda compute API, to be hardware accelerated within a WSL 
> > environment.
> >
> > Only the rendering/compute aspect of the GPU are projected to the 
> > virtual machine, no display functionality is exposed. Further, at 
> > this time there are no presentation integration. So although the 
> > D3D12 API can be use to render graphics offscreen, there is no path 
> > (yet) for pixel to flow from the Linux environment back onto the 
> > Windows host desktop. This GPU stack is effectively side-by-side 
> > with the native Linux graphics stack.
>
> Okay I've had some caffiene and absorbed some more of this.
>
> This is a driver that connects a binary blob interface in the Windows 
> kernel drivers to a binary blob that you run inside a Linux guest.
> It's a binary transport between two binary pieces. Personally this 
> holds little of interest to me, I can see why it might be nice to have 
> this upstream, but I don't forsee any other Linux distributor ever 
> enabling it or having to ship it, it's purely a WSL2 pipe. I'm not 
> saying I'd be happy to see this in the tree, since I don't see the 
> value of maintaining it upstream, but it probably should just exists 
> in a drivers/hyperv type area.

Yup as-is (especially with the goal of this being aimed at ml/compute
only) drivers/hyperv sounds a bunch more reasonable than drivers/gpu.

> Having said that, I hit one stumbling block:
> "Further, at this time there are no presentation integration. "
>
> If we upstream this driver as-is into some hyperv specific place, and 
> you decide to add presentation integration this is more than likely 
> going to mean you will want to interact with dma-bufs and dma-fences.
> If the driver is hidden away in a hyperv place it's likely we won't 
> even notice that feature landing until it's too late.

I've recently added regex matches to MAINTAINERS so we'll see 
dma_buf/fence/anything show up on dri-devel. So that part is solved hopefully.

> I would like to see a coherent plan for presentation support (not 
> code, just an architectural diagram), because I think when you 
> contemplate how that works it will change the picture of how this 
> driver looks and intergrates into the rest of the Linux graphics 
> ecosystem.

Yeah once we have the feature-creep to presentation support all the integration 
fun starts, with all the questions about "why does this not look like any other 
linux gpu driver". We have that already with nvidia insisting they just can't 
implement any of the upstream gpu uapi we have, but at least they're not 
in-tree, so not our problem from an upstream maintainership pov.

But once this dx12 pipe is landed and then we want to extend it it's still 
going to have all the "we can't ever release the sources to any of the parts we 
usually expect to be open for gpu drivers in upstream"
problems. Then we're stuck at a rather awkward point of why one vendor gets an 
exception and all the others dont.

> As-is I'd rather this didn't land under my purview, since I don't see 
> the value this adds to the Linux ecosystem at all, and I think it's 
> important when putting a burden on upstream that you provide some 
> value.

Well there is some in the form of "more hw/platform support". But given that 
gpus evolved rather fast, including the entire integration ecosystem (it's by 
far not just the hw drivers that move quickly). So that value deprecates a lot 
faster than for other kernel subsystems.
And all that's left is the pain of not breaking anything without actually being 
able to evolve the overall stack in any meaningful way.
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - 
+https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fblog.f+fwll.ch%2F&data=02%7C01%7Cspronovo%40microsoft.com%7C3f18e46192b24c+ccf6a008d7fc489063%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372552
+60910735230&amp;sdata=hAIV1wJ29WF9IXTvJm3dr4StCwPzF0GdO2iWPyfnElg%3D&am
+p;reserved=0


-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - 
https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fblog.ffwll.ch%2F&data=02%7C01%7Cspronovo%40microsoft.com%7C5e9c6fb0890844aef87c08d7fc91265e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637255572659025436&sdata=me6tZuh9c4YKPTcOdY5%2FlgE%2B52usSI%2F5QRUCsCy%2FTfQ%3D&reserved=0
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Reply via email to