RE: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
Thanks, Oak > -Original Message- > From: dri-devel On Behalf Of > Zeng, Oak > Sent: June 14, 2022 5:13 PM > To: Vishwanathapura, Niranjana ; > Landwerlin, Lionel G > Cc: Intel GFX ; Wilson, Chris P > ; Hellstrom, Thomas > ; Maling list - DRI developers de...@lists.freedesktop.org>; Vetter, Daniel ; > Christian König > Subject: RE: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design > document > > > > Thanks, > Oak > > > -Original Message- > > From: Vishwanathapura, Niranjana > > Sent: June 14, 2022 1:02 PM > > To: Landwerlin, Lionel G > > Cc: Zeng, Oak ; Intel GFX > g...@lists.freedesktop.org>; Maling list - DRI developers > de...@lists.freedesktop.org>; Hellstrom, Thomas > > ; Wilson, Chris P > ; > > Vetter, Daniel ; Christian König > > > > Subject: Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design > > document > > > > On Tue, Jun 14, 2022 at 10:04:00AM +0300, Lionel Landwerlin wrote: > > >On 13/06/2022 21:02, Niranjana Vishwanathapura wrote: > > >>On Mon, Jun 13, 2022 at 06:33:07AM -0700, Zeng, Oak wrote: > > >>> > > >>> > > >>>Regards, > > >>>Oak > > >>> > > >>>>-Original Message- > > >>>>From: Intel-gfx On > > >>>>Behalf Of Niranjana > > >>>>Vishwanathapura > > >>>>Sent: June 10, 2022 1:43 PM > > >>>>To: Landwerlin, Lionel G > > >>>>Cc: Intel GFX ; Maling list - > > >>>>DRI developers > >>>>de...@lists.freedesktop.org>; Hellstrom, Thomas > > >>>>; > > >>>>Wilson, Chris P ; Vetter, Daniel > > >>>>; Christian König > > > > >>>>Subject: Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND > > >>>>feature design > > >>>>document > > >>>> > > >>>>On Fri, Jun 10, 2022 at 11:18:14AM +0300, Lionel Landwerlin wrote: > > >>>>>On 10/06/2022 10:54, Niranjana Vishwanathapura wrote: > > >>>>>>On Fri, Jun 10, 2022 at 09:53:24AM +0300, Lionel Landwerlin wrote: > > >>>>>>>On 09/06/2022 22:31, Niranjana Vishwanathapura wrote: > > >>>>>>>>On Thu, Jun 09, 2022 at 05:49:09PM +0300, Lionel Landwerlin > wrote: > > >>>>>>>>> On 09/06/2022 00:55, Jason Ekstrand wrote: > > >>>>>>>>> > > >>>>>>>>> On Wed, Jun 8, 2022 at 4:44 PM Niranjana Vishwanathapura > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>> On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko > > >>>>Ursulin wrote: > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> >On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: > > >>>>>>>>> >>On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana > > >>>>>>>>>Vishwanathapura > > >>>>>>>>> wrote: > > >>>>>>>>> >>>On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason > > >>>>>>>>>Ekstrand wrote: > > >>>>>>>>> >>>> On Fri, Jun 3, 2022 at 6:52 PM Niranjana > > >>>>Vishwanathapura > > >>>>>>>>> >>>> wrote: > > >>>>>>>>> >>>> > > >>>>>>>>> >>>> On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel > > >>>>>>>>>Landwerlin > > >>>>>>>>> wrote: > > >>>>>>>>> >>>> > On 02/06/2022 23:35, Jason Ekstrand wrote: > > >>>>>>>>> >>>> > > > >>>>>>>>> >>>> > On Thu, Jun 2, 2022 at 3:11 PM Niranjana > > >>>>>>>>>Vishwanathapura > > >>>>>>>>> >>>> > wrote: > > >>>>>>>>> >>>> > > > >>>>>>>>> >>>> > On Wed, Jun 01, 2022 at 01:28:36PM > > >>>>-0700, Matt
RE: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
Thanks, Oak > -Original Message- > From: Vishwanathapura, Niranjana > Sent: June 14, 2022 1:02 PM > To: Landwerlin, Lionel G > Cc: Zeng, Oak ; Intel GFX g...@lists.freedesktop.org>; Maling list - DRI developers de...@lists.freedesktop.org>; Hellstrom, Thomas > ; Wilson, Chris P ; > Vetter, Daniel ; Christian König > > Subject: Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design > document > > On Tue, Jun 14, 2022 at 10:04:00AM +0300, Lionel Landwerlin wrote: > >On 13/06/2022 21:02, Niranjana Vishwanathapura wrote: > >>On Mon, Jun 13, 2022 at 06:33:07AM -0700, Zeng, Oak wrote: > >>> > >>> > >>>Regards, > >>>Oak > >>> > >>>>-Original Message- > >>>>From: Intel-gfx On > >>>>Behalf Of Niranjana > >>>>Vishwanathapura > >>>>Sent: June 10, 2022 1:43 PM > >>>>To: Landwerlin, Lionel G > >>>>Cc: Intel GFX ; Maling list - > >>>>DRI developers >>>>de...@lists.freedesktop.org>; Hellstrom, Thomas > >>>>; > >>>>Wilson, Chris P ; Vetter, Daniel > >>>>; Christian König > > >>>>Subject: Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND > >>>>feature design > >>>>document > >>>> > >>>>On Fri, Jun 10, 2022 at 11:18:14AM +0300, Lionel Landwerlin wrote: > >>>>>On 10/06/2022 10:54, Niranjana Vishwanathapura wrote: > >>>>>>On Fri, Jun 10, 2022 at 09:53:24AM +0300, Lionel Landwerlin wrote: > >>>>>>>On 09/06/2022 22:31, Niranjana Vishwanathapura wrote: > >>>>>>>>On Thu, Jun 09, 2022 at 05:49:09PM +0300, Lionel Landwerlin wrote: > >>>>>>>>> On 09/06/2022 00:55, Jason Ekstrand wrote: > >>>>>>>>> > >>>>>>>>> On Wed, Jun 8, 2022 at 4:44 PM Niranjana Vishwanathapura > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko > >>>>Ursulin wrote: > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> >On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: > >>>>>>>>> >>On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana > >>>>>>>>>Vishwanathapura > >>>>>>>>> wrote: > >>>>>>>>> >>>On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason > >>>>>>>>>Ekstrand wrote: > >>>>>>>>> >>>> On Fri, Jun 3, 2022 at 6:52 PM Niranjana > >>>>Vishwanathapura > >>>>>>>>> >>>> wrote: > >>>>>>>>> >>>> > >>>>>>>>> >>>> On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel > >>>>>>>>>Landwerlin > >>>>>>>>> wrote: > >>>>>>>>> >>>> > On 02/06/2022 23:35, Jason Ekstrand wrote: > >>>>>>>>> >>>> > > >>>>>>>>> >>>> > On Thu, Jun 2, 2022 at 3:11 PM Niranjana > >>>>>>>>>Vishwanathapura > >>>>>>>>> >>>> > wrote: > >>>>>>>>> >>>> > > >>>>>>>>> >>>> > On Wed, Jun 01, 2022 at 01:28:36PM > >>>>-0700, Matthew > >>>>>>>>> >>>>Brost wrote: > >>>>>>>>> >>>> > >On Wed, Jun 01, 2022 at 05:25:49PM > >>>>+0300, Lionel > >>>>>>>>> Landwerlin > >>>>>>>>> >>>> wrote: > >>>>>>>>> >>>> > >> On 17/05/2022 21:32, Niranjana Vishwanathapura > >>>>>>>>> wrote: > >>>>>>>>> >>>> > >> > +VM_BIND/UNBIND ioctl will immediately start > >>>>>>>>> >>>> binding/unbinding > >>>>>>>>> >>>> > the
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Tue, Jun 14, 2022 at 10:04:00AM +0300, Lionel Landwerlin wrote: On 13/06/2022 21:02, Niranjana Vishwanathapura wrote: On Mon, Jun 13, 2022 at 06:33:07AM -0700, Zeng, Oak wrote: Regards, Oak -Original Message- From: Intel-gfx On Behalf Of Niranjana Vishwanathapura Sent: June 10, 2022 1:43 PM To: Landwerlin, Lionel G Cc: Intel GFX ; Maling list - DRI developers de...@lists.freedesktop.org>; Hellstrom, Thomas ; Wilson, Chris P ; Vetter, Daniel ; Christian König Subject: Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document On Fri, Jun 10, 2022 at 11:18:14AM +0300, Lionel Landwerlin wrote: On 10/06/2022 10:54, Niranjana Vishwanathapura wrote: On Fri, Jun 10, 2022 at 09:53:24AM +0300, Lionel Landwerlin wrote: On 09/06/2022 22:31, Niranjana Vishwanathapura wrote: On Thu, Jun 09, 2022 at 05:49:09PM +0300, Lionel Landwerlin wrote: On 09/06/2022 00:55, Jason Ekstrand wrote: On Wed, Jun 8, 2022 at 4:44 PM Niranjana Vishwanathapura wrote: On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko Ursulin wrote: > > >On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: >>On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana Vishwanathapura wrote: >>>On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason Ekstrand wrote: >>>> On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura >>>> wrote: >>>> >>>> On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin wrote: >>>> > On 02/06/2022 23:35, Jason Ekstrand wrote: >>>> > >>>> > On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura >>>> > wrote: >>>> > >>>> > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew >>>>Brost wrote: >>>> > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin >>>> wrote: >>>> > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: >>>> > >> > +VM_BIND/UNBIND ioctl will immediately start >>>> binding/unbinding >>>> > the mapping in an >>>> > >> > +async worker. The binding and unbinding will >>>>work like a >>>> special >>>> > GPU engine. >>>> > >> > +The binding and unbinding operations are serialized and >>>> will >>>> > wait on specified >>>> > >> > +input fences before the operation and will signal the >>>> output >>>> > fences upon the >>>> > >> > +completion of the operation. Due to serialization, >>>> completion of >>>> > an operation >>>> > >> > +will also indicate that all previous operations >>>>are also >>>> > complete. >>>> > >> >>>> > >> I guess we should avoid saying "will immediately start >>>> > binding/unbinding" if >>>> > >> there are fences involved. >>>> > >> >>>> > >> And the fact that it's happening in an async >>>>worker seem to >>>> imply >>>> > it's not >>>> > >> immediate. >>>> > >> >>>> > >>>> > Ok, will fix. >>>> > This was added because in earlier design binding was deferred >>>> until >>>> > next execbuff. >>>> > But now it is non-deferred (immediate in that sense). >>>>But yah, >>>> this is >>>> > confusing >>>> > and will fix it. >>>> > >>>> > >> >>>> > >> I have a question on the behavior of the bind >>>>operation when >>>> no >>>> > input fence >>>> > >> is provided. Let say I do : >>>> > >> >>>> > >> VM_BIND (out_fence=fence1) >>>> > >> >>>> > >> VM_
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On 13/06/2022 21:02, Niranjana Vishwanathapura wrote: On Mon, Jun 13, 2022 at 06:33:07AM -0700, Zeng, Oak wrote: Regards, Oak -Original Message- From: Intel-gfx On Behalf Of Niranjana Vishwanathapura Sent: June 10, 2022 1:43 PM To: Landwerlin, Lionel G Cc: Intel GFX ; Maling list - DRI developers de...@lists.freedesktop.org>; Hellstrom, Thomas ; Wilson, Chris P ; Vetter, Daniel ; Christian König Subject: Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document On Fri, Jun 10, 2022 at 11:18:14AM +0300, Lionel Landwerlin wrote: >On 10/06/2022 10:54, Niranjana Vishwanathapura wrote: >>On Fri, Jun 10, 2022 at 09:53:24AM +0300, Lionel Landwerlin wrote: >>>On 09/06/2022 22:31, Niranjana Vishwanathapura wrote: >>>>On Thu, Jun 09, 2022 at 05:49:09PM +0300, Lionel Landwerlin wrote: >>>>> On 09/06/2022 00:55, Jason Ekstrand wrote: >>>>> >>>>> On Wed, Jun 8, 2022 at 4:44 PM Niranjana Vishwanathapura >>>>> wrote: >>>>> >>>>> On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko Ursulin wrote: >>>>> > >>>>> > >>>>> >On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: >>>>> >>On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana >>>>>Vishwanathapura >>>>> wrote: >>>>> >>>On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason >>>>>Ekstrand wrote: >>>>> >>>> On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura >>>>> >>>> wrote: >>>>> >>>> >>>>> >>>> On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel >>>>>Landwerlin >>>>> wrote: >>>>> >>>> > On 02/06/2022 23:35, Jason Ekstrand wrote: >>>>> >>>> > >>>>> >>>> > On Thu, Jun 2, 2022 at 3:11 PM Niranjana >>>>>Vishwanathapura >>>>> >>>> > wrote: >>>>> >>>> > >>>>> >>>> > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew >>>>> >>>>Brost wrote: >>>>> >>>> > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel >>>>> Landwerlin >>>>> >>>> wrote: >>>>> >>>> > >> On 17/05/2022 21:32, Niranjana Vishwanathapura >>>>> wrote: >>>>> >>>> > >> > +VM_BIND/UNBIND ioctl will immediately start >>>>> >>>> binding/unbinding >>>>> >>>> > the mapping in an >>>>> >>>> > >> > +async worker. The binding and >>>>>unbinding will >>>>> >>>>work like a >>>>> >>>> special >>>>> >>>> > GPU engine. >>>>> >>>> > >> > +The binding and unbinding operations are >>>>> serialized and >>>>> >>>> will >>>>> >>>> > wait on specified >>>>> >>>> > >> > +input fences before the operation >>>>>and will signal >>>>> the >>>>> >>>> output >>>>> >>>> > fences upon the >>>>> >>>> > >> > +completion of the operation. Due to >>>>> serialization, >>>>> >>>> completion of >>>>> >>>> > an operation >>>>> >>>> > >> > +will also indicate that all >>>>>previous operations >>>>> >>>>are also >>>>> >>>> > complete. >>>>> >>>> > >> >>>>> >>>> > >> I guess we should avoid saying "will >>>>>immediately >>>>> start >>>>> >>>> > binding/unbinding" if >>>>> >>>> > >> there are fences involved. >>>>> >>>> > >> >>>>> >>>> > >> And the fact that i
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Mon, Jun 13, 2022 at 06:33:07AM -0700, Zeng, Oak wrote: Regards, Oak -Original Message- From: Intel-gfx On Behalf Of Niranjana Vishwanathapura Sent: June 10, 2022 1:43 PM To: Landwerlin, Lionel G Cc: Intel GFX ; Maling list - DRI developers ; Hellstrom, Thomas ; Wilson, Chris P ; Vetter, Daniel ; Christian König Subject: Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document On Fri, Jun 10, 2022 at 11:18:14AM +0300, Lionel Landwerlin wrote: >On 10/06/2022 10:54, Niranjana Vishwanathapura wrote: >>On Fri, Jun 10, 2022 at 09:53:24AM +0300, Lionel Landwerlin wrote: >>>On 09/06/2022 22:31, Niranjana Vishwanathapura wrote: >>>>On Thu, Jun 09, 2022 at 05:49:09PM +0300, Lionel Landwerlin wrote: >>>>> On 09/06/2022 00:55, Jason Ekstrand wrote: >>>>> >>>>>On Wed, Jun 8, 2022 at 4:44 PM Niranjana Vishwanathapura >>>>> wrote: >>>>> >>>>> On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko Ursulin wrote: >>>>> > >>>>> > >>>>> >On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: >>>>> >>On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana >>>>>Vishwanathapura >>>>> wrote: >>>>> >>>On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason >>>>>Ekstrand wrote: >>>>> >>>> On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura >>>>> >>>> wrote: >>>>> >>>> >>>>> >>>> On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel >>>>>Landwerlin >>>>> wrote: >>>>> >>>> > On 02/06/2022 23:35, Jason Ekstrand wrote: >>>>> >>>> > >>>>> >>>> > On Thu, Jun 2, 2022 at 3:11 PM Niranjana >>>>>Vishwanathapura >>>>> >>>> > wrote: >>>>> >>>> > >>>>> >>>> > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew >>>>> >>>>Brost wrote: >>>>> >>>> > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel >>>>> Landwerlin >>>>> >>>> wrote: >>>>> >>>> > >> On 17/05/2022 21:32, Niranjana Vishwanathapura >>>>> wrote: >>>>> >>>> > >> > +VM_BIND/UNBIND ioctl will immediately start >>>>> >>>> binding/unbinding >>>>> >>>> > the mapping in an >>>>> >>>> > >> > +async worker. The binding and >>>>>unbinding will >>>>> >>>>work like a >>>>> >>>> special >>>>> >>>> > GPU engine. >>>>> >>>> > >> > +The binding and unbinding operations are >>>>> serialized and >>>>> >>>> will >>>>> >>>> > wait on specified >>>>> >>>> > >> > +input fences before the operation >>>>>and will signal >>>>> the >>>>> >>>> output >>>>> >>>> > fences upon the >>>>> >>>> > >> > +completion of the operation. Due to >>>>> serialization, >>>>> >>>> completion of >>>>> >>>> > an operation >>>>> >>>> > >> > +will also indicate that all >>>>>previous operations >>>>> >>>>are also >>>>> >>>> > complete. >>>>> >>>> > >> >>>>> >>>> > >> I guess we should avoid saying "will >>>>>immediately >>>>> start >>>>> >>>> > binding/unbinding" if >>>>> >>>> > >> there are fences involved. >>>>> >>>> > >> >>>>> >>>> > >> And the fact that it's happening in an
RE: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
Regards, Oak > -Original Message- > From: Intel-gfx On Behalf Of > Niranjana > Vishwanathapura > Sent: June 10, 2022 1:43 PM > To: Landwerlin, Lionel G > Cc: Intel GFX ; Maling list - DRI developers > de...@lists.freedesktop.org>; Hellstrom, Thomas ; > Wilson, Chris P ; Vetter, Daniel > ; Christian König > Subject: Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design > document > > On Fri, Jun 10, 2022 at 11:18:14AM +0300, Lionel Landwerlin wrote: > >On 10/06/2022 10:54, Niranjana Vishwanathapura wrote: > >>On Fri, Jun 10, 2022 at 09:53:24AM +0300, Lionel Landwerlin wrote: > >>>On 09/06/2022 22:31, Niranjana Vishwanathapura wrote: > >>>>On Thu, Jun 09, 2022 at 05:49:09PM +0300, Lionel Landwerlin wrote: > >>>>> On 09/06/2022 00:55, Jason Ekstrand wrote: > >>>>> > >>>>> On Wed, Jun 8, 2022 at 4:44 PM Niranjana Vishwanathapura > >>>>> wrote: > >>>>> > >>>>> On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko Ursulin wrote: > >>>>> > > >>>>> > > >>>>> >On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: > >>>>> >>On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana > >>>>>Vishwanathapura > >>>>> wrote: > >>>>> >>>On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason > >>>>>Ekstrand wrote: > >>>>> >>>> On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura > >>>>> >>>> wrote: > >>>>> >>>> > >>>>> >>>> On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel > >>>>>Landwerlin > >>>>> wrote: > >>>>> >>>> > On 02/06/2022 23:35, Jason Ekstrand wrote: > >>>>> >>>> > > >>>>> >>>> > On Thu, Jun 2, 2022 at 3:11 PM Niranjana > >>>>>Vishwanathapura > >>>>> >>>> > wrote: > >>>>> >>>> > > >>>>> >>>> > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew > >>>>> >>>>Brost wrote: > >>>>> >>>> > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel > >>>>> Landwerlin > >>>>> >>>> wrote: > >>>>> >>>> > >> On 17/05/2022 21:32, Niranjana Vishwanathapura > >>>>> wrote: > >>>>> >>>> > >> > +VM_BIND/UNBIND ioctl will immediately start > >>>>> >>>> binding/unbinding > >>>>> >>>> > the mapping in an > >>>>> >>>> > >> > +async worker. The binding and > >>>>>unbinding will > >>>>> >>>>work like a > >>>>> >>>> special > >>>>> >>>> > GPU engine. > >>>>> >>>> > >> > +The binding and unbinding operations are > >>>>> serialized and > >>>>> >>>> will > >>>>> >>>> > wait on specified > >>>>> >>>> > >> > +input fences before the operation > >>>>>and will signal > >>>>> the > >>>>> >>>> output > >>>>> >>>> > fences upon the > >>>>> >>>> > >> > +completion of the operation. Due to > >>>>> serialization, > >>>>> >>>> completion of > >>>>> >>>> > an operation > >>>>> >>>> > >> > +will also indicate that all > >>>>>previous operations > >>>>> >>>>are also > >>>>> >>>> > complete. > >>>>> >>>> > >> > >>>>> >>>> > >> I guess we should avoid saying "will > >>>>>immediately > >>>
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Fri, Jun 10, 2022 at 11:18:14AM +0300, Lionel Landwerlin wrote: On 10/06/2022 10:54, Niranjana Vishwanathapura wrote: On Fri, Jun 10, 2022 at 09:53:24AM +0300, Lionel Landwerlin wrote: On 09/06/2022 22:31, Niranjana Vishwanathapura wrote: On Thu, Jun 09, 2022 at 05:49:09PM +0300, Lionel Landwerlin wrote: On 09/06/2022 00:55, Jason Ekstrand wrote: On Wed, Jun 8, 2022 at 4:44 PM Niranjana Vishwanathapura wrote: On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko Ursulin wrote: > > >On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: >>On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana Vishwanathapura wrote: >>>On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason Ekstrand wrote: On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura wrote: On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin wrote: > On 02/06/2022 23:35, Jason Ekstrand wrote: > > On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura > wrote: > > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > >> > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding > the mapping in an > >> > +async worker. The binding and unbinding will work like a special > GPU engine. > >> > +The binding and unbinding operations are serialized and will > wait on specified > >> > +input fences before the operation and will signal the output > fences upon the > >> > +completion of the operation. Due to serialization, completion of > an operation > >> > +will also indicate that all previous operations are also > complete. > >> > >> I guess we should avoid saying "will immediately start > binding/unbinding" if > >> there are fences involved. > >> > >> And the fact that it's happening in an async worker seem to imply > it's not > >> immediate. > >> > > Ok, will fix. > This was added because in earlier design binding was deferred until > next execbuff. > But now it is non-deferred (immediate in that sense). But yah, this is > confusing > and will fix it. > > >> > >> I have a question on the behavior of the bind operation when no > input fence > >> is provided. Let say I do : > >> > >> VM_BIND (out_fence=fence1) > >> > >> VM_BIND (out_fence=fence2) > >> > >> VM_BIND (out_fence=fence3) > >> > >> > >> In what order are the fences going to be signaled? > >> > >> In the order of VM_BIND ioctls? Or out of order? > >> > >> Because you wrote "serialized I assume it's : in order > >> > > Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and unbind > will use > the same queue and hence are ordered. > > >> > >> One thing I didn't realize is that because we only get one > "VM_BIND" engine, > >> there is a disconnect from the Vulkan specification. > >> > >> In Vulkan VM_BIND operations are serialized but per engine. > >> > >> So you could have something like this : > >> > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > >> > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > >> > >> > >> fence1 is not signaled > >> > >> fence3 is signaled > >> >
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On 10/06/2022 10:54, Niranjana Vishwanathapura wrote: On Fri, Jun 10, 2022 at 09:53:24AM +0300, Lionel Landwerlin wrote: On 09/06/2022 22:31, Niranjana Vishwanathapura wrote: On Thu, Jun 09, 2022 at 05:49:09PM +0300, Lionel Landwerlin wrote: On 09/06/2022 00:55, Jason Ekstrand wrote: On Wed, Jun 8, 2022 at 4:44 PM Niranjana Vishwanathapura wrote: On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko Ursulin wrote: > > >On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: >>On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana Vishwanathapura wrote: >>>On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason Ekstrand wrote: On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura wrote: On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin wrote: > On 02/06/2022 23:35, Jason Ekstrand wrote: > > On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura > wrote: > > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > >> > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding > the mapping in an > >> > +async worker. The binding and unbinding will work like a special > GPU engine. > >> > +The binding and unbinding operations are serialized and will > wait on specified > >> > +input fences before the operation and will signal the output > fences upon the > >> > +completion of the operation. Due to serialization, completion of > an operation > >> > +will also indicate that all previous operations are also > complete. > >> > >> I guess we should avoid saying "will immediately start > binding/unbinding" if > >> there are fences involved. > >> > >> And the fact that it's happening in an async worker seem to imply > it's not > >> immediate. > >> > > Ok, will fix. > This was added because in earlier design binding was deferred until > next execbuff. > But now it is non-deferred (immediate in that sense). But yah, this is > confusing > and will fix it. > > >> > >> I have a question on the behavior of the bind operation when no > input fence > >> is provided. Let say I do : > >> > >> VM_BIND (out_fence=fence1) > >> > >> VM_BIND (out_fence=fence2) > >> > >> VM_BIND (out_fence=fence3) > >> > >> > >> In what order are the fences going to be signaled? > >> > >> In the order of VM_BIND ioctls? Or out of order? > >> > >> Because you wrote "serialized I assume it's : in order > >> > > Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and unbind > will use > the same queue and hence are ordered. > > >> > >> One thing I didn't realize is that because we only get one > "VM_BIND" engine, > >> there is a disconnect from the Vulkan specification. > >> > >> In Vulkan VM_BIND operations are serialized but per engine. > >> > >> So you could have something like this : > >> > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > >> > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > >> > >> > >> fence1 is not signaled > >> > >> fence3 is signaled > >> > >> So the second VM_BIND will proceed before the first
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Fri, Jun 10, 2022 at 09:53:24AM +0300, Lionel Landwerlin wrote: On 09/06/2022 22:31, Niranjana Vishwanathapura wrote: On Thu, Jun 09, 2022 at 05:49:09PM +0300, Lionel Landwerlin wrote: On 09/06/2022 00:55, Jason Ekstrand wrote: On Wed, Jun 8, 2022 at 4:44 PM Niranjana Vishwanathapura wrote: On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko Ursulin wrote: > > >On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: >>On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana Vishwanathapura wrote: >>>On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason Ekstrand wrote: On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura wrote: On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin wrote: > On 02/06/2022 23:35, Jason Ekstrand wrote: > > On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura > wrote: > > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > >> > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding > the mapping in an > >> > +async worker. The binding and unbinding will work like a special > GPU engine. > >> > +The binding and unbinding operations are serialized and will > wait on specified > >> > +input fences before the operation and will signal the output > fences upon the > >> > +completion of the operation. Due to serialization, completion of > an operation > >> > +will also indicate that all previous operations are also > complete. > >> > >> I guess we should avoid saying "will immediately start > binding/unbinding" if > >> there are fences involved. > >> > >> And the fact that it's happening in an async worker seem to imply > it's not > >> immediate. > >> > > Ok, will fix. > This was added because in earlier design binding was deferred until > next execbuff. > But now it is non-deferred (immediate in that sense). But yah, this is > confusing > and will fix it. > > >> > >> I have a question on the behavior of the bind operation when no > input fence > >> is provided. Let say I do : > >> > >> VM_BIND (out_fence=fence1) > >> > >> VM_BIND (out_fence=fence2) > >> > >> VM_BIND (out_fence=fence3) > >> > >> > >> In what order are the fences going to be signaled? > >> > >> In the order of VM_BIND ioctls? Or out of order? > >> > >> Because you wrote "serialized I assume it's : in order > >> > > Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and unbind > will use > the same queue and hence are ordered. > > >> > >> One thing I didn't realize is that because we only get one > "VM_BIND" engine, > >> there is a disconnect from the Vulkan specification. > >> > >> In Vulkan VM_BIND operations are serialized but per engine. > >> > >> So you could have something like this : > >> > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > >> > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > >> > >> > >> fence1 is not signaled > >> > >> fence3 is signaled > >> > >> So the second VM_BIND will proceed before the first VM_BIND. > >> > >>
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On 09/06/2022 22:31, Niranjana Vishwanathapura wrote: On Thu, Jun 09, 2022 at 05:49:09PM +0300, Lionel Landwerlin wrote: On 09/06/2022 00:55, Jason Ekstrand wrote: On Wed, Jun 8, 2022 at 4:44 PM Niranjana Vishwanathapura wrote: On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko Ursulin wrote: > > >On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: >>On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana Vishwanathapura wrote: >>>On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason Ekstrand wrote: On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura wrote: On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin wrote: > On 02/06/2022 23:35, Jason Ekstrand wrote: > > On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura > wrote: > > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > >> > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding > the mapping in an > >> > +async worker. The binding and unbinding will work like a special > GPU engine. > >> > +The binding and unbinding operations are serialized and will > wait on specified > >> > +input fences before the operation and will signal the output > fences upon the > >> > +completion of the operation. Due to serialization, completion of > an operation > >> > +will also indicate that all previous operations are also > complete. > >> > >> I guess we should avoid saying "will immediately start > binding/unbinding" if > >> there are fences involved. > >> > >> And the fact that it's happening in an async worker seem to imply > it's not > >> immediate. > >> > > Ok, will fix. > This was added because in earlier design binding was deferred until > next execbuff. > But now it is non-deferred (immediate in that sense). But yah, this is > confusing > and will fix it. > > >> > >> I have a question on the behavior of the bind operation when no > input fence > >> is provided. Let say I do : > >> > >> VM_BIND (out_fence=fence1) > >> > >> VM_BIND (out_fence=fence2) > >> > >> VM_BIND (out_fence=fence3) > >> > >> > >> In what order are the fences going to be signaled? > >> > >> In the order of VM_BIND ioctls? Or out of order? > >> > >> Because you wrote "serialized I assume it's : in order > >> > > Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and unbind > will use > the same queue and hence are ordered. > > >> > >> One thing I didn't realize is that because we only get one > "VM_BIND" engine, > >> there is a disconnect from the Vulkan specification. > >> > >> In Vulkan VM_BIND operations are serialized but per engine. > >> > >> So you could have something like this : > >> > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > >> > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > >> > >> > >> fence1 is not signaled > >> > >> fence3 is signaled > >> > >> So the second VM_BIND will proceed before the first VM_BIND. > >> > >> > >> I guess we can deal with that scenario in use
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Thu, Jun 09, 2022 at 05:49:09PM +0300, Lionel Landwerlin wrote:
On 09/06/2022 00:55, Jason Ekstrand wrote:
On Wed, Jun 8, 2022 at 4:44 PM Niranjana Vishwanathapura
wrote:
On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko Ursulin wrote:
>
>
>On 07/06/2022 22:32, Niranjana Vishwanathapura wrote:
>>On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana Vishwanathapura
wrote:
>>>On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason Ekstrand wrote:
On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura
wrote:
On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin
wrote:
> On 02/06/2022 23:35, Jason Ekstrand wrote:
>
> On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura
> wrote:
>
> On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew
Brost wrote:
> >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel
Landwerlin
wrote:
> >> On 17/05/2022 21:32, Niranjana Vishwanathapura
wrote:
> >> > +VM_BIND/UNBIND ioctl will immediately start
binding/unbinding
> the mapping in an
> >> > +async worker. The binding and unbinding will
work like a
special
> GPU engine.
> >> > +The binding and unbinding operations are
serialized and
will
> wait on specified
> >> > +input fences before the operation and will signal
the
output
> fences upon the
> >> > +completion of the operation. Due to
serialization,
completion of
> an operation
> >> > +will also indicate that all previous operations
are also
> complete.
> >>
> >> I guess we should avoid saying "will immediately
start
> binding/unbinding" if
> >> there are fences involved.
> >>
> >> And the fact that it's happening in an async
worker seem to
imply
> it's not
> >> immediate.
> >>
>
> Ok, will fix.
> This was added because in earlier design binding was
deferred
until
> next execbuff.
> But now it is non-deferred (immediate in that sense).
But yah,
this is
> confusing
> and will fix it.
>
> >>
> >> I have a question on the behavior of the bind
operation when
no
> input fence
> >> is provided. Let say I do :
> >>
> >> VM_BIND (out_fence=fence1)
> >>
> >> VM_BIND (out_fence=fence2)
> >>
> >> VM_BIND (out_fence=fence3)
> >>
> >>
> >> In what order are the fences going to be signaled?
> >>
> >> In the order of VM_BIND ioctls? Or out of order?
> >>
> >> Because you wrote "serialized I assume it's : in
order
> >>
>
> Yes, in the order of VM_BIND/UNBIND ioctls. Note that
bind and
unbind
> will use
> the same queue and hence are ordered.
>
> >>
> >> One thing I didn't realize is that because we only
get one
> "VM_BIND" engine,
> >> there is a disconnect from the Vulkan specification.
> >>
> >> In Vulkan VM_BIND operations are serialized but
per engine.
> >>
> >> So you could have something like this :
> >>
> >> VM_BIND (engine=rcs0, in_fence=fence1,
out_fence=fence2)
> >>
> >> VM_BIND (engine=ccs0, in_fence=fence3,
out_fence=fence4)
> >>
> >>
> >> fence1 is not signaled
> >>
> >> fence3 is signaled
> >>
> >> So the second VM_BIND will proceed before the
first VM_BIND.
> >>
> >>
> >> I guess we can deal with that scenario in
userspace by doing
the
> wait
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On 09/06/2022 00:55, Jason Ekstrand wrote: On Wed, Jun 8, 2022 at 4:44 PM Niranjana Vishwanathapura wrote: On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko Ursulin wrote: > > >On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: >>On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana Vishwanathapura wrote: >>>On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason Ekstrand wrote: On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura wrote: On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin wrote: > On 02/06/2022 23:35, Jason Ekstrand wrote: > > On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura > wrote: > > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > >> > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding > the mapping in an > >> > +async worker. The binding and unbinding will work like a special > GPU engine. > >> > +The binding and unbinding operations are serialized and will > wait on specified > >> > +input fences before the operation and will signal the output > fences upon the > >> > +completion of the operation. Due to serialization, completion of > an operation > >> > +will also indicate that all previous operations are also > complete. > >> > >> I guess we should avoid saying "will immediately start > binding/unbinding" if > >> there are fences involved. > >> > >> And the fact that it's happening in an async worker seem to imply > it's not > >> immediate. > >> > > Ok, will fix. > This was added because in earlier design binding was deferred until > next execbuff. > But now it is non-deferred (immediate in that sense). But yah, this is > confusing > and will fix it. > > >> > >> I have a question on the behavior of the bind operation when no > input fence > >> is provided. Let say I do : > >> > >> VM_BIND (out_fence=fence1) > >> > >> VM_BIND (out_fence=fence2) > >> > >> VM_BIND (out_fence=fence3) > >> > >> > >> In what order are the fences going to be signaled? > >> > >> In the order of VM_BIND ioctls? Or out of order? > >> > >> Because you wrote "serialized I assume it's : in order > >> > > Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and unbind > will use > the same queue and hence are ordered. > > >> > >> One thing I didn't realize is that because we only get one > "VM_BIND" engine, > >> there is a disconnect from the Vulkan specification. > >> > >> In Vulkan VM_BIND operations are serialized but per engine. > >> > >> So you could have something like this : > >> > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > >> > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > >> > >> > >> fence1 is not signaled > >> > >> fence3 is signaled > >> > >> So the second VM_BIND will proceed before the first VM_BIND. > >> > >> > >> I guess we can deal with that scenario in userspace by doing the > wait > >> ourselves in one thread per engines. > >> > >> But then it makes the VM_BIND input fences useless. > >> > >> > >> Daniel : what do you think? Should be rework this or just deal with > wait >
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Wed, Jun 08, 2022 at 04:55:38PM -0500, Jason Ekstrand wrote: On Wed, Jun 8, 2022 at 4:44 PM Niranjana Vishwanathapura wrote: On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko Ursulin wrote: > > >On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: >>On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana Vishwanathapura wrote: >>>On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason Ekstrand wrote: On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura wrote: On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin wrote: > On 02/06/2022 23:35, Jason Ekstrand wrote: > > On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura > wrote: > > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > >> > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding > the mapping in an > >> > +async worker. The binding and unbinding will work like a special > GPU engine. > >> > +The binding and unbinding operations are serialized and will > wait on specified > >> > +input fences before the operation and will signal the output > fences upon the > >> > +completion of the operation. Due to serialization, completion of > an operation > >> > +will also indicate that all previous operations are also > complete. > >> > >> I guess we should avoid saying "will immediately start > binding/unbinding" if > >> there are fences involved. > >> > >> And the fact that it's happening in an async worker seem to imply > it's not > >> immediate. > >> > > Ok, will fix. > This was added because in earlier design binding was deferred until > next execbuff. > But now it is non-deferred (immediate in that sense). But yah, this is > confusing > and will fix it. > > >> > >> I have a question on the behavior of the bind operation when no > input fence > >> is provided. Let say I do : > >> > >> VM_BIND (out_fence=fence1) > >> > >> VM_BIND (out_fence=fence2) > >> > >> VM_BIND (out_fence=fence3) > >> > >> > >> In what order are the fences going to be signaled? > >> > >> In the order of VM_BIND ioctls? Or out of order? > >> > >> Because you wrote "serialized I assume it's : in order > >> > > Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and unbind > will use > the same queue and hence are ordered. > > >> > >> One thing I didn't realize is that because we only get one > "VM_BIND" engine, > >> there is a disconnect from the Vulkan specification. > >> > >> In Vulkan VM_BIND operations are serialized but per engine. > >> > >> So you could have something like this : > >> > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > >> > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > >> > >> > >> fence1 is not signaled > >> > >> fence3 is signaled > >> > >> So the second VM_BIND will proceed before the first VM_BIND. > >> > >> > >> I guess we can deal with that scenario in userspace by doing the > wait > >> ourselves in one thread per engines. > >> > >> But then it makes the VM_BIND input fences useless. > >> > >> > >> Daniel : what do you think? Should be rework this or just deal with > wait
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Wed, Jun 8, 2022 at 4:44 PM Niranjana Vishwanathapura < niranjana.vishwanathap...@intel.com> wrote: > On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko Ursulin wrote: > > > > > >On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: > >>On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana Vishwanathapura > wrote: > >>>On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason Ekstrand wrote: > On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura > wrote: > > On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin wrote: > > On 02/06/2022 23:35, Jason Ekstrand wrote: > > > > On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura > > wrote: > > > > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew > Brost wrote: > > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin > wrote: > > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > > >> > +VM_BIND/UNBIND ioctl will immediately start > binding/unbinding > > the mapping in an > > >> > +async worker. The binding and unbinding will > work like a > special > > GPU engine. > > >> > +The binding and unbinding operations are serialized > and > will > > wait on specified > > >> > +input fences before the operation and will signal the > output > > fences upon the > > >> > +completion of the operation. Due to serialization, > completion of > > an operation > > >> > +will also indicate that all previous operations > are also > > complete. > > >> > > >> I guess we should avoid saying "will immediately start > > binding/unbinding" if > > >> there are fences involved. > > >> > > >> And the fact that it's happening in an async > worker seem to > imply > > it's not > > >> immediate. > > >> > > > > Ok, will fix. > > This was added because in earlier design binding was > deferred > until > > next execbuff. > > But now it is non-deferred (immediate in that sense). > But yah, > this is > > confusing > > and will fix it. > > > > >> > > >> I have a question on the behavior of the bind > operation when > no > > input fence > > >> is provided. Let say I do : > > >> > > >> VM_BIND (out_fence=fence1) > > >> > > >> VM_BIND (out_fence=fence2) > > >> > > >> VM_BIND (out_fence=fence3) > > >> > > >> > > >> In what order are the fences going to be signaled? > > >> > > >> In the order of VM_BIND ioctls? Or out of order? > > >> > > >> Because you wrote "serialized I assume it's : in order > > >> > > > > Yes, in the order of VM_BIND/UNBIND ioctls. Note that > bind and > unbind > > will use > > the same queue and hence are ordered. > > > > >> > > >> One thing I didn't realize is that because we only get > one > > "VM_BIND" engine, > > >> there is a disconnect from the Vulkan specification. > > >> > > >> In Vulkan VM_BIND operations are serialized but > per engine. > > >> > > >> So you could have something like this : > > >> > > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > > >> > > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > > >> > > >> > > >> fence1 is not signaled > > >> > > >> fence3 is signaled > > >> > > >> So the second VM_BIND will proceed before the > first VM_BIND. > > >> > > >> > > >> I guess we can deal with that scenario in > userspace by doing > the > > wait > > >> ourselves in one thread per engines. > > >> > > >> But then it makes the VM_BIND input fences useless. > > >> > > >> > > >> Daniel : what do you think? Should be rework this or just > deal with > > wait > > >> fences in userspace? > > >> > > > > > >My opinion is rework this but make the ordering via > an engine > param > > optional. > > > > > >e.g. A VM can be configured so all binds are ordered > within the > >>
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Wed, Jun 08, 2022 at 08:33:25AM +0100, Tvrtko Ursulin wrote: On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana Vishwanathapura wrote: On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason Ekstrand wrote: On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura wrote: On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin wrote: > On 02/06/2022 23:35, Jason Ekstrand wrote: > > On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura > wrote: > > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > >> > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding > the mapping in an > >> > +async worker. The binding and unbinding will work like a special > GPU engine. > >> > +The binding and unbinding operations are serialized and will > wait on specified > >> > +input fences before the operation and will signal the output > fences upon the > >> > +completion of the operation. Due to serialization, completion of > an operation > >> > +will also indicate that all previous operations are also > complete. > >> > >> I guess we should avoid saying "will immediately start > binding/unbinding" if > >> there are fences involved. > >> > >> And the fact that it's happening in an async worker seem to imply > it's not > >> immediate. > >> > > Ok, will fix. > This was added because in earlier design binding was deferred until > next execbuff. > But now it is non-deferred (immediate in that sense). But yah, this is > confusing > and will fix it. > > >> > >> I have a question on the behavior of the bind operation when no > input fence > >> is provided. Let say I do : > >> > >> VM_BIND (out_fence=fence1) > >> > >> VM_BIND (out_fence=fence2) > >> > >> VM_BIND (out_fence=fence3) > >> > >> > >> In what order are the fences going to be signaled? > >> > >> In the order of VM_BIND ioctls? Or out of order? > >> > >> Because you wrote "serialized I assume it's : in order > >> > > Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and unbind > will use > the same queue and hence are ordered. > > >> > >> One thing I didn't realize is that because we only get one > "VM_BIND" engine, > >> there is a disconnect from the Vulkan specification. > >> > >> In Vulkan VM_BIND operations are serialized but per engine. > >> > >> So you could have something like this : > >> > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > >> > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > >> > >> > >> fence1 is not signaled > >> > >> fence3 is signaled > >> > >> So the second VM_BIND will proceed before the first VM_BIND. > >> > >> > >> I guess we can deal with that scenario in userspace by doing the > wait > >> ourselves in one thread per engines. > >> > >> But then it makes the VM_BIND input fences useless. > >> > >> > >> Daniel : what do you think? Should be rework this or just deal with > wait > >> fences in userspace? > >> > > > >My opinion is rework this but make the ordering via an engine param > optional. > > > >e.g. A VM can be configured so all binds are ordered within the VM > > > >e.g. A VM can be configured so all binds accept an engine argument > (in > >the case of the i915 likely this is a gem context handle) and binds > >ordered with respect to that engine. > > > >This gives UMDs options as the later likely consumes more KMD > resources > >so if a different UMD can live with binds being ordered within the VM > >they can use a mode consuming less resources. > > > > I think we need to be careful here if we are looking for some out of > (submission) order completion of vm_bind/unbind. > In-order completion means, in a batch of binds and unbinds to be > completed in-order, user only needs to specify in-fence for the > first bind/unbind call and the our-fen
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On 07/06/2022 22:32, Niranjana Vishwanathapura wrote: On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana Vishwanathapura wrote: On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason Ekstrand wrote: On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura wrote: On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin wrote: > On 02/06/2022 23:35, Jason Ekstrand wrote: > > On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura > wrote: > > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > >> > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding > the mapping in an > >> > +async worker. The binding and unbinding will work like a special > GPU engine. > >> > +The binding and unbinding operations are serialized and will > wait on specified > >> > +input fences before the operation and will signal the output > fences upon the > >> > +completion of the operation. Due to serialization, completion of > an operation > >> > +will also indicate that all previous operations are also > complete. > >> > >> I guess we should avoid saying "will immediately start > binding/unbinding" if > >> there are fences involved. > >> > >> And the fact that it's happening in an async worker seem to imply > it's not > >> immediate. > >> > > Ok, will fix. > This was added because in earlier design binding was deferred until > next execbuff. > But now it is non-deferred (immediate in that sense). But yah, this is > confusing > and will fix it. > > >> > >> I have a question on the behavior of the bind operation when no > input fence > >> is provided. Let say I do : > >> > >> VM_BIND (out_fence=fence1) > >> > >> VM_BIND (out_fence=fence2) > >> > >> VM_BIND (out_fence=fence3) > >> > >> > >> In what order are the fences going to be signaled? > >> > >> In the order of VM_BIND ioctls? Or out of order? > >> > >> Because you wrote "serialized I assume it's : in order > >> > > Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and unbind > will use > the same queue and hence are ordered. > > >> > >> One thing I didn't realize is that because we only get one > "VM_BIND" engine, > >> there is a disconnect from the Vulkan specification. > >> > >> In Vulkan VM_BIND operations are serialized but per engine. > >> > >> So you could have something like this : > >> > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > >> > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > >> > >> > >> fence1 is not signaled > >> > >> fence3 is signaled > >> > >> So the second VM_BIND will proceed before the first VM_BIND. > >> > >> > >> I guess we can deal with that scenario in userspace by doing the > wait > >> ourselves in one thread per engines. > >> > >> But then it makes the VM_BIND input fences useless. > >> > >> > >> Daniel : what do you think? Should be rework this or just deal with > wait > >> fences in userspace? > >> > > > >My opinion is rework this but make the ordering via an engine param > optional. > > > >e.g. A VM can be configured so all binds are ordered within the VM > > > >e.g. A VM can be configured so all binds accept an engine argument > (in > >the case of the i915 likely this is a gem context handle) and binds > >ordered with respect to that engine. > > > >This gives UMDs options as the later likely consumes more KMD > resources > >so if a different UMD can live with binds being ordered within the VM > >they can use a mode consuming less resources. > > > > I think we need to be careful here if we are looking for some out of > (submission) order completion of vm_bind/unbind. > In-order completion means, in a batch of binds and unbinds to be > completed in-order, user only needs to specify in-fence for the > first bind/unbind call and the our-fence for the last bind/unbind > call. Also, the VA rel
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Tue, Jun 07, 2022 at 11:18:11AM -0700, Niranjana Vishwanathapura wrote: On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason Ekstrand wrote: On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura wrote: On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin wrote: > On 02/06/2022 23:35, Jason Ekstrand wrote: > > On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura > wrote: > > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > >> > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding > the mapping in an > >> > +async worker. The binding and unbinding will work like a special > GPU engine. > >> > +The binding and unbinding operations are serialized and will > wait on specified > >> > +input fences before the operation and will signal the output > fences upon the > >> > +completion of the operation. Due to serialization, completion of > an operation > >> > +will also indicate that all previous operations are also > complete. > >> > >> I guess we should avoid saying "will immediately start > binding/unbinding" if > >> there are fences involved. > >> > >> And the fact that it's happening in an async worker seem to imply > it's not > >> immediate. > >> > > Ok, will fix. > This was added because in earlier design binding was deferred until > next execbuff. > But now it is non-deferred (immediate in that sense). But yah, this is > confusing > and will fix it. > > >> > >> I have a question on the behavior of the bind operation when no > input fence > >> is provided. Let say I do : > >> > >> VM_BIND (out_fence=fence1) > >> > >> VM_BIND (out_fence=fence2) > >> > >> VM_BIND (out_fence=fence3) > >> > >> > >> In what order are the fences going to be signaled? > >> > >> In the order of VM_BIND ioctls? Or out of order? > >> > >> Because you wrote "serialized I assume it's : in order > >> > > Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and unbind > will use > the same queue and hence are ordered. > > >> > >> One thing I didn't realize is that because we only get one > "VM_BIND" engine, > >> there is a disconnect from the Vulkan specification. > >> > >> In Vulkan VM_BIND operations are serialized but per engine. > >> > >> So you could have something like this : > >> > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > >> > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > >> > >> > >> fence1 is not signaled > >> > >> fence3 is signaled > >> > >> So the second VM_BIND will proceed before the first VM_BIND. > >> > >> > >> I guess we can deal with that scenario in userspace by doing the > wait > >> ourselves in one thread per engines. > >> > >> But then it makes the VM_BIND input fences useless. > >> > >> > >> Daniel : what do you think? Should be rework this or just deal with > wait > >> fences in userspace? > >> > > > >My opinion is rework this but make the ordering via an engine param > optional. > > > >e.g. A VM can be configured so all binds are ordered within the VM > > > >e.g. A VM can be configured so all binds accept an engine argument > (in > >the case of the i915 likely this is a gem context handle) and binds > >ordered with respect to that engine. > > > >This gives UMDs options as the later likely consumes more KMD > resources > >so if a different UMD can live with binds being ordered within the VM > >they can use a mode consuming less resources. > > > > I think we need to be careful here if we are looking for some out of > (submission) order completion of vm_bind/unbind. > In-order completion means, in a batch of binds and unbinds to be > completed in-order, user only needs to specify in-fence for the > first bind/unbind call and the our-fence for the last bind/unbind > call. Also, the VA released by an unbind call can be re-used by > any subsequent bind call in that in-or
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Tue, Jun 07, 2022 at 12:12:03PM -0500, Jason Ekstrand wrote: On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura wrote: On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin wrote: > On 02/06/2022 23:35, Jason Ekstrand wrote: > > On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura > wrote: > > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > >> > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding > the mapping in an > >> > +async worker. The binding and unbinding will work like a special > GPU engine. > >> > +The binding and unbinding operations are serialized and will > wait on specified > >> > +input fences before the operation and will signal the output > fences upon the > >> > +completion of the operation. Due to serialization, completion of > an operation > >> > +will also indicate that all previous operations are also > complete. > >> > >> I guess we should avoid saying "will immediately start > binding/unbinding" if > >> there are fences involved. > >> > >> And the fact that it's happening in an async worker seem to imply > it's not > >> immediate. > >> > > Ok, will fix. > This was added because in earlier design binding was deferred until > next execbuff. > But now it is non-deferred (immediate in that sense). But yah, this is > confusing > and will fix it. > > >> > >> I have a question on the behavior of the bind operation when no > input fence > >> is provided. Let say I do : > >> > >> VM_BIND (out_fence=fence1) > >> > >> VM_BIND (out_fence=fence2) > >> > >> VM_BIND (out_fence=fence3) > >> > >> > >> In what order are the fences going to be signaled? > >> > >> In the order of VM_BIND ioctls? Or out of order? > >> > >> Because you wrote "serialized I assume it's : in order > >> > > Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and unbind > will use > the same queue and hence are ordered. > > >> > >> One thing I didn't realize is that because we only get one > "VM_BIND" engine, > >> there is a disconnect from the Vulkan specification. > >> > >> In Vulkan VM_BIND operations are serialized but per engine. > >> > >> So you could have something like this : > >> > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > >> > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > >> > >> > >> fence1 is not signaled > >> > >> fence3 is signaled > >> > >> So the second VM_BIND will proceed before the first VM_BIND. > >> > >> > >> I guess we can deal with that scenario in userspace by doing the > wait > >> ourselves in one thread per engines. > >> > >> But then it makes the VM_BIND input fences useless. > >> > >> > >> Daniel : what do you think? Should be rework this or just deal with > wait > >> fences in userspace? > >> > > > >My opinion is rework this but make the ordering via an engine param > optional. > > > >e.g. A VM can be configured so all binds are ordered within the VM > > > >e.g. A VM can be configured so all binds accept an engine argument > (in > >the case of the i915 likely this is a gem context handle) and binds > >ordered with respect to that engine. > > > >This gives UMDs options as the later likely consumes more KMD > resources > >so if a different UMD can live with binds being ordered within the VM > >they can use a mode consuming less resources. > > > > I think we need to be careful here if we are looking for some out of > (submission) order completion of vm_bind/unbind. > In-order completion means, in a batch of binds and unbinds to be > completed in-order, user only needs to specify in-fence for the > first bind/unbind call and the our-fence for the last bind/unbind > call. Also, the VA released by an unbind call can b
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Fri, Jun 3, 2022 at 6:52 PM Niranjana Vishwanathapura < niranjana.vishwanathap...@intel.com> wrote: > On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin wrote: > > On 02/06/2022 23:35, Jason Ekstrand wrote: > > > > On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura > > wrote: > > > > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: > > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > > >> > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding > > the mapping in an > > >> > +async worker. The binding and unbinding will work like a > special > > GPU engine. > > >> > +The binding and unbinding operations are serialized and will > > wait on specified > > >> > +input fences before the operation and will signal the output > > fences upon the > > >> > +completion of the operation. Due to serialization, > completion of > > an operation > > >> > +will also indicate that all previous operations are also > > complete. > > >> > > >> I guess we should avoid saying "will immediately start > > binding/unbinding" if > > >> there are fences involved. > > >> > > >> And the fact that it's happening in an async worker seem to > imply > > it's not > > >> immediate. > > >> > > > > Ok, will fix. > > This was added because in earlier design binding was deferred until > > next execbuff. > > But now it is non-deferred (immediate in that sense). But yah, > this is > > confusing > > and will fix it. > > > > >> > > >> I have a question on the behavior of the bind operation when no > > input fence > > >> is provided. Let say I do : > > >> > > >> VM_BIND (out_fence=fence1) > > >> > > >> VM_BIND (out_fence=fence2) > > >> > > >> VM_BIND (out_fence=fence3) > > >> > > >> > > >> In what order are the fences going to be signaled? > > >> > > >> In the order of VM_BIND ioctls? Or out of order? > > >> > > >> Because you wrote "serialized I assume it's : in order > > >> > > > > Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and > unbind > > will use > > the same queue and hence are ordered. > > > > >> > > >> One thing I didn't realize is that because we only get one > > "VM_BIND" engine, > > >> there is a disconnect from the Vulkan specification. > > >> > > >> In Vulkan VM_BIND operations are serialized but per engine. > > >> > > >> So you could have something like this : > > >> > > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > > >> > > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > > >> > > >> > > >> fence1 is not signaled > > >> > > >> fence3 is signaled > > >> > > >> So the second VM_BIND will proceed before the first VM_BIND. > > >> > > >> > > >> I guess we can deal with that scenario in userspace by doing the > > wait > > >> ourselves in one thread per engines. > > >> > > >> But then it makes the VM_BIND input fences useless. > > >> > > >> > > >> Daniel : what do you think? Should be rework this or just deal > with > > wait > > >> fences in userspace? > > >> > > > > > >My opinion is rework this but make the ordering via an engine > param > > optional. > > > > > >e.g. A VM can be configured so all binds are ordered within the VM > > > > > >e.g. A VM can be configured so all binds accept an engine argument > > (in > > >the case of the i915 likely this is a gem context handle) and > binds > > >ordered with respect to that engine. > > > > > >This gives UMDs options as the later likely consumes more KMD > > resources > > >so if a different UMD can live with binds being ordered within > the VM > > >they can use a mode consuming less resources. > > > > > > > I think we need to be careful here if we are looking for some out > of > > (submission) order completion of vm_bind/unbind. > > In-order completion means, in a batch of binds and unbinds to be > > completed in-order, user only needs to specify in-fence for the > > first bind/unbind call and the our-fence for the last bind/unbind > > call. Also, the VA released by an unbind call can be re-used by > > any subsequent bind call in that in-order batch. > > > > These things will break if binding/unbinding were to be allowed to > > go out of order (of submission) and user need to be extra careful > > not to run into pre-mature triggereing of out-fence and bind > failing > > as V
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Fri, Jun 03, 2022 at 10:20:25AM +0300, Lionel Landwerlin wrote: On 02/06/2022 23:35, Jason Ekstrand wrote: On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura wrote: On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: >> > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding the mapping in an >> > +async worker. The binding and unbinding will work like a special GPU engine. >> > +The binding and unbinding operations are serialized and will wait on specified >> > +input fences before the operation and will signal the output fences upon the >> > +completion of the operation. Due to serialization, completion of an operation >> > +will also indicate that all previous operations are also complete. >> >> I guess we should avoid saying "will immediately start binding/unbinding" if >> there are fences involved. >> >> And the fact that it's happening in an async worker seem to imply it's not >> immediate. >> Ok, will fix. This was added because in earlier design binding was deferred until next execbuff. But now it is non-deferred (immediate in that sense). But yah, this is confusing and will fix it. >> >> I have a question on the behavior of the bind operation when no input fence >> is provided. Let say I do : >> >> VM_BIND (out_fence=fence1) >> >> VM_BIND (out_fence=fence2) >> >> VM_BIND (out_fence=fence3) >> >> >> In what order are the fences going to be signaled? >> >> In the order of VM_BIND ioctls? Or out of order? >> >> Because you wrote "serialized I assume it's : in order >> Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and unbind will use the same queue and hence are ordered. >> >> One thing I didn't realize is that because we only get one "VM_BIND" engine, >> there is a disconnect from the Vulkan specification. >> >> In Vulkan VM_BIND operations are serialized but per engine. >> >> So you could have something like this : >> >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) >> >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) >> >> >> fence1 is not signaled >> >> fence3 is signaled >> >> So the second VM_BIND will proceed before the first VM_BIND. >> >> >> I guess we can deal with that scenario in userspace by doing the wait >> ourselves in one thread per engines. >> >> But then it makes the VM_BIND input fences useless. >> >> >> Daniel : what do you think? Should be rework this or just deal with wait >> fences in userspace? >> > >My opinion is rework this but make the ordering via an engine param optional. > >e.g. A VM can be configured so all binds are ordered within the VM > >e.g. A VM can be configured so all binds accept an engine argument (in >the case of the i915 likely this is a gem context handle) and binds >ordered with respect to that engine. > >This gives UMDs options as the later likely consumes more KMD resources >so if a different UMD can live with binds being ordered within the VM >they can use a mode consuming less resources. > I think we need to be careful here if we are looking for some out of (submission) order completion of vm_bind/unbind. In-order completion means, in a batch of binds and unbinds to be completed in-order, user only needs to specify in-fence for the first bind/unbind call and the our-fence for the last bind/unbind call. Also, the VA released by an unbind call can be re-used by any subsequent bind call in that in-order batch. These things will break if binding/unbinding were to be allowed to go out of order (of submission) and user need to be extra careful not to run into pre-mature triggereing of out-fence and bind failing as VA is still in use etc. Also, VM_BIND binds the provided mapping on the specified address space (VM). So, the uapi is not engine/context specific. We can however add a 'queue' to the uapi which can be one from the pre-defined queues, I915_VM_BIND_QUEUE_0 I915_VM_BIND_QUEUE_1 ... I915_VM_BIND_QUEUE_(N-1) KMD will spawn an async work queue for each queue which will only bind the mappings on that queue in the order of submission. User can assign the queue to per engine or anything like that. But again here, user need t
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On 02/06/2022 23:35, Jason Ekstrand wrote: On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura wrote: On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: >> > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding the mapping in an >> > +async worker. The binding and unbinding will work like a special GPU engine. >> > +The binding and unbinding operations are serialized and will wait on specified >> > +input fences before the operation and will signal the output fences upon the >> > +completion of the operation. Due to serialization, completion of an operation >> > +will also indicate that all previous operations are also complete. >> >> I guess we should avoid saying "will immediately start binding/unbinding" if >> there are fences involved. >> >> And the fact that it's happening in an async worker seem to imply it's not >> immediate. >> Ok, will fix. This was added because in earlier design binding was deferred until next execbuff. But now it is non-deferred (immediate in that sense). But yah, this is confusing and will fix it. >> >> I have a question on the behavior of the bind operation when no input fence >> is provided. Let say I do : >> >> VM_BIND (out_fence=fence1) >> >> VM_BIND (out_fence=fence2) >> >> VM_BIND (out_fence=fence3) >> >> >> In what order are the fences going to be signaled? >> >> In the order of VM_BIND ioctls? Or out of order? >> >> Because you wrote "serialized I assume it's : in order >> Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and unbind will use the same queue and hence are ordered. >> >> One thing I didn't realize is that because we only get one "VM_BIND" engine, >> there is a disconnect from the Vulkan specification. >> >> In Vulkan VM_BIND operations are serialized but per engine. >> >> So you could have something like this : >> >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) >> >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) >> >> >> fence1 is not signaled >> >> fence3 is signaled >> >> So the second VM_BIND will proceed before the first VM_BIND. >> >> >> I guess we can deal with that scenario in userspace by doing the wait >> ourselves in one thread per engines. >> >> But then it makes the VM_BIND input fences useless. >> >> >> Daniel : what do you think? Should be rework this or just deal with wait >> fences in userspace? >> > >My opinion is rework this but make the ordering via an engine param optional. > >e.g. A VM can be configured so all binds are ordered within the VM > >e.g. A VM can be configured so all binds accept an engine argument (in >the case of the i915 likely this is a gem context handle) and binds >ordered with respect to that engine. > >This gives UMDs options as the later likely consumes more KMD resources >so if a different UMD can live with binds being ordered within the VM >they can use a mode consuming less resources. > I think we need to be careful here if we are looking for some out of (submission) order completion of vm_bind/unbind. In-order completion means, in a batch of binds and unbinds to be completed in-order, user only needs to specify in-fence for the first bind/unbind call and the our-fence for the last bind/unbind call. Also, the VA released by an unbind call can be re-used by any subsequent bind call in that in-order batch. These things will break if binding/unbinding were to be allowed to go out of order (of submission) and user need to be extra careful not to run into pre-mature triggereing of out-fence and bind failing as VA is still in use etc. Also, VM_BIND binds the provided mapping on the specified address space (VM). So, the uapi is not engine/context specific. We can however add a 'queue' to the uapi which can be one from the pre-defined queues, I915_VM_BIND_QUEUE_0 I915_VM_BIND_QUEUE_1 ... I915_VM_BIND_QUEUE_(N-1) KMD will spawn an async work queue for each queue which will only bind the mappings on that queue in the order of submission. User can assign the queue to per engine or anything like that. But again here, user need to be careful and not deadlock these queues with circular dependency of fences. I prefer adding this later an as extension based on whether it is really helping with the implementation. I can tell you right now that having everything on a single in-order queue will not get us the perf we want. What vulkan r
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Thu, Jun 2, 2022 at 3:11 PM Niranjana Vishwanathapura < niranjana.vishwanathap...@intel.com> wrote: > On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: > >On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > >> > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding the > mapping in an > >> > +async worker. The binding and unbinding will work like a special GPU > engine. > >> > +The binding and unbinding operations are serialized and will wait on > specified > >> > +input fences before the operation and will signal the output fences > upon the > >> > +completion of the operation. Due to serialization, completion of an > operation > >> > +will also indicate that all previous operations are also complete. > >> > >> I guess we should avoid saying "will immediately start > binding/unbinding" if > >> there are fences involved. > >> > >> And the fact that it's happening in an async worker seem to imply it's > not > >> immediate. > >> > > Ok, will fix. > This was added because in earlier design binding was deferred until next > execbuff. > But now it is non-deferred (immediate in that sense). But yah, this is > confusing > and will fix it. > > >> > >> I have a question on the behavior of the bind operation when no input > fence > >> is provided. Let say I do : > >> > >> VM_BIND (out_fence=fence1) > >> > >> VM_BIND (out_fence=fence2) > >> > >> VM_BIND (out_fence=fence3) > >> > >> > >> In what order are the fences going to be signaled? > >> > >> In the order of VM_BIND ioctls? Or out of order? > >> > >> Because you wrote "serialized I assume it's : in order > >> > > Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and unbind will > use > the same queue and hence are ordered. > > >> > >> One thing I didn't realize is that because we only get one "VM_BIND" > engine, > >> there is a disconnect from the Vulkan specification. > >> > >> In Vulkan VM_BIND operations are serialized but per engine. > >> > >> So you could have something like this : > >> > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > >> > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > >> > >> > >> fence1 is not signaled > >> > >> fence3 is signaled > >> > >> So the second VM_BIND will proceed before the first VM_BIND. > >> > >> > >> I guess we can deal with that scenario in userspace by doing the wait > >> ourselves in one thread per engines. > >> > >> But then it makes the VM_BIND input fences useless. > >> > >> > >> Daniel : what do you think? Should be rework this or just deal with wait > >> fences in userspace? > >> > > > >My opinion is rework this but make the ordering via an engine param > optional. > > > >e.g. A VM can be configured so all binds are ordered within the VM > > > >e.g. A VM can be configured so all binds accept an engine argument (in > >the case of the i915 likely this is a gem context handle) and binds > >ordered with respect to that engine. > > > >This gives UMDs options as the later likely consumes more KMD resources > >so if a different UMD can live with binds being ordered within the VM > >they can use a mode consuming less resources. > > > > I think we need to be careful here if we are looking for some out of > (submission) order completion of vm_bind/unbind. > In-order completion means, in a batch of binds and unbinds to be > completed in-order, user only needs to specify in-fence for the > first bind/unbind call and the our-fence for the last bind/unbind > call. Also, the VA released by an unbind call can be re-used by > any subsequent bind call in that in-order batch. > > These things will break if binding/unbinding were to be allowed to > go out of order (of submission) and user need to be extra careful > not to run into pre-mature triggereing of out-fence and bind failing > as VA is still in use etc. > > Also, VM_BIND binds the provided mapping on the specified address space > (VM). So, the uapi is not engine/context specific. > > We can however add a 'queue' to the uapi which can be one from the > pre-defined queues, > I915_VM_BIND_QUEUE_0 > I915_VM_BIND_QUEUE_1 > ... > I915_VM_BIND_QUEUE_(N-1) > > KMD will spawn an async work queue for each queue which will only > bind the mappings on that queue in the order of submission. > User can assign the queue to per engine or anything like that. > > But again here, user need to be careful and not deadlock these > queues with circular dependency of fences. > > I prefer adding this later an as extension based on whether it > is really helping with the implementation. > I can tell you right now that having everything on a single in-order queue will not get us the perf we want. What vulkan really wants is one of two things: 1. No implicit ordering of VM_BIND ops. They just happen in whatever their dependencies are resolved and we ensure ordering ourselves by having a syncobj in the VkQueue. 2. The ability to create multiple VM_BIND queues.
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Thu, Jun 02, 2022 at 09:22:46AM -0700, Matthew Brost wrote: On Thu, Jun 02, 2022 at 08:42:13AM +0300, Lionel Landwerlin wrote: On 02/06/2022 00:18, Matthew Brost wrote: > On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > > On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > > > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding the mapping in an > > > +async worker. The binding and unbinding will work like a special GPU engine. > > > +The binding and unbinding operations are serialized and will wait on specified > > > +input fences before the operation and will signal the output fences upon the > > > +completion of the operation. Due to serialization, completion of an operation > > > +will also indicate that all previous operations are also complete. > > I guess we should avoid saying "will immediately start binding/unbinding" if > > there are fences involved. > > > > And the fact that it's happening in an async worker seem to imply it's not > > immediate. > > > > > > I have a question on the behavior of the bind operation when no input fence > > is provided. Let say I do : > > > > VM_BIND (out_fence=fence1) > > > > VM_BIND (out_fence=fence2) > > > > VM_BIND (out_fence=fence3) > > > > > > In what order are the fences going to be signaled? > > > > In the order of VM_BIND ioctls? Or out of order? > > > > Because you wrote "serialized I assume it's : in order > > > > > > One thing I didn't realize is that because we only get one "VM_BIND" engine, > > there is a disconnect from the Vulkan specification. > > > > In Vulkan VM_BIND operations are serialized but per engine. > > > > So you could have something like this : > > > > VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > > > > VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > > > Question - let's say this done after the above operations: > > EXEC (engine=ccs0, in_fence=NULL, out_fence=NULL) > > Is the exec ordered with respected to bind (i.e. would fence3 & 4 be > signaled before the exec starts)? > > Matt Hi Matt, From the vulkan point of view, everything is serialized within an engine (we map that to a VkQueue). So with : EXEC (engine=ccs0, in_fence=NULL, out_fence=NULL) VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) EXEC completes first then VM_BIND executes. To be even clearer : EXEC (engine=ccs0, in_fence=fence2, out_fence=NULL) VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) EXEC will wait until fence2 is signaled. Once fence2 is signaled, EXEC proceeds, finishes and only after it is done, VM_BIND executes. It would kind of like having the VM_BIND operation be another batch executed from the ringbuffer buffer. Yea this makes sense. I think of VM_BINDs as more or less just another version of an EXEC and this fits with that. Note that VM_BIND itself can bind while and EXEC (GPU job) is running. (Say, getting binds ready for next submission). It is up to user though, how to use it. In practice I don't think we can share a ring but we should be able to present an engine (again likely a gem context in i915) to the user that orders VM_BINDs / EXECs if that is what Vulkan expects, at least I think. I have responded in the other thread on this. Niranjana Hopefully Niranjana + Daniel agree. Matt -Lionel > > > fence1 is not signaled > > > > fence3 is signaled > > > > So the second VM_BIND will proceed before the first VM_BIND. > > > > > > I guess we can deal with that scenario in userspace by doing the wait > > ourselves in one thread per engines. > > > > But then it makes the VM_BIND input fences useless. > > > > > > Daniel : what do you think? Should be rework this or just deal with wait > > fences in userspace? > > > > > > Sorry I noticed this late. > > > > > > -Lionel > > > >
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Thu, Jun 2, 2022 at 7:42 AM Lionel Landwerlin wrote: > > On 02/06/2022 00:18, Matthew Brost wrote: > > On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > >> On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > >>> +VM_BIND/UNBIND ioctl will immediately start binding/unbinding the > >>> mapping in an > >>> +async worker. The binding and unbinding will work like a special GPU > >>> engine. > >>> +The binding and unbinding operations are serialized and will wait on > >>> specified > >>> +input fences before the operation and will signal the output fences upon > >>> the > >>> +completion of the operation. Due to serialization, completion of an > >>> operation > >>> +will also indicate that all previous operations are also complete. > >> I guess we should avoid saying "will immediately start binding/unbinding" > >> if > >> there are fences involved. > >> > >> And the fact that it's happening in an async worker seem to imply it's not > >> immediate. > >> > >> > >> I have a question on the behavior of the bind operation when no input fence > >> is provided. Let say I do : > >> > >> VM_BIND (out_fence=fence1) > >> > >> VM_BIND (out_fence=fence2) > >> > >> VM_BIND (out_fence=fence3) > >> > >> > >> In what order are the fences going to be signaled? > >> > >> In the order of VM_BIND ioctls? Or out of order? > >> > >> Because you wrote "serialized I assume it's : in order > >> > >> > >> One thing I didn't realize is that because we only get one "VM_BIND" > >> engine, > >> there is a disconnect from the Vulkan specification. Note that in Vulkan not every queue has to support sparse binding, so one could consider a dedicated sparse binding only queue family. > >> > >> In Vulkan VM_BIND operations are serialized but per engine. > >> > >> So you could have something like this : > >> > >> VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > >> > >> VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > >> > > Question - let's say this done after the above operations: > > > > EXEC (engine=ccs0, in_fence=NULL, out_fence=NULL) > > > > Is the exec ordered with respected to bind (i.e. would fence3 & 4 be > > signaled before the exec starts)? > > > > Matt > > > Hi Matt, > > From the vulkan point of view, everything is serialized within an > engine (we map that to a VkQueue). > > So with : > > EXEC (engine=ccs0, in_fence=NULL, out_fence=NULL) > VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > > EXEC completes first then VM_BIND executes. > > > To be even clearer : > > EXEC (engine=ccs0, in_fence=fence2, out_fence=NULL) > VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > > > EXEC will wait until fence2 is signaled. > Once fence2 is signaled, EXEC proceeds, finishes and only after it is done, > VM_BIND executes. > > It would kind of like having the VM_BIND operation be another batch executed > from the ringbuffer buffer. > > -Lionel > > > > > >> fence1 is not signaled > >> > >> fence3 is signaled > >> > >> So the second VM_BIND will proceed before the first VM_BIND. > >> > >> > >> I guess we can deal with that scenario in userspace by doing the wait > >> ourselves in one thread per engines. > >> > >> But then it makes the VM_BIND input fences useless. I posed the same question on my series for AMD (https://patchwork.freedesktop.org/series/104578/), albeit for slightly different reasons.: if one creates a new VkMemory object, you generally want that mapped ASAP, as you can't track (in a VK_KHR_descriptor_indexing world) whether the next submit is going to use this VkMemory object and hence have to assume the worst (i.e. wait till the map/bind is complete before executing the next submission). If all binds/unbinds (or maps/unmaps) happen in-order that means an operation with input fences could delay stuff we want ASAP. Of course waiting in userspace does have disadvantages: 1) more overhead between fence signalling and the operation, potentially causing slightly bigger GPU bubbles. 2) You can't get an out fence early. Within the driver we can mostly work around this but sync_fd exports, WSI and such will be messy. 3) moving the queue to a thread might make things slightly less ideal due to scheduling delays. Removing the in-order working in the kernel generally seems like madness to me as it is very hard to keep track of the state of the virtual address space (to e.g. track umapping stuff before freeing memory or moving memory around) the one game I tried (FH5 over vkd3d-proton) does sparse mapping as follows: separate queue: 1) 0 cmdbuffer submit with 0 input semaphores and 1 output semaphore 2) sparse bind with input semaphore from 1 and 1 output semaphore 3) 0 cmdbuffer submit with input semaphore from 2 and 1 output fence 4) wait on that fence on the CPU which works very well if we just wait for the sparse bind input semaphore in userspace, but I'm still working on seeing if this is the common usecase or an outlier. > >> > >> > >> Daniel : what do you thi
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Wed, Jun 01, 2022 at 01:28:36PM -0700, Matthew Brost wrote: On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding the mapping in an > +async worker. The binding and unbinding will work like a special GPU engine. > +The binding and unbinding operations are serialized and will wait on specified > +input fences before the operation and will signal the output fences upon the > +completion of the operation. Due to serialization, completion of an operation > +will also indicate that all previous operations are also complete. I guess we should avoid saying "will immediately start binding/unbinding" if there are fences involved. And the fact that it's happening in an async worker seem to imply it's not immediate. Ok, will fix. This was added because in earlier design binding was deferred until next execbuff. But now it is non-deferred (immediate in that sense). But yah, this is confusing and will fix it. I have a question on the behavior of the bind operation when no input fence is provided. Let say I do : VM_BIND (out_fence=fence1) VM_BIND (out_fence=fence2) VM_BIND (out_fence=fence3) In what order are the fences going to be signaled? In the order of VM_BIND ioctls? Or out of order? Because you wrote "serialized I assume it's : in order Yes, in the order of VM_BIND/UNBIND ioctls. Note that bind and unbind will use the same queue and hence are ordered. One thing I didn't realize is that because we only get one "VM_BIND" engine, there is a disconnect from the Vulkan specification. In Vulkan VM_BIND operations are serialized but per engine. So you could have something like this : VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) fence1 is not signaled fence3 is signaled So the second VM_BIND will proceed before the first VM_BIND. I guess we can deal with that scenario in userspace by doing the wait ourselves in one thread per engines. But then it makes the VM_BIND input fences useless. Daniel : what do you think? Should be rework this or just deal with wait fences in userspace? My opinion is rework this but make the ordering via an engine param optional. e.g. A VM can be configured so all binds are ordered within the VM e.g. A VM can be configured so all binds accept an engine argument (in the case of the i915 likely this is a gem context handle) and binds ordered with respect to that engine. This gives UMDs options as the later likely consumes more KMD resources so if a different UMD can live with binds being ordered within the VM they can use a mode consuming less resources. I think we need to be careful here if we are looking for some out of (submission) order completion of vm_bind/unbind. In-order completion means, in a batch of binds and unbinds to be completed in-order, user only needs to specify in-fence for the first bind/unbind call and the our-fence for the last bind/unbind call. Also, the VA released by an unbind call can be re-used by any subsequent bind call in that in-order batch. These things will break if binding/unbinding were to be allowed to go out of order (of submission) and user need to be extra careful not to run into pre-mature triggereing of out-fence and bind failing as VA is still in use etc. Also, VM_BIND binds the provided mapping on the specified address space (VM). So, the uapi is not engine/context specific. We can however add a 'queue' to the uapi which can be one from the pre-defined queues, I915_VM_BIND_QUEUE_0 I915_VM_BIND_QUEUE_1 ... I915_VM_BIND_QUEUE_(N-1) KMD will spawn an async work queue for each queue which will only bind the mappings on that queue in the order of submission. User can assign the queue to per engine or anything like that. But again here, user need to be careful and not deadlock these queues with circular dependency of fences. I prefer adding this later an as extension based on whether it is really helping with the implementation. Daniel, any thoughts? Niranjana Matt Sorry I noticed this late. -Lionel
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Thu, Jun 02, 2022 at 08:42:13AM +0300, Lionel Landwerlin wrote: > On 02/06/2022 00:18, Matthew Brost wrote: > > On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > > > On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > > > > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding the > > > > mapping in an > > > > +async worker. The binding and unbinding will work like a special GPU > > > > engine. > > > > +The binding and unbinding operations are serialized and will wait on > > > > specified > > > > +input fences before the operation and will signal the output fences > > > > upon the > > > > +completion of the operation. Due to serialization, completion of an > > > > operation > > > > +will also indicate that all previous operations are also complete. > > > I guess we should avoid saying "will immediately start binding/unbinding" > > > if > > > there are fences involved. > > > > > > And the fact that it's happening in an async worker seem to imply it's not > > > immediate. > > > > > > > > > I have a question on the behavior of the bind operation when no input > > > fence > > > is provided. Let say I do : > > > > > > VM_BIND (out_fence=fence1) > > > > > > VM_BIND (out_fence=fence2) > > > > > > VM_BIND (out_fence=fence3) > > > > > > > > > In what order are the fences going to be signaled? > > > > > > In the order of VM_BIND ioctls? Or out of order? > > > > > > Because you wrote "serialized I assume it's : in order > > > > > > > > > One thing I didn't realize is that because we only get one "VM_BIND" > > > engine, > > > there is a disconnect from the Vulkan specification. > > > > > > In Vulkan VM_BIND operations are serialized but per engine. > > > > > > So you could have something like this : > > > > > > VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > > > > > > VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > > > > > Question - let's say this done after the above operations: > > > > EXEC (engine=ccs0, in_fence=NULL, out_fence=NULL) > > > > Is the exec ordered with respected to bind (i.e. would fence3 & 4 be > > signaled before the exec starts)? > > > > Matt > > > Hi Matt, > > From the vulkan point of view, everything is serialized within an engine (we > map that to a VkQueue). > > So with : > > EXEC (engine=ccs0, in_fence=NULL, out_fence=NULL) > VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > > EXEC completes first then VM_BIND executes. > > > To be even clearer : > > EXEC (engine=ccs0, in_fence=fence2, out_fence=NULL) > VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > > > EXEC will wait until fence2 is signaled. > Once fence2 is signaled, EXEC proceeds, finishes and only after it is done, > VM_BIND executes. > > It would kind of like having the VM_BIND operation be another batch executed > from the ringbuffer buffer. > Yea this makes sense. I think of VM_BINDs as more or less just another version of an EXEC and this fits with that. In practice I don't think we can share a ring but we should be able to present an engine (again likely a gem context in i915) to the user that orders VM_BINDs / EXECs if that is what Vulkan expects, at least I think. Hopefully Niranjana + Daniel agree. Matt > -Lionel > > > > > > > fence1 is not signaled > > > > > > fence3 is signaled > > > > > > So the second VM_BIND will proceed before the first VM_BIND. > > > > > > > > > I guess we can deal with that scenario in userspace by doing the wait > > > ourselves in one thread per engines. > > > > > > But then it makes the VM_BIND input fences useless. > > > > > > > > > Daniel : what do you think? Should be rework this or just deal with wait > > > fences in userspace? > > > > > > > > > Sorry I noticed this late. > > > > > > > > > -Lionel > > > > > > >
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On 02/06/2022 00:18, Matthew Brost wrote: On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: +VM_BIND/UNBIND ioctl will immediately start binding/unbinding the mapping in an +async worker. The binding and unbinding will work like a special GPU engine. +The binding and unbinding operations are serialized and will wait on specified +input fences before the operation and will signal the output fences upon the +completion of the operation. Due to serialization, completion of an operation +will also indicate that all previous operations are also complete. I guess we should avoid saying "will immediately start binding/unbinding" if there are fences involved. And the fact that it's happening in an async worker seem to imply it's not immediate. I have a question on the behavior of the bind operation when no input fence is provided. Let say I do : VM_BIND (out_fence=fence1) VM_BIND (out_fence=fence2) VM_BIND (out_fence=fence3) In what order are the fences going to be signaled? In the order of VM_BIND ioctls? Or out of order? Because you wrote "serialized I assume it's : in order One thing I didn't realize is that because we only get one "VM_BIND" engine, there is a disconnect from the Vulkan specification. In Vulkan VM_BIND operations are serialized but per engine. So you could have something like this : VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) Question - let's say this done after the above operations: EXEC (engine=ccs0, in_fence=NULL, out_fence=NULL) Is the exec ordered with respected to bind (i.e. would fence3 & 4 be signaled before the exec starts)? Matt Hi Matt, From the vulkan point of view, everything is serialized within an engine (we map that to a VkQueue). So with : EXEC (engine=ccs0, in_fence=NULL, out_fence=NULL) VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) EXEC completes first then VM_BIND executes. To be even clearer : EXEC (engine=ccs0, in_fence=fence2, out_fence=NULL) VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) EXEC will wait until fence2 is signaled. Once fence2 is signaled, EXEC proceeds, finishes and only after it is done, VM_BIND executes. It would kind of like having the VM_BIND operation be another batch executed from the ringbuffer buffer. -Lionel fence1 is not signaled fence3 is signaled So the second VM_BIND will proceed before the first VM_BIND. I guess we can deal with that scenario in userspace by doing the wait ourselves in one thread per engines. But then it makes the VM_BIND input fences useless. Daniel : what do you think? Should be rework this or just deal with wait fences in userspace? Sorry I noticed this late. -Lionel
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding the mapping > > in an > > +async worker. The binding and unbinding will work like a special GPU > > engine. > > +The binding and unbinding operations are serialized and will wait on > > specified > > +input fences before the operation and will signal the output fences upon > > the > > +completion of the operation. Due to serialization, completion of an > > operation > > +will also indicate that all previous operations are also complete. > > I guess we should avoid saying "will immediately start binding/unbinding" if > there are fences involved. > > And the fact that it's happening in an async worker seem to imply it's not > immediate. > > > I have a question on the behavior of the bind operation when no input fence > is provided. Let say I do : > > VM_BIND (out_fence=fence1) > > VM_BIND (out_fence=fence2) > > VM_BIND (out_fence=fence3) > > > In what order are the fences going to be signaled? > > In the order of VM_BIND ioctls? Or out of order? > > Because you wrote "serialized I assume it's : in order > > > One thing I didn't realize is that because we only get one "VM_BIND" engine, > there is a disconnect from the Vulkan specification. > > In Vulkan VM_BIND operations are serialized but per engine. > > So you could have something like this : > > VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > > VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > Question - let's say this done after the above operations: EXEC (engine=ccs0, in_fence=NULL, out_fence=NULL) Is the exec ordered with respected to bind (i.e. would fence3 & 4 be signaled before the exec starts)? Matt > > fence1 is not signaled > > fence3 is signaled > > So the second VM_BIND will proceed before the first VM_BIND. > > > I guess we can deal with that scenario in userspace by doing the wait > ourselves in one thread per engines. > > But then it makes the VM_BIND input fences useless. > > > Daniel : what do you think? Should be rework this or just deal with wait > fences in userspace? > > > Sorry I noticed this late. > > > -Lionel > >
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On Wed, Jun 01, 2022 at 05:25:49PM +0300, Lionel Landwerlin wrote: > On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: > > +VM_BIND/UNBIND ioctl will immediately start binding/unbinding the mapping > > in an > > +async worker. The binding and unbinding will work like a special GPU > > engine. > > +The binding and unbinding operations are serialized and will wait on > > specified > > +input fences before the operation and will signal the output fences upon > > the > > +completion of the operation. Due to serialization, completion of an > > operation > > +will also indicate that all previous operations are also complete. > > I guess we should avoid saying "will immediately start binding/unbinding" if > there are fences involved. > > And the fact that it's happening in an async worker seem to imply it's not > immediate. > > > I have a question on the behavior of the bind operation when no input fence > is provided. Let say I do : > > VM_BIND (out_fence=fence1) > > VM_BIND (out_fence=fence2) > > VM_BIND (out_fence=fence3) > > > In what order are the fences going to be signaled? > > In the order of VM_BIND ioctls? Or out of order? > > Because you wrote "serialized I assume it's : in order > > > One thing I didn't realize is that because we only get one "VM_BIND" engine, > there is a disconnect from the Vulkan specification. > > In Vulkan VM_BIND operations are serialized but per engine. > > So you could have something like this : > > VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) > > VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) > > > fence1 is not signaled > > fence3 is signaled > > So the second VM_BIND will proceed before the first VM_BIND. > > > I guess we can deal with that scenario in userspace by doing the wait > ourselves in one thread per engines. > > But then it makes the VM_BIND input fences useless. > > > Daniel : what do you think? Should be rework this or just deal with wait > fences in userspace? > My opinion is rework this but make the ordering via an engine param optional. e.g. A VM can be configured so all binds are ordered within the VM e.g. A VM can be configured so all binds accept an engine argument (in the case of the i915 likely this is a gem context handle) and binds ordered with respect to that engine. This gives UMDs options as the later likely consumes more KMD resources so if a different UMD can live with binds being ordered within the VM they can use a mode consuming less resources. Matt > > Sorry I noticed this late. > > > -Lionel > >
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On 17/05/2022 21:32, Niranjana Vishwanathapura wrote: +VM_BIND/UNBIND ioctl will immediately start binding/unbinding the mapping in an +async worker. The binding and unbinding will work like a special GPU engine. +The binding and unbinding operations are serialized and will wait on specified +input fences before the operation and will signal the output fences upon the +completion of the operation. Due to serialization, completion of an operation +will also indicate that all previous operations are also complete. I guess we should avoid saying "will immediately start binding/unbinding" if there are fences involved. And the fact that it's happening in an async worker seem to imply it's not immediate. I have a question on the behavior of the bind operation when no input fence is provided. Let say I do : VM_BIND (out_fence=fence1) VM_BIND (out_fence=fence2) VM_BIND (out_fence=fence3) In what order are the fences going to be signaled? In the order of VM_BIND ioctls? Or out of order? Because you wrote "serialized I assume it's : in order One thing I didn't realize is that because we only get one "VM_BIND" engine, there is a disconnect from the Vulkan specification. In Vulkan VM_BIND operations are serialized but per engine. So you could have something like this : VM_BIND (engine=rcs0, in_fence=fence1, out_fence=fence2) VM_BIND (engine=ccs0, in_fence=fence3, out_fence=fence4) fence1 is not signaled fence3 is signaled So the second VM_BIND will proceed before the first VM_BIND. I guess we can deal with that scenario in userspace by doing the wait ourselves in one thread per engines. But then it makes the VM_BIND input fences useless. Daniel : what do you think? Should be rework this or just deal with wait fences in userspace? Sorry I noticed this late. -Lionel
Re: [Intel-gfx] [RFC v3 1/3] drm/doc/rfc: VM_BIND feature design document
On 20/05/2022 01:52, Zanoni, Paulo R wrote: On Tue, 2022-05-17 at 11:32 -0700, Niranjana Vishwanathapura wrote: VM_BIND design document with description of intended use cases. v2: Add more documentation and format as per review comments from Daniel. Signed-off-by: Niranjana Vishwanathapura --- diff --git a/Documentation/gpu/rfc/i915_vm_bind.rst b/Documentation/gpu/rfc/i915_vm_bind.rst new file mode 100644 index ..f1be560d313c --- /dev/null +++ b/Documentation/gpu/rfc/i915_vm_bind.rst @@ -0,0 +1,304 @@ +== +I915 VM_BIND feature design and use cases +== + +VM_BIND feature + +DRM_I915_GEM_VM_BIND/UNBIND ioctls allows UMD to bind/unbind GEM buffer +objects (BOs) or sections of a BOs at specified GPU virtual addresses on a +specified address space (VM). These mappings (also referred to as persistent +mappings) will be persistent across multiple GPU submissions (execbuff calls) +issued by the UMD, without user having to provide a list of all required +mappings during each submission (as required by older execbuff mode). + +VM_BIND/UNBIND ioctls will support 'in' and 'out' fences to allow userpace +to specify how the binding/unbinding should sync with other operations +like the GPU job submission. These fences will be timeline 'drm_syncobj's +for non-Compute contexts (See struct drm_i915_vm_bind_ext_timeline_fences). +For Compute contexts, they will be user/memory fences (See struct +drm_i915_vm_bind_ext_user_fence). + +VM_BIND feature is advertised to user via I915_PARAM_HAS_VM_BIND. +User has to opt-in for VM_BIND mode of binding for an address space (VM) +during VM creation time via I915_VM_CREATE_FLAGS_USE_VM_BIND extension. + +VM_BIND/UNBIND ioctl will immediately start binding/unbinding the mapping in an +async worker. The binding and unbinding will work like a special GPU engine. +The binding and unbinding operations are serialized and will wait on specified +input fences before the operation and will signal the output fences upon the +completion of the operation. Due to serialization, completion of an operation +will also indicate that all previous operations are also complete. + +VM_BIND features include: + +* Multiple Virtual Address (VA) mappings can map to the same physical pages + of an object (aliasing). +* VA mapping can map to a partial section of the BO (partial binding). +* Support capture of persistent mappings in the dump upon GPU error. +* TLB is flushed upon unbind completion. Batching of TLB flushes in some + use cases will be helpful. +* Asynchronous vm_bind and vm_unbind support with 'in' and 'out' fences. +* Support for userptr gem objects (no special uapi is required for this). + +Execbuff ioctl in VM_BIND mode +--- +The execbuff ioctl handling in VM_BIND mode differs significantly from the +older method. A VM in VM_BIND mode will not support older execbuff mode of +binding. In VM_BIND mode, execbuff ioctl will not accept any execlist. Hence, +no support for implicit sync. It is expected that the below work will be able +to support requirements of object dependency setting in all use cases: + +"dma-buf: Add an API for exporting sync files" +(https://lwn.net/Articles/859290/) I would really like to have more details here. The link provided points to new ioctls and we're not very familiar with those yet, so I think you should really clarify the interaction between the new additions here. Having some sample code would be really nice too. For Mesa at least (and I believe for the other drivers too) we always have a few exported buffers in every execbuf call, and we rely on the implicit synchronization provided by execbuf to make sure everything works. The execbuf ioctl also has some code to flush caches during implicit synchronization AFAIR, so I would guess we rely on it too and whatever else the Kernel does. Is that covered by the new ioctls? In addition, as far as I remember, one of the big improvements of vm_bind was that it would help reduce ioctl latency and cpu overhead. But if making execbuf faster comes at the cost of requiring additional ioctls calls for implicit synchronization, which is required on ever execbuf call, then I wonder if we'll even get any faster at all. Comparing old execbuf vs plain new execbuf without the new required ioctls won't make sense. But maybe I'm wrong and we won't need to call these new ioctls around every single execbuf ioctl we submit? Again, more clarification and some code examples here would be really nice. This is a big change on an important part of the API, we should clarify the new expected usage. Hey Paulo, I think in the case of X11/Wayland, we'll be doing 1 or 2 extra ioctls per frame which seems pretty reasonable. Essentially we need to set the dependencies on the buffer we´re going to tell the display engine (gnome-shell/kde/bare-display-hw) to use. In the Vulkan case, we're t
