Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Tomasz Figa]

Hi Laurent,

On Thu, Oct 26, 2017 at 12:48 AM, Laurent Pinchart
 wrote:
> Hello,
>
> On Monday, 23 October 2017 11:45:01 EEST Alexandre Courbot wrote:
>> On Thu, Oct 19, 2017 at 11:43 PM, Sakari Ailus  wrote:
>> > On Thu, Sep 28, 2017 at 06:50:18PM +0900, Alexandre Courbot wrote:
[snip]
>> > Both request and jobs APIs have the concept of a request, or a job, which
>> > is created by the user and then different buffers or controls can be bound
>> > to that request. (Other configuration isn't really excluded but it's
>> > non-trivial to implement in practice.) This is common for both.
>>
>> Yes, the main difference being that the current proposal manages the
>> jobs flow implicitly by default, to ease the most common uses of this
>> API (codecs & camera). It still maintains the ability to control them
>> more finely similarly to the previous request API proposals.
>
> Implicit job handling might make your codec use-cases simpler, but it does
> *not* ease camera support.

I'd appreciate some example use cases for explicit job handling that
would benefit the camera.

[snip]
>> >   Also --- when an association with a video devnode file descriptor and a
>> >   job with a request is made, when does it cease to exist? When the job is
>> >   released? When the job is done?
>>
>> Association is made between a job queue (to which an undefined number
>> of jobs can be queued) and a set of device nodes. A job queue remains
>> active as long as its file descriptor is not closed. So the short
>> answer to your question is that the devnode remains part of the queue
>> until the file descriptor obtained by opening /dev/v4l2_jobqueue (and
>> initialized using VIDIOC_JOBQUEUE_INIT) is opened. This is of course
>> subject to change if /dev/v4l2_jobqueue disappears, but I would like
>> to retain the idea of managing the jobs queue via its own file
>> descriptor.
>
> This makes it quite inconvenient to change controls both through a request and
> directly, a userspace application would need to open subdevs and video nodes
> twice, keeping one "direct" fd and associating the other one with the queue.
> That's a complexity increase for userspace without any advantage in my
> opinion.

There is the .which field within v4l2_ext_controls struct, which could
be used to determine whether the operation applies to a request or
directly.

[snip]
>> > Another matter is making videobuf2 helpful here: we should have, if not in
>> > the videobuf2 framework itself, then around it helper function(s) to
>> > manage the submission of buffers to a driver. You can get things working
>> > pretty easily but the error handling is very painful: what do you do, for
>> > instance, with buffers queued with a request if queueing the request
>> > itself fails, possibly because the user hasn't provided enough buffers
>> > with the request? Mark the buffers errorneous and return them to the user?
>> > Probably so, but that requires the user to dequeue the buffers and gather
>> > the request again. I presume this would only happen in special
>> > circumstances though, and not typically in an application using requests.
>> > This, and many other special cases still must be handled by the kernel.
>>
>> Error handling is still pretty weak in that version. I would like to
>> get an overall agreement on the general direction before looking at
>> this more closely though, as I suppose getting things right will take
>> some time.
>
> I believe error handling would be much simpler if we passed all request
> parameters through a single ioctl, as we would have one clear point where to
> perform validation with all required information available.

Compared to fine granularity of particular ioctls, this would make
determining the error cause more difficult or at least would require
providing a special interface to userspace just to communicate the
exact reason of the failure.

Best regards,
Tomasz

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Tomasz Figa]

Hi Laurent,

On Thu, Oct 26, 2017 at 12:48 AM, Laurent Pinchart
 wrote:
> Hello,
>
> On Monday, 23 October 2017 11:45:01 EEST Alexandre Courbot wrote:
>> On Thu, Oct 19, 2017 at 11:43 PM, Sakari Ailus  wrote:
>> > On Thu, Sep 28, 2017 at 06:50:18PM +0900, Alexandre Courbot wrote:
[snip]
>> > Both request and jobs APIs have the concept of a request, or a job, which
>> > is created by the user and then different buffers or controls can be bound
>> > to that request. (Other configuration isn't really excluded but it's
>> > non-trivial to implement in practice.) This is common for both.
>>
>> Yes, the main difference being that the current proposal manages the
>> jobs flow implicitly by default, to ease the most common uses of this
>> API (codecs & camera). It still maintains the ability to control them
>> more finely similarly to the previous request API proposals.
>
> Implicit job handling might make your codec use-cases simpler, but it does
> *not* ease camera support.

I'd appreciate some example use cases for explicit job handling that
would benefit the camera.

[snip]
>> >   Also --- when an association with a video devnode file descriptor and a
>> >   job with a request is made, when does it cease to exist? When the job is
>> >   released? When the job is done?
>>
>> Association is made between a job queue (to which an undefined number
>> of jobs can be queued) and a set of device nodes. A job queue remains
>> active as long as its file descriptor is not closed. So the short
>> answer to your question is that the devnode remains part of the queue
>> until the file descriptor obtained by opening /dev/v4l2_jobqueue (and
>> initialized using VIDIOC_JOBQUEUE_INIT) is opened. This is of course
>> subject to change if /dev/v4l2_jobqueue disappears, but I would like
>> to retain the idea of managing the jobs queue via its own file
>> descriptor.
>
> This makes it quite inconvenient to change controls both through a request and
> directly, a userspace application would need to open subdevs and video nodes
> twice, keeping one "direct" fd and associating the other one with the queue.
> That's a complexity increase for userspace without any advantage in my
> opinion.

There is the .which field within v4l2_ext_controls struct, which could
be used to determine whether the operation applies to a request or
directly.

[snip]
>> > Another matter is making videobuf2 helpful here: we should have, if not in
>> > the videobuf2 framework itself, then around it helper function(s) to
>> > manage the submission of buffers to a driver. You can get things working
>> > pretty easily but the error handling is very painful: what do you do, for
>> > instance, with buffers queued with a request if queueing the request
>> > itself fails, possibly because the user hasn't provided enough buffers
>> > with the request? Mark the buffers errorneous and return them to the user?
>> > Probably so, but that requires the user to dequeue the buffers and gather
>> > the request again. I presume this would only happen in special
>> > circumstances though, and not typically in an application using requests.
>> > This, and many other special cases still must be handled by the kernel.
>>
>> Error handling is still pretty weak in that version. I would like to
>> get an overall agreement on the general direction before looking at
>> this more closely though, as I suppose getting things right will take
>> some time.
>
> I believe error handling would be much simpler if we passed all request
> parameters through a single ioctl, as we would have one clear point where to
> perform validation with all required information available.

Compared to fine granularity of particular ioctls, this would make
determining the error cause more difficult or at least would require
providing a special interface to userspace just to communicate the
exact reason of the failure.

Best regards,
Tomasz

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Tomasz Figa]

Hi Sakari,

On Thu, Oct 19, 2017 at 11:43 PM, Sakari Ailus  wrote:
> Hi Alexandre,
>
> On Thu, Sep 28, 2017 at 06:50:18PM +0900, Alexandre Courbot wrote:
>> Hi everyone,
>>
[snip]
>
> Still it shouldn't be forgotten that if the framework is geared towards
> helping drivers "running one job at a time" the scope will be limited to
> memory-to-memory devices; streaming devices, e.g. all kinds of cameras have
> multiple requests being processed simultaneously (or frames are bound to be
> lost, something we can't allow to happen due to framework design).

I'd be interested in some further explanation of these multiple
requests being processed simultaneously and why they couldn't be
included in a single job.

Best regards,
Tomasz

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Tomasz Figa]

Hi Sakari,

On Thu, Oct 19, 2017 at 11:43 PM, Sakari Ailus  wrote:
> Hi Alexandre,
>
> On Thu, Sep 28, 2017 at 06:50:18PM +0900, Alexandre Courbot wrote:
>> Hi everyone,
>>
[snip]
>
> Still it shouldn't be forgotten that if the framework is geared towards
> helping drivers "running one job at a time" the scope will be limited to
> memory-to-memory devices; streaming devices, e.g. all kinds of cameras have
> multiple requests being processed simultaneously (or frames are bound to be
> lost, something we can't allow to happen due to framework design).

I'd be interested in some further explanation of these multiple
requests being processed simultaneously and why they couldn't be
included in a single job.

Best regards,
Tomasz

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Laurent Pinchart]

Hi Hans,

On Wednesday, 25 October 2017 19:19:27 EEST Hans Verkuil wrote:
> On 10/25/2017 05:48 PM, Laurent Pinchart wrote:
> > Hello,
> > 
> > On Monday, 23 October 2017 11:45:01 EEST Alexandre Courbot wrote:
> >> On Thu, Oct 19, 2017 at 11:43 PM, Sakari Ailus  
wrote:
> >>> On Thu, Sep 28, 2017 at 06:50:18PM +0900, Alexandre Courbot wrote:
>  Hi everyone,
>  
>  Here is a new attempt at the "request" (which I propose to rename
>  "jobs")
>  API for V4L2, hopefully in a manner that can converge to something that
>  will be merged. The core ideas should be easy to grasp for those
>  familiar with the previous attemps, yet there are a few important
>  differences.
>  
>  Most notably, user-space does not need to explicitly allocate and
>  manage
>  requests/jobs (but still can if this makes sense). We noticed that only
>  specific use-cases require such an explicit management, and opted for a
>  jobs queue that controls the flow of work over a set of opened devices.
>  This should simplify user-space code quite a bit, while still retaining
>  the ability to manage states explicitly like the previous request API
>  proposals allowed to do.
>  
>  The jobs API defines a few new concepts that user-space can use to
>  control the workflow on a set of opened V4L2 devices:
>  
>  A JOB QUEUE can be created from a set of opened FDs that are part of a
>  pipeline and need to cooperate (be it capture, m2m, or media controller
>  devices).
>  
>  A JOB can then be set up with regular (if slightly modified) V4L2
>  ioctls,
>  and then submitted to the job queue. Once the job queue schedules the
>  job, its parameters (controls, etc) are applied to the devices of the
>  queue, and itsd buffers are processed. Immediately after a job is
>  submitted, the next job is ready to be set up without further user
>  action.
>  
>  Once a job completes, it must be dequeued and user-space can then read
>  back its properties (notably controls) at completion time.
>  
>  Internally, the state of jobs is managed through STATE HANDLERS. Each
>  driver supporting the jobs API needs to specify an implementation of a
>  state handler. Fortunately, most drivers can rely on the generic state
>  handler implementation that simply records and replays a job's
>  parameter
>  using standard V4L2 functions. Thanks to this, adding jobs API support
>  to a driver relying on the control framework and vb2 only requires a
>  dozen lines of codes.
>  
>  Drivers with specific needs or opportunities for optimization can
>  however
>  provide their own implementation of a state handler. This may in
>  particular be beneficial for hardware that supports configuration or
>  command buffers (thinking about VSP1 here).
>  
>  This is still very early work, and focus has been on the following
>  points:
>  
>  * Provide something that anybody can test (currently using vim2m and
>  vivid),
>  * Reuse the current V4L2 APIs as much as possible,
>  * Remain flexible enough to accomodate the inevitable changes that will
>  be requested,
>  * Keep line count low, even if functionality is missing at the moment.
>  
>  Please keep this in mind while going through the patches. In
>  particular,
>  at the moment the parameters of a job are limited to integer controls.
>  I
>  know that much more is expected, but V4L2 has quite a learning curve
>  and
>  I preferred to focus on the general concepts for now. More is coming
>  though! :)
>  
>  I have written two small example programs that demonstrate the use of
>  this API:
>  
>  - With a codec device (vim2m):
>  https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42
>  
>  - With a capture device (vivid):
>  https://gist.github.com/Gnurou/5052e6ab41e7c55164b75d2970bc5a04
>  
>  Considering the history with the request API, I don't expect everything
>  proposed here to be welcome or understood immediately. In particular I
>  apologize for not reusing any of the previous attempts - I was just
>  more
>  comfortable laying down my ideas from scratch.
>  
>  If this proposal is not dismissed as complete garbage I will also be
>  happy to discuss it in-person at the mini-summit in Prague. :)
> >>> 
> >>> Thank you for the initiative and the patchset.
> >>> 
> >>> While reviewing this patchset, I'm concentrating primarily on the
> >>> approach
> >>> taken and the design, not so much in the actual implementation which I
> >>> don't think matters much at this moment.
> >> 
> >> Thanks, that is exactly how I hoped things would go for the moment.
> >> 
> >>> It's difficult to avoid seeing many similarities with the Request API
> >>> patches posted 

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Laurent Pinchart]

Hi Hans,

On Wednesday, 25 October 2017 19:19:27 EEST Hans Verkuil wrote:
> On 10/25/2017 05:48 PM, Laurent Pinchart wrote:
> > Hello,
> > 
> > On Monday, 23 October 2017 11:45:01 EEST Alexandre Courbot wrote:
> >> On Thu, Oct 19, 2017 at 11:43 PM, Sakari Ailus  
wrote:
> >>> On Thu, Sep 28, 2017 at 06:50:18PM +0900, Alexandre Courbot wrote:
>  Hi everyone,
>  
>  Here is a new attempt at the "request" (which I propose to rename
>  "jobs")
>  API for V4L2, hopefully in a manner that can converge to something that
>  will be merged. The core ideas should be easy to grasp for those
>  familiar with the previous attemps, yet there are a few important
>  differences.
>  
>  Most notably, user-space does not need to explicitly allocate and
>  manage
>  requests/jobs (but still can if this makes sense). We noticed that only
>  specific use-cases require such an explicit management, and opted for a
>  jobs queue that controls the flow of work over a set of opened devices.
>  This should simplify user-space code quite a bit, while still retaining
>  the ability to manage states explicitly like the previous request API
>  proposals allowed to do.
>  
>  The jobs API defines a few new concepts that user-space can use to
>  control the workflow on a set of opened V4L2 devices:
>  
>  A JOB QUEUE can be created from a set of opened FDs that are part of a
>  pipeline and need to cooperate (be it capture, m2m, or media controller
>  devices).
>  
>  A JOB can then be set up with regular (if slightly modified) V4L2
>  ioctls,
>  and then submitted to the job queue. Once the job queue schedules the
>  job, its parameters (controls, etc) are applied to the devices of the
>  queue, and itsd buffers are processed. Immediately after a job is
>  submitted, the next job is ready to be set up without further user
>  action.
>  
>  Once a job completes, it must be dequeued and user-space can then read
>  back its properties (notably controls) at completion time.
>  
>  Internally, the state of jobs is managed through STATE HANDLERS. Each
>  driver supporting the jobs API needs to specify an implementation of a
>  state handler. Fortunately, most drivers can rely on the generic state
>  handler implementation that simply records and replays a job's
>  parameter
>  using standard V4L2 functions. Thanks to this, adding jobs API support
>  to a driver relying on the control framework and vb2 only requires a
>  dozen lines of codes.
>  
>  Drivers with specific needs or opportunities for optimization can
>  however
>  provide their own implementation of a state handler. This may in
>  particular be beneficial for hardware that supports configuration or
>  command buffers (thinking about VSP1 here).
>  
>  This is still very early work, and focus has been on the following
>  points:
>  
>  * Provide something that anybody can test (currently using vim2m and
>  vivid),
>  * Reuse the current V4L2 APIs as much as possible,
>  * Remain flexible enough to accomodate the inevitable changes that will
>  be requested,
>  * Keep line count low, even if functionality is missing at the moment.
>  
>  Please keep this in mind while going through the patches. In
>  particular,
>  at the moment the parameters of a job are limited to integer controls.
>  I
>  know that much more is expected, but V4L2 has quite a learning curve
>  and
>  I preferred to focus on the general concepts for now. More is coming
>  though! :)
>  
>  I have written two small example programs that demonstrate the use of
>  this API:
>  
>  - With a codec device (vim2m):
>  https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42
>  
>  - With a capture device (vivid):
>  https://gist.github.com/Gnurou/5052e6ab41e7c55164b75d2970bc5a04
>  
>  Considering the history with the request API, I don't expect everything
>  proposed here to be welcome or understood immediately. In particular I
>  apologize for not reusing any of the previous attempts - I was just
>  more
>  comfortable laying down my ideas from scratch.
>  
>  If this proposal is not dismissed as complete garbage I will also be
>  happy to discuss it in-person at the mini-summit in Prague. :)
> >>> 
> >>> Thank you for the initiative and the patchset.
> >>> 
> >>> While reviewing this patchset, I'm concentrating primarily on the
> >>> approach
> >>> taken and the design, not so much in the actual implementation which I
> >>> don't think matters much at this moment.
> >> 
> >> Thanks, that is exactly how I hoped things would go for the moment.
> >> 
> >>> It's difficult to avoid seeing many similarities with the Request API
> >>> patches posted earlier on. And not only 

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Hans Verkuil]

On 10/25/2017 05:48 PM, Laurent Pinchart wrote:
> Hello,
> 
> On Monday, 23 October 2017 11:45:01 EEST Alexandre Courbot wrote:
>> On Thu, Oct 19, 2017 at 11:43 PM, Sakari Ailus  wrote:
>>> On Thu, Sep 28, 2017 at 06:50:18PM +0900, Alexandre Courbot wrote:
 Hi everyone,

 Here is a new attempt at the "request" (which I propose to rename "jobs")
 API for V4L2, hopefully in a manner that can converge to something that
 will be merged. The core ideas should be easy to grasp for those
 familiar with the previous attemps, yet there are a few important
 differences.

 Most notably, user-space does not need to explicitly allocate and manage
 requests/jobs (but still can if this makes sense). We noticed that only
 specific use-cases require such an explicit management, and opted for a
 jobs queue that controls the flow of work over a set of opened devices.
 This should simplify user-space code quite a bit, while still retaining
 the ability to manage states explicitly like the previous request API
 proposals allowed to do.

 The jobs API defines a few new concepts that user-space can use to
 control the workflow on a set of opened V4L2 devices:

 A JOB QUEUE can be created from a set of opened FDs that are part of a
 pipeline and need to cooperate (be it capture, m2m, or media controller
 devices).

 A JOB can then be set up with regular (if slightly modified) V4L2 ioctls,
 and then submitted to the job queue. Once the job queue schedules the
 job, its parameters (controls, etc) are applied to the devices of the
 queue, and itsd buffers are processed. Immediately after a job is
 submitted, the next job is ready to be set up without further user
 action.

 Once a job completes, it must be dequeued and user-space can then read
 back its properties (notably controls) at completion time.

 Internally, the state of jobs is managed through STATE HANDLERS. Each
 driver supporting the jobs API needs to specify an implementation of a
 state handler. Fortunately, most drivers can rely on the generic state
 handler implementation that simply records and replays a job's parameter
 using standard V4L2 functions. Thanks to this, adding jobs API support
 to a driver relying on the control framework and vb2 only requires a
 dozen lines of codes.

 Drivers with specific needs or opportunities for optimization can however
 provide their own implementation of a state handler. This may in
 particular be beneficial for hardware that supports configuration or
 command buffers (thinking about VSP1 here).

 This is still very early work, and focus has been on the following
 points:

 * Provide something that anybody can test (currently using vim2m and
 vivid),
 * Reuse the current V4L2 APIs as much as possible,
 * Remain flexible enough to accomodate the inevitable changes that will
 be requested,
 * Keep line count low, even if functionality is missing at the moment.

 Please keep this in mind while going through the patches. In particular,
 at the moment the parameters of a job are limited to integer controls. I
 know that much more is expected, but V4L2 has quite a learning curve and
 I preferred to focus on the general concepts for now. More is coming
 though! :)

 I have written two small example programs that demonstrate the use of
 this API:

 - With a codec device (vim2m):
 https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42

 - With a capture device (vivid):
 https://gist.github.com/Gnurou/5052e6ab41e7c55164b75d2970bc5a04

 Considering the history with the request API, I don't expect everything
 proposed here to be welcome or understood immediately. In particular I
 apologize for not reusing any of the previous attempts - I was just more
 comfortable laying down my ideas from scratch.

 If this proposal is not dismissed as complete garbage I will also be
 happy to discuss it in-person at the mini-summit in Prague. :)
>>>
>>> Thank you for the initiative and the patchset.
>>>
>>> While reviewing this patchset, I'm concentrating primarily on the approach
>>> taken and the design, not so much in the actual implementation which I
>>> don't think matters much at this moment.
>>
>> Thanks, that is exactly how I hoped things would go for the moment.
>>
>>> It's difficult to avoid seeing many similarities with the Request API
>>> patches posted earlier on. And not only that, rather you have to start
>>> looking for the differences in what I could call details, while important
>>> design decisions could sometimes be only visible in what appear details at
>>> this point.
>>
>> I was not quite sure whether I should base this work on one of the
>> existing patchsets (and in this case, which one) or 

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Hans Verkuil]

On 10/25/2017 05:48 PM, Laurent Pinchart wrote:
> Hello,
> 
> On Monday, 23 October 2017 11:45:01 EEST Alexandre Courbot wrote:
>> On Thu, Oct 19, 2017 at 11:43 PM, Sakari Ailus  wrote:
>>> On Thu, Sep 28, 2017 at 06:50:18PM +0900, Alexandre Courbot wrote:
 Hi everyone,

 Here is a new attempt at the "request" (which I propose to rename "jobs")
 API for V4L2, hopefully in a manner that can converge to something that
 will be merged. The core ideas should be easy to grasp for those
 familiar with the previous attemps, yet there are a few important
 differences.

 Most notably, user-space does not need to explicitly allocate and manage
 requests/jobs (but still can if this makes sense). We noticed that only
 specific use-cases require such an explicit management, and opted for a
 jobs queue that controls the flow of work over a set of opened devices.
 This should simplify user-space code quite a bit, while still retaining
 the ability to manage states explicitly like the previous request API
 proposals allowed to do.

 The jobs API defines a few new concepts that user-space can use to
 control the workflow on a set of opened V4L2 devices:

 A JOB QUEUE can be created from a set of opened FDs that are part of a
 pipeline and need to cooperate (be it capture, m2m, or media controller
 devices).

 A JOB can then be set up with regular (if slightly modified) V4L2 ioctls,
 and then submitted to the job queue. Once the job queue schedules the
 job, its parameters (controls, etc) are applied to the devices of the
 queue, and itsd buffers are processed. Immediately after a job is
 submitted, the next job is ready to be set up without further user
 action.

 Once a job completes, it must be dequeued and user-space can then read
 back its properties (notably controls) at completion time.

 Internally, the state of jobs is managed through STATE HANDLERS. Each
 driver supporting the jobs API needs to specify an implementation of a
 state handler. Fortunately, most drivers can rely on the generic state
 handler implementation that simply records and replays a job's parameter
 using standard V4L2 functions. Thanks to this, adding jobs API support
 to a driver relying on the control framework and vb2 only requires a
 dozen lines of codes.

 Drivers with specific needs or opportunities for optimization can however
 provide their own implementation of a state handler. This may in
 particular be beneficial for hardware that supports configuration or
 command buffers (thinking about VSP1 here).

 This is still very early work, and focus has been on the following
 points:

 * Provide something that anybody can test (currently using vim2m and
 vivid),
 * Reuse the current V4L2 APIs as much as possible,
 * Remain flexible enough to accomodate the inevitable changes that will
 be requested,
 * Keep line count low, even if functionality is missing at the moment.

 Please keep this in mind while going through the patches. In particular,
 at the moment the parameters of a job are limited to integer controls. I
 know that much more is expected, but V4L2 has quite a learning curve and
 I preferred to focus on the general concepts for now. More is coming
 though! :)

 I have written two small example programs that demonstrate the use of
 this API:

 - With a codec device (vim2m):
 https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42

 - With a capture device (vivid):
 https://gist.github.com/Gnurou/5052e6ab41e7c55164b75d2970bc5a04

 Considering the history with the request API, I don't expect everything
 proposed here to be welcome or understood immediately. In particular I
 apologize for not reusing any of the previous attempts - I was just more
 comfortable laying down my ideas from scratch.

 If this proposal is not dismissed as complete garbage I will also be
 happy to discuss it in-person at the mini-summit in Prague. :)
>>>
>>> Thank you for the initiative and the patchset.
>>>
>>> While reviewing this patchset, I'm concentrating primarily on the approach
>>> taken and the design, not so much in the actual implementation which I
>>> don't think matters much at this moment.
>>
>> Thanks, that is exactly how I hoped things would go for the moment.
>>
>>> It's difficult to avoid seeing many similarities with the Request API
>>> patches posted earlier on. And not only that, rather you have to start
>>> looking for the differences in what I could call details, while important
>>> design decisions could sometimes be only visible in what appear details at
>>> this point.
>>
>> I was not quite sure whether I should base this work on one of the
>> existing patchsets (and in this case, which one) or start from
>> scratch. 

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Laurent Pinchart]

Hello,

On Monday, 23 October 2017 11:45:01 EEST Alexandre Courbot wrote:
> On Thu, Oct 19, 2017 at 11:43 PM, Sakari Ailus  wrote:
> > On Thu, Sep 28, 2017 at 06:50:18PM +0900, Alexandre Courbot wrote:
> >> Hi everyone,
> >> 
> >> Here is a new attempt at the "request" (which I propose to rename "jobs")
> >> API for V4L2, hopefully in a manner that can converge to something that
> >> will be merged. The core ideas should be easy to grasp for those
> >> familiar with the previous attemps, yet there are a few important
> >> differences.
> >> 
> >> Most notably, user-space does not need to explicitly allocate and manage
> >> requests/jobs (but still can if this makes sense). We noticed that only
> >> specific use-cases require such an explicit management, and opted for a
> >> jobs queue that controls the flow of work over a set of opened devices.
> >> This should simplify user-space code quite a bit, while still retaining
> >> the ability to manage states explicitly like the previous request API
> >> proposals allowed to do.
> >> 
> >> The jobs API defines a few new concepts that user-space can use to
> >> control the workflow on a set of opened V4L2 devices:
> >> 
> >> A JOB QUEUE can be created from a set of opened FDs that are part of a
> >> pipeline and need to cooperate (be it capture, m2m, or media controller
> >> devices).
> >> 
> >> A JOB can then be set up with regular (if slightly modified) V4L2 ioctls,
> >> and then submitted to the job queue. Once the job queue schedules the
> >> job, its parameters (controls, etc) are applied to the devices of the
> >> queue, and itsd buffers are processed. Immediately after a job is
> >> submitted, the next job is ready to be set up without further user
> >> action.
> >> 
> >> Once a job completes, it must be dequeued and user-space can then read
> >> back its properties (notably controls) at completion time.
> >> 
> >> Internally, the state of jobs is managed through STATE HANDLERS. Each
> >> driver supporting the jobs API needs to specify an implementation of a
> >> state handler. Fortunately, most drivers can rely on the generic state
> >> handler implementation that simply records and replays a job's parameter
> >> using standard V4L2 functions. Thanks to this, adding jobs API support
> >> to a driver relying on the control framework and vb2 only requires a
> >> dozen lines of codes.
> >> 
> >> Drivers with specific needs or opportunities for optimization can however
> >> provide their own implementation of a state handler. This may in
> >> particular be beneficial for hardware that supports configuration or
> >> command buffers (thinking about VSP1 here).
> >> 
> >> This is still very early work, and focus has been on the following
> >> points:
> >> 
> >> * Provide something that anybody can test (currently using vim2m and
> >> vivid),
> >> * Reuse the current V4L2 APIs as much as possible,
> >> * Remain flexible enough to accomodate the inevitable changes that will
> >> be requested,
> >> * Keep line count low, even if functionality is missing at the moment.
> >> 
> >> Please keep this in mind while going through the patches. In particular,
> >> at the moment the parameters of a job are limited to integer controls. I
> >> know that much more is expected, but V4L2 has quite a learning curve and
> >> I preferred to focus on the general concepts for now. More is coming
> >> though! :)
> >> 
> >> I have written two small example programs that demonstrate the use of
> >> this API:
> >> 
> >> - With a codec device (vim2m):
> >> https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42
> >> 
> >> - With a capture device (vivid):
> >> https://gist.github.com/Gnurou/5052e6ab41e7c55164b75d2970bc5a04
> >> 
> >> Considering the history with the request API, I don't expect everything
> >> proposed here to be welcome or understood immediately. In particular I
> >> apologize for not reusing any of the previous attempts - I was just more
> >> comfortable laying down my ideas from scratch.
> >> 
> >> If this proposal is not dismissed as complete garbage I will also be
> >> happy to discuss it in-person at the mini-summit in Prague. :)
> > 
> > Thank you for the initiative and the patchset.
> > 
> > While reviewing this patchset, I'm concentrating primarily on the approach
> > taken and the design, not so much in the actual implementation which I
> > don't think matters much at this moment.
> 
> Thanks, that is exactly how I hoped things would go for the moment.
> 
> > It's difficult to avoid seeing many similarities with the Request API
> > patches posted earlier on. And not only that, rather you have to start
> > looking for the differences in what I could call details, while important
> > design decisions could sometimes be only visible in what appear details at
> > this point.
> 
> I was not quite sure whether I should base this work on one of the
> existing patchsets (and in this case, which one) or start from
> scratch. This being my first 

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Laurent Pinchart]

Hello,

On Monday, 23 October 2017 11:45:01 EEST Alexandre Courbot wrote:
> On Thu, Oct 19, 2017 at 11:43 PM, Sakari Ailus  wrote:
> > On Thu, Sep 28, 2017 at 06:50:18PM +0900, Alexandre Courbot wrote:
> >> Hi everyone,
> >> 
> >> Here is a new attempt at the "request" (which I propose to rename "jobs")
> >> API for V4L2, hopefully in a manner that can converge to something that
> >> will be merged. The core ideas should be easy to grasp for those
> >> familiar with the previous attemps, yet there are a few important
> >> differences.
> >> 
> >> Most notably, user-space does not need to explicitly allocate and manage
> >> requests/jobs (but still can if this makes sense). We noticed that only
> >> specific use-cases require such an explicit management, and opted for a
> >> jobs queue that controls the flow of work over a set of opened devices.
> >> This should simplify user-space code quite a bit, while still retaining
> >> the ability to manage states explicitly like the previous request API
> >> proposals allowed to do.
> >> 
> >> The jobs API defines a few new concepts that user-space can use to
> >> control the workflow on a set of opened V4L2 devices:
> >> 
> >> A JOB QUEUE can be created from a set of opened FDs that are part of a
> >> pipeline and need to cooperate (be it capture, m2m, or media controller
> >> devices).
> >> 
> >> A JOB can then be set up with regular (if slightly modified) V4L2 ioctls,
> >> and then submitted to the job queue. Once the job queue schedules the
> >> job, its parameters (controls, etc) are applied to the devices of the
> >> queue, and itsd buffers are processed. Immediately after a job is
> >> submitted, the next job is ready to be set up without further user
> >> action.
> >> 
> >> Once a job completes, it must be dequeued and user-space can then read
> >> back its properties (notably controls) at completion time.
> >> 
> >> Internally, the state of jobs is managed through STATE HANDLERS. Each
> >> driver supporting the jobs API needs to specify an implementation of a
> >> state handler. Fortunately, most drivers can rely on the generic state
> >> handler implementation that simply records and replays a job's parameter
> >> using standard V4L2 functions. Thanks to this, adding jobs API support
> >> to a driver relying on the control framework and vb2 only requires a
> >> dozen lines of codes.
> >> 
> >> Drivers with specific needs or opportunities for optimization can however
> >> provide their own implementation of a state handler. This may in
> >> particular be beneficial for hardware that supports configuration or
> >> command buffers (thinking about VSP1 here).
> >> 
> >> This is still very early work, and focus has been on the following
> >> points:
> >> 
> >> * Provide something that anybody can test (currently using vim2m and
> >> vivid),
> >> * Reuse the current V4L2 APIs as much as possible,
> >> * Remain flexible enough to accomodate the inevitable changes that will
> >> be requested,
> >> * Keep line count low, even if functionality is missing at the moment.
> >> 
> >> Please keep this in mind while going through the patches. In particular,
> >> at the moment the parameters of a job are limited to integer controls. I
> >> know that much more is expected, but V4L2 has quite a learning curve and
> >> I preferred to focus on the general concepts for now. More is coming
> >> though! :)
> >> 
> >> I have written two small example programs that demonstrate the use of
> >> this API:
> >> 
> >> - With a codec device (vim2m):
> >> https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42
> >> 
> >> - With a capture device (vivid):
> >> https://gist.github.com/Gnurou/5052e6ab41e7c55164b75d2970bc5a04
> >> 
> >> Considering the history with the request API, I don't expect everything
> >> proposed here to be welcome or understood immediately. In particular I
> >> apologize for not reusing any of the previous attempts - I was just more
> >> comfortable laying down my ideas from scratch.
> >> 
> >> If this proposal is not dismissed as complete garbage I will also be
> >> happy to discuss it in-person at the mini-summit in Prague. :)
> > 
> > Thank you for the initiative and the patchset.
> > 
> > While reviewing this patchset, I'm concentrating primarily on the approach
> > taken and the design, not so much in the actual implementation which I
> > don't think matters much at this moment.
> 
> Thanks, that is exactly how I hoped things would go for the moment.
> 
> > It's difficult to avoid seeing many similarities with the Request API
> > patches posted earlier on. And not only that, rather you have to start
> > looking for the differences in what I could call details, while important
> > design decisions could sometimes be only visible in what appear details at
> > this point.
> 
> I was not quite sure whether I should base this work on one of the
> existing patchsets (and in this case, which one) or start from
> scratch. This being my first contribution to a new 

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Gustavo Padovan]

2017-10-16 Hans Verkuil :

> Hi Alexandre,
> 
> Thank you very much for working on this. Much appreciated!
> 
> I only did a very high-level review of the patch series: there is not much
> point IMHO of doing a detailed review given the upcoming discussions in
> Prague. It's better to wait until we agree with the high-level API.
> 
> Regarding the public API: the ioctls seem sane. It's all very similar to the
> other implementations we've seen.
> 
> I'm still not sure about the name 'job', but this is 'just' a naming issue.
> 
> The part where I have more doubts is the need to create a new device node.
> 
> For the upcoming meeting I would like to discuss whether this cannot be added
> to the media API.
> 
> Originally the plan was that the media API would be subsystem-agnostic and 
> could
> also be used by ALSA/DRM/etc. This never happened and I also am not aware of 
> any
> movement in that area.
> 
> I am wondering whether we should just be realistic and abandon the 'subsystem
> agnostic' part and be willing to add e.g. the job support to the media API.

Stupid question here: is there any techinically possible way to support
it through the media API while being subsystem agnostic?

Gustavo

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Gustavo Padovan]

2017-10-16 Hans Verkuil :

> Hi Alexandre,
> 
> Thank you very much for working on this. Much appreciated!
> 
> I only did a very high-level review of the patch series: there is not much
> point IMHO of doing a detailed review given the upcoming discussions in
> Prague. It's better to wait until we agree with the high-level API.
> 
> Regarding the public API: the ioctls seem sane. It's all very similar to the
> other implementations we've seen.
> 
> I'm still not sure about the name 'job', but this is 'just' a naming issue.
> 
> The part where I have more doubts is the need to create a new device node.
> 
> For the upcoming meeting I would like to discuss whether this cannot be added
> to the media API.
> 
> Originally the plan was that the media API would be subsystem-agnostic and 
> could
> also be used by ALSA/DRM/etc. This never happened and I also am not aware of 
> any
> movement in that area.
> 
> I am wondering whether we should just be realistic and abandon the 'subsystem
> agnostic' part and be willing to add e.g. the job support to the media API.

Stupid question here: is there any techinically possible way to support
it through the media API while being subsystem agnostic?

Gustavo

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Alexandre Courbot]

Hi Sakari, thanks for the feedback!

On Thu, Oct 19, 2017 at 11:43 PM, Sakari Ailus  wrote:
> Hi Alexandre,
>
> On Thu, Sep 28, 2017 at 06:50:18PM +0900, Alexandre Courbot wrote:
>> Hi everyone,
>>
>> Here is a new attempt at the "request" (which I propose to rename "jobs") API
>> for V4L2, hopefully in a manner that can converge to something that will be
>> merged. The core ideas should be easy to grasp for those familiar with the
>> previous attemps, yet there are a few important differences.
>>
>> Most notably, user-space does not need to explicitly allocate and manage
>> requests/jobs (but still can if this makes sense). We noticed that only 
>> specific
>> use-cases require such an explicit management, and opted for a jobs queue 
>> that
>> controls the flow of work over a set of opened devices. This should simplify
>> user-space code quite a bit, while still retaining the ability to manage 
>> states
>> explicitly like the previous request API proposals allowed to do.
>>
>> The jobs API defines a few new concepts that user-space can use to control 
>> the
>> workflow on a set of opened V4L2 devices:
>>
>> A JOB QUEUE can be created from a set of opened FDs that are part of a 
>> pipeline
>> and need to cooperate (be it capture, m2m, or media controller devices).
>>
>> A JOB can then be set up with regular (if slightly modified) V4L2 ioctls, and
>> then submitted to the job queue. Once the job queue schedules the job, its
>> parameters (controls, etc) are applied to the devices of the queue, and itsd
>> buffers are processed. Immediately after a job is submitted, the next job is
>> ready to be set up without further user action.
>>
>> Once a job completes, it must be dequeued and user-space can then read back 
>> its
>> properties (notably controls) at completion time.
>>
>> Internally, the state of jobs is managed through STATE HANDLERS. Each driver
>> supporting the jobs API needs to specify an implementation of a state 
>> handler.
>> Fortunately, most drivers can rely on the generic state handler 
>> implementation
>> that simply records and replays a job's parameter using standard V4L2 
>> functions.
>> Thanks to this, adding jobs API support to a driver relying on the control
>> framework and vb2 only requires a dozen lines of codes.
>>
>> Drivers with specific needs or opportunities for optimization can however
>> provide their own implementation of a state handler. This may in particular 
>> be
>> beneficial for hardware that supports configuration or command buffers 
>> (thinking
>> about VSP1 here).
>>
>> This is still very early work, and focus has been on the following points:
>>
>> * Provide something that anybody can test (currently using vim2m and vivid),
>> * Reuse the current V4L2 APIs as much as possible,
>> * Remain flexible enough to accomodate the inevitable changes that will be
>>   requested,
>> * Keep line count low, even if functionality is missing at the moment.
>>
>> Please keep this in mind while going through the patches. In particular, at 
>> the
>> moment the parameters of a job are limited to integer controls. I know that 
>> much
>> more is expected, but V4L2 has quite a learning curve and I preferred to 
>> focus
>> on the general concepts for now. More is coming though! :)
>>
>> I have written two small example programs that demonstrate the use of this 
>> API:
>>
>> - With a codec device (vim2m): 
>> https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42
>>
>> - With a capture device (vivid): 
>> https://gist.github.com/Gnurou/5052e6ab41e7c55164b75d2970bc5a04
>>
>> Considering the history with the request API, I don't expect everything 
>> proposed
>> here to be welcome or understood immediately. In particular I apologize for 
>> not
>> reusing any of the previous attempts - I was just more comfortable laying 
>> down
>> my ideas from scratch.
>>
>> If this proposal is not dismissed as complete garbage I will also be happy to
>> discuss it in-person at the mini-summit in Prague. :)
>
> Thank you for the initiative and the patchset.
>
> While reviewing this patchset, I'm concentrating primarily on the approach
> taken and the design, not so much in the actual implementation which I
> don't think matters much at this moment.

Thanks, that is exactly how I hoped things would go for the moment.

>
> It's difficult to avoid seeing many similarities with the Request API
> patches posted earlier on. And not only that, rather you have to start
> looking for the differences in what I could call details, while important
> design decisions could sometimes be only visible in what appear details at
> this point.

I was not quite sure whether I should base this work on one of the
existing patchsets (and in this case, which one) or start from
scratch. This being my first contribution to a new area of the kernel
for me, I decided to start from scratch as it would yield more
educative value.

>
> Both request and jobs APIs have the concept of a 

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Alexandre Courbot]

Hi Sakari, thanks for the feedback!

On Thu, Oct 19, 2017 at 11:43 PM, Sakari Ailus  wrote:
> Hi Alexandre,
>
> On Thu, Sep 28, 2017 at 06:50:18PM +0900, Alexandre Courbot wrote:
>> Hi everyone,
>>
>> Here is a new attempt at the "request" (which I propose to rename "jobs") API
>> for V4L2, hopefully in a manner that can converge to something that will be
>> merged. The core ideas should be easy to grasp for those familiar with the
>> previous attemps, yet there are a few important differences.
>>
>> Most notably, user-space does not need to explicitly allocate and manage
>> requests/jobs (but still can if this makes sense). We noticed that only 
>> specific
>> use-cases require such an explicit management, and opted for a jobs queue 
>> that
>> controls the flow of work over a set of opened devices. This should simplify
>> user-space code quite a bit, while still retaining the ability to manage 
>> states
>> explicitly like the previous request API proposals allowed to do.
>>
>> The jobs API defines a few new concepts that user-space can use to control 
>> the
>> workflow on a set of opened V4L2 devices:
>>
>> A JOB QUEUE can be created from a set of opened FDs that are part of a 
>> pipeline
>> and need to cooperate (be it capture, m2m, or media controller devices).
>>
>> A JOB can then be set up with regular (if slightly modified) V4L2 ioctls, and
>> then submitted to the job queue. Once the job queue schedules the job, its
>> parameters (controls, etc) are applied to the devices of the queue, and itsd
>> buffers are processed. Immediately after a job is submitted, the next job is
>> ready to be set up without further user action.
>>
>> Once a job completes, it must be dequeued and user-space can then read back 
>> its
>> properties (notably controls) at completion time.
>>
>> Internally, the state of jobs is managed through STATE HANDLERS. Each driver
>> supporting the jobs API needs to specify an implementation of a state 
>> handler.
>> Fortunately, most drivers can rely on the generic state handler 
>> implementation
>> that simply records and replays a job's parameter using standard V4L2 
>> functions.
>> Thanks to this, adding jobs API support to a driver relying on the control
>> framework and vb2 only requires a dozen lines of codes.
>>
>> Drivers with specific needs or opportunities for optimization can however
>> provide their own implementation of a state handler. This may in particular 
>> be
>> beneficial for hardware that supports configuration or command buffers 
>> (thinking
>> about VSP1 here).
>>
>> This is still very early work, and focus has been on the following points:
>>
>> * Provide something that anybody can test (currently using vim2m and vivid),
>> * Reuse the current V4L2 APIs as much as possible,
>> * Remain flexible enough to accomodate the inevitable changes that will be
>>   requested,
>> * Keep line count low, even if functionality is missing at the moment.
>>
>> Please keep this in mind while going through the patches. In particular, at 
>> the
>> moment the parameters of a job are limited to integer controls. I know that 
>> much
>> more is expected, but V4L2 has quite a learning curve and I preferred to 
>> focus
>> on the general concepts for now. More is coming though! :)
>>
>> I have written two small example programs that demonstrate the use of this 
>> API:
>>
>> - With a codec device (vim2m): 
>> https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42
>>
>> - With a capture device (vivid): 
>> https://gist.github.com/Gnurou/5052e6ab41e7c55164b75d2970bc5a04
>>
>> Considering the history with the request API, I don't expect everything 
>> proposed
>> here to be welcome or understood immediately. In particular I apologize for 
>> not
>> reusing any of the previous attempts - I was just more comfortable laying 
>> down
>> my ideas from scratch.
>>
>> If this proposal is not dismissed as complete garbage I will also be happy to
>> discuss it in-person at the mini-summit in Prague. :)
>
> Thank you for the initiative and the patchset.
>
> While reviewing this patchset, I'm concentrating primarily on the approach
> taken and the design, not so much in the actual implementation which I
> don't think matters much at this moment.

Thanks, that is exactly how I hoped things would go for the moment.

>
> It's difficult to avoid seeing many similarities with the Request API
> patches posted earlier on. And not only that, rather you have to start
> looking for the differences in what I could call details, while important
> design decisions could sometimes be only visible in what appear details at
> this point.

I was not quite sure whether I should base this work on one of the
existing patchsets (and in this case, which one) or start from
scratch. This being my first contribution to a new area of the kernel
for me, I decided to start from scratch as it would yield more
educative value.

>
> Both request and jobs APIs have the concept of a request, or a job, which
> 

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Alexandre Courbot]

Hi Hans,

On Mon, Oct 16, 2017 at 8:06 PM, Hans Verkuil  wrote:
> Hi Alexandre,
>
> Thank you very much for working on this. Much appreciated!

Thanks! I feel like there is still a long way before we get this right
though, so your guidance is highly appreciated! :)

> I only did a very high-level review of the patch series: there is not much
> point IMHO of doing a detailed review given the upcoming discussions in
> Prague. It's better to wait until we agree with the high-level API.

Agree on this.

> Regarding the public API: the ioctls seem sane. It's all very similar to the
> other implementations we've seen.
>
> I'm still not sure about the name 'job', but this is 'just' a naming issue.
>
> The part where I have more doubts is the need to create a new device node.
>
> For the upcoming meeting I would like to discuss whether this cannot be added
> to the media API.

Yeah, this device node is not exactly well-received. Although I wanted
to see what people would think about it (and am taking note of
everyone's doubts), its main purpose for now is to make my life easier
and the job API available no matter which driver I am working on. If
we move this to the media API, we need said media API activated, and
of course a media node, which is not a guarantee on every setup.

My point here is that although the current patchset does not reflect
this, the jobs API is also supposed to control media devices. In fact
it is supposed to allow control on a whole range of V4L2 features,
which is why I am hesitating tying it to an existing subsystem at the
moment. Also device and media nodes are currently created by
individual drivers. Jobs allow to control said devices together, thus
in my (maybe flawed!) logic it made more sense if they came from a
global node created by the framework.

Note that I am willing to be flexible on that, and the current setup
is mostly for my convenience as I develop this.

>
> Originally the plan was that the media API would be subsystem-agnostic and 
> could
> also be used by ALSA/DRM/etc. This never happened and I also am not aware of 
> any
> movement in that area.
>
> I am wondering whether we should just be realistic and abandon the 'subsystem
> agnostic' part and be willing to add e.g. the job support to the media API.
>
> We can also consider controlling sub-devices via the media device node instead
> of creating separate v4l-subdev device nodes.
>
> This does not necessarily preclude the use of the media device by other
> subsystems, but it certainly ties it much closer to the media subsystem. On 
> the
> other hand, it does make life easier for us (and I like easy!).
>
> This is just a brainstorm at the moment, but I am interested what others think
> of making /dev/mediaX specific to the media subsystem (at least we chose the
> right name for that device node!).

V4L2 is still fairly new to me, so I am probably still confused, but
aren't /dev/mediaX nodes created by the media subsystem currently?

Cheers,
Alex.

>
> Obviously, if we go into that direction, then that will have an obvious impact
> on the jobs API, especially if we want to be able to control subdevs through 
> the
> media device instead of through the v4l-subdev devices.
>
> Regards,
>
> Hans
>
> On 09/28/2017 11:50 AM, Alexandre Courbot wrote:
>> Hi everyone,
>>
>> Here is a new attempt at the "request" (which I propose to rename "jobs") API
>> for V4L2, hopefully in a manner that can converge to something that will be
>> merged. The core ideas should be easy to grasp for those familiar with the
>> previous attemps, yet there are a few important differences.
>>
>> Most notably, user-space does not need to explicitly allocate and manage
>> requests/jobs (but still can if this makes sense). We noticed that only 
>> specific
>> use-cases require such an explicit management, and opted for a jobs queue 
>> that
>> controls the flow of work over a set of opened devices. This should simplify
>> user-space code quite a bit, while still retaining the ability to manage 
>> states
>> explicitly like the previous request API proposals allowed to do.
>>
>> The jobs API defines a few new concepts that user-space can use to control 
>> the
>> workflow on a set of opened V4L2 devices:
>>
>> A JOB QUEUE can be created from a set of opened FDs that are part of a 
>> pipeline
>> and need to cooperate (be it capture, m2m, or media controller devices).
>>
>> A JOB can then be set up with regular (if slightly modified) V4L2 ioctls, and
>> then submitted to the job queue. Once the job queue schedules the job, its
>> parameters (controls, etc) are applied to the devices of the queue, and itsd
>> buffers are processed. Immediately after a job is submitted, the next job is
>> ready to be set up without further user action.
>>
>> Once a job completes, it must be dequeued and user-space can then read back 
>> its
>> properties (notably controls) at completion time.
>>
>> Internally, the state of jobs is managed 

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Alexandre Courbot]

Hi Hans,

On Mon, Oct 16, 2017 at 8:06 PM, Hans Verkuil  wrote:
> Hi Alexandre,
>
> Thank you very much for working on this. Much appreciated!

Thanks! I feel like there is still a long way before we get this right
though, so your guidance is highly appreciated! :)

> I only did a very high-level review of the patch series: there is not much
> point IMHO of doing a detailed review given the upcoming discussions in
> Prague. It's better to wait until we agree with the high-level API.

Agree on this.

> Regarding the public API: the ioctls seem sane. It's all very similar to the
> other implementations we've seen.
>
> I'm still not sure about the name 'job', but this is 'just' a naming issue.
>
> The part where I have more doubts is the need to create a new device node.
>
> For the upcoming meeting I would like to discuss whether this cannot be added
> to the media API.

Yeah, this device node is not exactly well-received. Although I wanted
to see what people would think about it (and am taking note of
everyone's doubts), its main purpose for now is to make my life easier
and the job API available no matter which driver I am working on. If
we move this to the media API, we need said media API activated, and
of course a media node, which is not a guarantee on every setup.

My point here is that although the current patchset does not reflect
this, the jobs API is also supposed to control media devices. In fact
it is supposed to allow control on a whole range of V4L2 features,
which is why I am hesitating tying it to an existing subsystem at the
moment. Also device and media nodes are currently created by
individual drivers. Jobs allow to control said devices together, thus
in my (maybe flawed!) logic it made more sense if they came from a
global node created by the framework.

Note that I am willing to be flexible on that, and the current setup
is mostly for my convenience as I develop this.

>
> Originally the plan was that the media API would be subsystem-agnostic and 
> could
> also be used by ALSA/DRM/etc. This never happened and I also am not aware of 
> any
> movement in that area.
>
> I am wondering whether we should just be realistic and abandon the 'subsystem
> agnostic' part and be willing to add e.g. the job support to the media API.
>
> We can also consider controlling sub-devices via the media device node instead
> of creating separate v4l-subdev device nodes.
>
> This does not necessarily preclude the use of the media device by other
> subsystems, but it certainly ties it much closer to the media subsystem. On 
> the
> other hand, it does make life easier for us (and I like easy!).
>
> This is just a brainstorm at the moment, but I am interested what others think
> of making /dev/mediaX specific to the media subsystem (at least we chose the
> right name for that device node!).

V4L2 is still fairly new to me, so I am probably still confused, but
aren't /dev/mediaX nodes created by the media subsystem currently?

Cheers,
Alex.

>
> Obviously, if we go into that direction, then that will have an obvious impact
> on the jobs API, especially if we want to be able to control subdevs through 
> the
> media device instead of through the v4l-subdev devices.
>
> Regards,
>
> Hans
>
> On 09/28/2017 11:50 AM, Alexandre Courbot wrote:
>> Hi everyone,
>>
>> Here is a new attempt at the "request" (which I propose to rename "jobs") API
>> for V4L2, hopefully in a manner that can converge to something that will be
>> merged. The core ideas should be easy to grasp for those familiar with the
>> previous attemps, yet there are a few important differences.
>>
>> Most notably, user-space does not need to explicitly allocate and manage
>> requests/jobs (but still can if this makes sense). We noticed that only 
>> specific
>> use-cases require such an explicit management, and opted for a jobs queue 
>> that
>> controls the flow of work over a set of opened devices. This should simplify
>> user-space code quite a bit, while still retaining the ability to manage 
>> states
>> explicitly like the previous request API proposals allowed to do.
>>
>> The jobs API defines a few new concepts that user-space can use to control 
>> the
>> workflow on a set of opened V4L2 devices:
>>
>> A JOB QUEUE can be created from a set of opened FDs that are part of a 
>> pipeline
>> and need to cooperate (be it capture, m2m, or media controller devices).
>>
>> A JOB can then be set up with regular (if slightly modified) V4L2 ioctls, and
>> then submitted to the job queue. Once the job queue schedules the job, its
>> parameters (controls, etc) are applied to the devices of the queue, and itsd
>> buffers are processed. Immediately after a job is submitted, the next job is
>> ready to be set up without further user action.
>>
>> Once a job completes, it must be dequeued and user-space can then read back 
>> its
>> properties (notably controls) at completion time.
>>
>> Internally, the state of jobs is managed through STATE HANDLERS. 

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Sakari Ailus]

Hi Alexandre,

On Thu, Sep 28, 2017 at 06:50:18PM +0900, Alexandre Courbot wrote:
> Hi everyone,
> 
> Here is a new attempt at the "request" (which I propose to rename "jobs") API
> for V4L2, hopefully in a manner that can converge to something that will be
> merged. The core ideas should be easy to grasp for those familiar with the
> previous attemps, yet there are a few important differences.
> 
> Most notably, user-space does not need to explicitly allocate and manage
> requests/jobs (but still can if this makes sense). We noticed that only 
> specific
> use-cases require such an explicit management, and opted for a jobs queue that
> controls the flow of work over a set of opened devices. This should simplify
> user-space code quite a bit, while still retaining the ability to manage 
> states
> explicitly like the previous request API proposals allowed to do.
> 
> The jobs API defines a few new concepts that user-space can use to control the
> workflow on a set of opened V4L2 devices:
> 
> A JOB QUEUE can be created from a set of opened FDs that are part of a 
> pipeline
> and need to cooperate (be it capture, m2m, or media controller devices).
> 
> A JOB can then be set up with regular (if slightly modified) V4L2 ioctls, and
> then submitted to the job queue. Once the job queue schedules the job, its
> parameters (controls, etc) are applied to the devices of the queue, and itsd
> buffers are processed. Immediately after a job is submitted, the next job is
> ready to be set up without further user action.
> 
> Once a job completes, it must be dequeued and user-space can then read back 
> its
> properties (notably controls) at completion time.
> 
> Internally, the state of jobs is managed through STATE HANDLERS. Each driver
> supporting the jobs API needs to specify an implementation of a state handler.
> Fortunately, most drivers can rely on the generic state handler implementation
> that simply records and replays a job's parameter using standard V4L2 
> functions.
> Thanks to this, adding jobs API support to a driver relying on the control
> framework and vb2 only requires a dozen lines of codes.
> 
> Drivers with specific needs or opportunities for optimization can however
> provide their own implementation of a state handler. This may in particular be
> beneficial for hardware that supports configuration or command buffers 
> (thinking
> about VSP1 here).
> 
> This is still very early work, and focus has been on the following points:
> 
> * Provide something that anybody can test (currently using vim2m and vivid),
> * Reuse the current V4L2 APIs as much as possible,
> * Remain flexible enough to accomodate the inevitable changes that will be
>   requested,
> * Keep line count low, even if functionality is missing at the moment.
> 
> Please keep this in mind while going through the patches. In particular, at 
> the
> moment the parameters of a job are limited to integer controls. I know that 
> much
> more is expected, but V4L2 has quite a learning curve and I preferred to focus
> on the general concepts for now. More is coming though! :)
> 
> I have written two small example programs that demonstrate the use of this 
> API:
> 
> - With a codec device (vim2m): 
> https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42
> 
> - With a capture device (vivid): 
> https://gist.github.com/Gnurou/5052e6ab41e7c55164b75d2970bc5a04
> 
> Considering the history with the request API, I don't expect everything 
> proposed
> here to be welcome or understood immediately. In particular I apologize for 
> not
> reusing any of the previous attempts - I was just more comfortable laying down
> my ideas from scratch.
> 
> If this proposal is not dismissed as complete garbage I will also be happy to
> discuss it in-person at the mini-summit in Prague. :)

Thank you for the initiative and the patchset.

While reviewing this patchset, I'm concentrating primarily on the approach
taken and the design, not so much in the actual implementation which I
don't think matters much at this moment.

It's difficult to avoid seeing many similarities with the Request API
patches posted earlier on. And not only that, rather you have to start
looking for the differences in what I could call details, while important
design decisions could sometimes be only visible in what appear details at
this point.

Both request and jobs APIs have the concept of a request, or a job, which
is created by the user and then different buffers or controls can be bound
to that request. (Other configuration isn't really excluded but it's
non-trivial to implement in practice.) This is common for both.

The differences begin however how the functionality within the scope is
actieved, in particular:

- A new user space interface (character device) is created for the jobs API
  whereas requests use existing Media controller interface. Therefore the
  jobs API is specific to V4L2. Consequently, the V4L2 jobs API may not
  support Media controller link state 

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Sakari Ailus]

Hi Alexandre,

On Thu, Sep 28, 2017 at 06:50:18PM +0900, Alexandre Courbot wrote:
> Hi everyone,
> 
> Here is a new attempt at the "request" (which I propose to rename "jobs") API
> for V4L2, hopefully in a manner that can converge to something that will be
> merged. The core ideas should be easy to grasp for those familiar with the
> previous attemps, yet there are a few important differences.
> 
> Most notably, user-space does not need to explicitly allocate and manage
> requests/jobs (but still can if this makes sense). We noticed that only 
> specific
> use-cases require such an explicit management, and opted for a jobs queue that
> controls the flow of work over a set of opened devices. This should simplify
> user-space code quite a bit, while still retaining the ability to manage 
> states
> explicitly like the previous request API proposals allowed to do.
> 
> The jobs API defines a few new concepts that user-space can use to control the
> workflow on a set of opened V4L2 devices:
> 
> A JOB QUEUE can be created from a set of opened FDs that are part of a 
> pipeline
> and need to cooperate (be it capture, m2m, or media controller devices).
> 
> A JOB can then be set up with regular (if slightly modified) V4L2 ioctls, and
> then submitted to the job queue. Once the job queue schedules the job, its
> parameters (controls, etc) are applied to the devices of the queue, and itsd
> buffers are processed. Immediately after a job is submitted, the next job is
> ready to be set up without further user action.
> 
> Once a job completes, it must be dequeued and user-space can then read back 
> its
> properties (notably controls) at completion time.
> 
> Internally, the state of jobs is managed through STATE HANDLERS. Each driver
> supporting the jobs API needs to specify an implementation of a state handler.
> Fortunately, most drivers can rely on the generic state handler implementation
> that simply records and replays a job's parameter using standard V4L2 
> functions.
> Thanks to this, adding jobs API support to a driver relying on the control
> framework and vb2 only requires a dozen lines of codes.
> 
> Drivers with specific needs or opportunities for optimization can however
> provide their own implementation of a state handler. This may in particular be
> beneficial for hardware that supports configuration or command buffers 
> (thinking
> about VSP1 here).
> 
> This is still very early work, and focus has been on the following points:
> 
> * Provide something that anybody can test (currently using vim2m and vivid),
> * Reuse the current V4L2 APIs as much as possible,
> * Remain flexible enough to accomodate the inevitable changes that will be
>   requested,
> * Keep line count low, even if functionality is missing at the moment.
> 
> Please keep this in mind while going through the patches. In particular, at 
> the
> moment the parameters of a job are limited to integer controls. I know that 
> much
> more is expected, but V4L2 has quite a learning curve and I preferred to focus
> on the general concepts for now. More is coming though! :)
> 
> I have written two small example programs that demonstrate the use of this 
> API:
> 
> - With a codec device (vim2m): 
> https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42
> 
> - With a capture device (vivid): 
> https://gist.github.com/Gnurou/5052e6ab41e7c55164b75d2970bc5a04
> 
> Considering the history with the request API, I don't expect everything 
> proposed
> here to be welcome or understood immediately. In particular I apologize for 
> not
> reusing any of the previous attempts - I was just more comfortable laying down
> my ideas from scratch.
> 
> If this proposal is not dismissed as complete garbage I will also be happy to
> discuss it in-person at the mini-summit in Prague. :)

Thank you for the initiative and the patchset.

While reviewing this patchset, I'm concentrating primarily on the approach
taken and the design, not so much in the actual implementation which I
don't think matters much at this moment.

It's difficult to avoid seeing many similarities with the Request API
patches posted earlier on. And not only that, rather you have to start
looking for the differences in what I could call details, while important
design decisions could sometimes be only visible in what appear details at
this point.

Both request and jobs APIs have the concept of a request, or a job, which
is created by the user and then different buffers or controls can be bound
to that request. (Other configuration isn't really excluded but it's
non-trivial to implement in practice.) This is common for both.

The differences begin however how the functionality within the scope is
actieved, in particular:

- A new user space interface (character device) is created for the jobs API
  whereas requests use existing Media controller interface. Therefore the
  jobs API is specific to V4L2. Consequently, the V4L2 jobs API may not
  support Media controller link state 

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Hans Verkuil]

Hi Alexandre,

Thank you very much for working on this. Much appreciated!

I only did a very high-level review of the patch series: there is not much
point IMHO of doing a detailed review given the upcoming discussions in
Prague. It's better to wait until we agree with the high-level API.

Regarding the public API: the ioctls seem sane. It's all very similar to the
other implementations we've seen.

I'm still not sure about the name 'job', but this is 'just' a naming issue.

The part where I have more doubts is the need to create a new device node.

For the upcoming meeting I would like to discuss whether this cannot be added
to the media API.

Originally the plan was that the media API would be subsystem-agnostic and could
also be used by ALSA/DRM/etc. This never happened and I also am not aware of any
movement in that area.

I am wondering whether we should just be realistic and abandon the 'subsystem
agnostic' part and be willing to add e.g. the job support to the media API.

We can also consider controlling sub-devices via the media device node instead
of creating separate v4l-subdev device nodes.

This does not necessarily preclude the use of the media device by other
subsystems, but it certainly ties it much closer to the media subsystem. On the
other hand, it does make life easier for us (and I like easy!).

This is just a brainstorm at the moment, but I am interested what others think
of making /dev/mediaX specific to the media subsystem (at least we chose the
right name for that device node!).

Obviously, if we go into that direction, then that will have an obvious impact
on the jobs API, especially if we want to be able to control subdevs through the
media device instead of through the v4l-subdev devices.

Regards,

Hans

On 09/28/2017 11:50 AM, Alexandre Courbot wrote:
> Hi everyone,
> 
> Here is a new attempt at the "request" (which I propose to rename "jobs") API
> for V4L2, hopefully in a manner that can converge to something that will be
> merged. The core ideas should be easy to grasp for those familiar with the
> previous attemps, yet there are a few important differences.
> 
> Most notably, user-space does not need to explicitly allocate and manage
> requests/jobs (but still can if this makes sense). We noticed that only 
> specific
> use-cases require such an explicit management, and opted for a jobs queue that
> controls the flow of work over a set of opened devices. This should simplify
> user-space code quite a bit, while still retaining the ability to manage 
> states
> explicitly like the previous request API proposals allowed to do.
> 
> The jobs API defines a few new concepts that user-space can use to control the
> workflow on a set of opened V4L2 devices:
> 
> A JOB QUEUE can be created from a set of opened FDs that are part of a 
> pipeline
> and need to cooperate (be it capture, m2m, or media controller devices).
> 
> A JOB can then be set up with regular (if slightly modified) V4L2 ioctls, and
> then submitted to the job queue. Once the job queue schedules the job, its
> parameters (controls, etc) are applied to the devices of the queue, and itsd
> buffers are processed. Immediately after a job is submitted, the next job is
> ready to be set up without further user action.
> 
> Once a job completes, it must be dequeued and user-space can then read back 
> its
> properties (notably controls) at completion time.
> 
> Internally, the state of jobs is managed through STATE HANDLERS. Each driver
> supporting the jobs API needs to specify an implementation of a state handler.
> Fortunately, most drivers can rely on the generic state handler implementation
> that simply records and replays a job's parameter using standard V4L2 
> functions.
> Thanks to this, adding jobs API support to a driver relying on the control
> framework and vb2 only requires a dozen lines of codes.
> 
> Drivers with specific needs or opportunities for optimization can however
> provide their own implementation of a state handler. This may in particular be
> beneficial for hardware that supports configuration or command buffers 
> (thinking
> about VSP1 here).
> 
> This is still very early work, and focus has been on the following points:
> 
> * Provide something that anybody can test (currently using vim2m and vivid),
> * Reuse the current V4L2 APIs as much as possible,
> * Remain flexible enough to accomodate the inevitable changes that will be
>   requested,
> * Keep line count low, even if functionality is missing at the moment.
> 
> Please keep this in mind while going through the patches. In particular, at 
> the
> moment the parameters of a job are limited to integer controls. I know that 
> much
> more is expected, but V4L2 has quite a learning curve and I preferred to focus
> on the general concepts for now. More is coming though! :)
> 
> I have written two small example programs that demonstrate the use of this 
> API:
> 
> - With a codec device (vim2m): 
> 

Re: [RFC PATCH 0/9] V4L2 Jobs API WIP

[Hans Verkuil]

Hi Alexandre,

Thank you very much for working on this. Much appreciated!

I only did a very high-level review of the patch series: there is not much
point IMHO of doing a detailed review given the upcoming discussions in
Prague. It's better to wait until we agree with the high-level API.

Regarding the public API: the ioctls seem sane. It's all very similar to the
other implementations we've seen.

I'm still not sure about the name 'job', but this is 'just' a naming issue.

The part where I have more doubts is the need to create a new device node.

For the upcoming meeting I would like to discuss whether this cannot be added
to the media API.

Originally the plan was that the media API would be subsystem-agnostic and could
also be used by ALSA/DRM/etc. This never happened and I also am not aware of any
movement in that area.

I am wondering whether we should just be realistic and abandon the 'subsystem
agnostic' part and be willing to add e.g. the job support to the media API.

We can also consider controlling sub-devices via the media device node instead
of creating separate v4l-subdev device nodes.

This does not necessarily preclude the use of the media device by other
subsystems, but it certainly ties it much closer to the media subsystem. On the
other hand, it does make life easier for us (and I like easy!).

This is just a brainstorm at the moment, but I am interested what others think
of making /dev/mediaX specific to the media subsystem (at least we chose the
right name for that device node!).

Obviously, if we go into that direction, then that will have an obvious impact
on the jobs API, especially if we want to be able to control subdevs through the
media device instead of through the v4l-subdev devices.

Regards,

Hans

On 09/28/2017 11:50 AM, Alexandre Courbot wrote:
> Hi everyone,
> 
> Here is a new attempt at the "request" (which I propose to rename "jobs") API
> for V4L2, hopefully in a manner that can converge to something that will be
> merged. The core ideas should be easy to grasp for those familiar with the
> previous attemps, yet there are a few important differences.
> 
> Most notably, user-space does not need to explicitly allocate and manage
> requests/jobs (but still can if this makes sense). We noticed that only 
> specific
> use-cases require such an explicit management, and opted for a jobs queue that
> controls the flow of work over a set of opened devices. This should simplify
> user-space code quite a bit, while still retaining the ability to manage 
> states
> explicitly like the previous request API proposals allowed to do.
> 
> The jobs API defines a few new concepts that user-space can use to control the
> workflow on a set of opened V4L2 devices:
> 
> A JOB QUEUE can be created from a set of opened FDs that are part of a 
> pipeline
> and need to cooperate (be it capture, m2m, or media controller devices).
> 
> A JOB can then be set up with regular (if slightly modified) V4L2 ioctls, and
> then submitted to the job queue. Once the job queue schedules the job, its
> parameters (controls, etc) are applied to the devices of the queue, and itsd
> buffers are processed. Immediately after a job is submitted, the next job is
> ready to be set up without further user action.
> 
> Once a job completes, it must be dequeued and user-space can then read back 
> its
> properties (notably controls) at completion time.
> 
> Internally, the state of jobs is managed through STATE HANDLERS. Each driver
> supporting the jobs API needs to specify an implementation of a state handler.
> Fortunately, most drivers can rely on the generic state handler implementation
> that simply records and replays a job's parameter using standard V4L2 
> functions.
> Thanks to this, adding jobs API support to a driver relying on the control
> framework and vb2 only requires a dozen lines of codes.
> 
> Drivers with specific needs or opportunities for optimization can however
> provide their own implementation of a state handler. This may in particular be
> beneficial for hardware that supports configuration or command buffers 
> (thinking
> about VSP1 here).
> 
> This is still very early work, and focus has been on the following points:
> 
> * Provide something that anybody can test (currently using vim2m and vivid),
> * Reuse the current V4L2 APIs as much as possible,
> * Remain flexible enough to accomodate the inevitable changes that will be
>   requested,
> * Keep line count low, even if functionality is missing at the moment.
> 
> Please keep this in mind while going through the patches. In particular, at 
> the
> moment the parameters of a job are limited to integer controls. I know that 
> much
> more is expected, but V4L2 has quite a learning curve and I preferred to focus
> on the general concepts for now. More is coming though! :)
> 
> I have written two small example programs that demonstrate the use of this 
> API:
> 
> - With a codec device (vim2m): 
> 

[RFC PATCH 0/9] V4L2 Jobs API WIP

[Alexandre Courbot]

Hi everyone,

Here is a new attempt at the "request" (which I propose to rename "jobs") API
for V4L2, hopefully in a manner that can converge to something that will be
merged. The core ideas should be easy to grasp for those familiar with the
previous attemps, yet there are a few important differences.

Most notably, user-space does not need to explicitly allocate and manage
requests/jobs (but still can if this makes sense). We noticed that only specific
use-cases require such an explicit management, and opted for a jobs queue that
controls the flow of work over a set of opened devices. This should simplify
user-space code quite a bit, while still retaining the ability to manage states
explicitly like the previous request API proposals allowed to do.

The jobs API defines a few new concepts that user-space can use to control the
workflow on a set of opened V4L2 devices:

A JOB QUEUE can be created from a set of opened FDs that are part of a pipeline
and need to cooperate (be it capture, m2m, or media controller devices).

A JOB can then be set up with regular (if slightly modified) V4L2 ioctls, and
then submitted to the job queue. Once the job queue schedules the job, its
parameters (controls, etc) are applied to the devices of the queue, and itsd
buffers are processed. Immediately after a job is submitted, the next job is
ready to be set up without further user action.

Once a job completes, it must be dequeued and user-space can then read back its
properties (notably controls) at completion time.

Internally, the state of jobs is managed through STATE HANDLERS. Each driver
supporting the jobs API needs to specify an implementation of a state handler.
Fortunately, most drivers can rely on the generic state handler implementation
that simply records and replays a job's parameter using standard V4L2 functions.
Thanks to this, adding jobs API support to a driver relying on the control
framework and vb2 only requires a dozen lines of codes.

Drivers with specific needs or opportunities for optimization can however
provide their own implementation of a state handler. This may in particular be
beneficial for hardware that supports configuration or command buffers (thinking
about VSP1 here).

This is still very early work, and focus has been on the following points:

* Provide something that anybody can test (currently using vim2m and vivid),
* Reuse the current V4L2 APIs as much as possible,
* Remain flexible enough to accomodate the inevitable changes that will be
  requested,
* Keep line count low, even if functionality is missing at the moment.

Please keep this in mind while going through the patches. In particular, at the
moment the parameters of a job are limited to integer controls. I know that much
more is expected, but V4L2 has quite a learning curve and I preferred to focus
on the general concepts for now. More is coming though! :)

I have written two small example programs that demonstrate the use of this API:

- With a codec device (vim2m): 
https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42

- With a capture device (vivid): 
https://gist.github.com/Gnurou/5052e6ab41e7c55164b75d2970bc5a04

Considering the history with the request API, I don't expect everything proposed
here to be welcome or understood immediately. In particular I apologize for not
reusing any of the previous attempts - I was just more comfortable laying down
my ideas from scratch.

If this proposal is not dismissed as complete garbage I will also be happy to
discuss it in-person at the mini-summit in Prague. :)

Cheers,
Alex.

Alexandre Courbot (9):
  [media] v4l2-core: add v4l2_is_v4l2_file function
  [media] v4l2-core: add core jobs API support
  [media] videobuf2: add support for jobs API
  [media] v4l2-ctrls: add support for jobs API
  [media] v4l2-job: add generic jobs ops
  [media] m2m: add generic support for jobs API
  [media] vim2m: add jobs API support
  [media] vivid: add jobs API support for capture device
  [media] document jobs API

 Documentation/media/intro.rst  |   2 +
 Documentation/media/media_uapi.rst |   1 +
 Documentation/media/uapi/jobs/jobs-api.rst |  23 +
 Documentation/media/uapi/jobs/jobs-example.rst |  69 ++
 Documentation/media/uapi/jobs/jobs-intro.rst   |  61 ++
 Documentation/media/uapi/jobs/jobs-queue.rst   |  73 ++
 Documentation/media/uapi/jobs/jobs-queue.svg   | 192 ++
 .../media/uapi/v4l/vidioc-g-ext-ctrls.rst  |   6 +
 drivers/media/platform/vim2m.c |  24 +
 drivers/media/platform/vivid/vivid-core.c  |  16 +
 drivers/media/platform/vivid/vivid-core.h  |   2 +
 drivers/media/platform/vivid/vivid-kthread-cap.c   |   5 +
 drivers/media/v4l2-core/Makefile   |   4 +-
 drivers/media/v4l2-core/v4l2-ctrls.c   |  50 +-
 drivers/media/v4l2-core/v4l2-dev.c |  12 +
 drivers/media/v4l2-core/v4l2-job-generic.c | 394 +++
 

[RFC PATCH 0/9] V4L2 Jobs API WIP

[Alexandre Courbot]

Hi everyone,

Here is a new attempt at the "request" (which I propose to rename "jobs") API
for V4L2, hopefully in a manner that can converge to something that will be
merged. The core ideas should be easy to grasp for those familiar with the
previous attemps, yet there are a few important differences.

Most notably, user-space does not need to explicitly allocate and manage
requests/jobs (but still can if this makes sense). We noticed that only specific
use-cases require such an explicit management, and opted for a jobs queue that
controls the flow of work over a set of opened devices. This should simplify
user-space code quite a bit, while still retaining the ability to manage states
explicitly like the previous request API proposals allowed to do.

The jobs API defines a few new concepts that user-space can use to control the
workflow on a set of opened V4L2 devices:

A JOB QUEUE can be created from a set of opened FDs that are part of a pipeline
and need to cooperate (be it capture, m2m, or media controller devices).

A JOB can then be set up with regular (if slightly modified) V4L2 ioctls, and
then submitted to the job queue. Once the job queue schedules the job, its
parameters (controls, etc) are applied to the devices of the queue, and itsd
buffers are processed. Immediately after a job is submitted, the next job is
ready to be set up without further user action.

Once a job completes, it must be dequeued and user-space can then read back its
properties (notably controls) at completion time.

Internally, the state of jobs is managed through STATE HANDLERS. Each driver
supporting the jobs API needs to specify an implementation of a state handler.
Fortunately, most drivers can rely on the generic state handler implementation
that simply records and replays a job's parameter using standard V4L2 functions.
Thanks to this, adding jobs API support to a driver relying on the control
framework and vb2 only requires a dozen lines of codes.

Drivers with specific needs or opportunities for optimization can however
provide their own implementation of a state handler. This may in particular be
beneficial for hardware that supports configuration or command buffers (thinking
about VSP1 here).

This is still very early work, and focus has been on the following points:

* Provide something that anybody can test (currently using vim2m and vivid),
* Reuse the current V4L2 APIs as much as possible,
* Remain flexible enough to accomodate the inevitable changes that will be
  requested,
* Keep line count low, even if functionality is missing at the moment.

Please keep this in mind while going through the patches. In particular, at the
moment the parameters of a job are limited to integer controls. I know that much
more is expected, but V4L2 has quite a learning curve and I preferred to focus
on the general concepts for now. More is coming though! :)

I have written two small example programs that demonstrate the use of this API:

- With a codec device (vim2m): 
https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42

- With a capture device (vivid): 
https://gist.github.com/Gnurou/5052e6ab41e7c55164b75d2970bc5a04

Considering the history with the request API, I don't expect everything proposed
here to be welcome or understood immediately. In particular I apologize for not
reusing any of the previous attempts - I was just more comfortable laying down
my ideas from scratch.

If this proposal is not dismissed as complete garbage I will also be happy to
discuss it in-person at the mini-summit in Prague. :)

Cheers,
Alex.

Alexandre Courbot (9):
  [media] v4l2-core: add v4l2_is_v4l2_file function
  [media] v4l2-core: add core jobs API support
  [media] videobuf2: add support for jobs API
  [media] v4l2-ctrls: add support for jobs API
  [media] v4l2-job: add generic jobs ops
  [media] m2m: add generic support for jobs API
  [media] vim2m: add jobs API support
  [media] vivid: add jobs API support for capture device
  [media] document jobs API

 Documentation/media/intro.rst  |   2 +
 Documentation/media/media_uapi.rst |   1 +
 Documentation/media/uapi/jobs/jobs-api.rst |  23 +
 Documentation/media/uapi/jobs/jobs-example.rst |  69 ++
 Documentation/media/uapi/jobs/jobs-intro.rst   |  61 ++
 Documentation/media/uapi/jobs/jobs-queue.rst   |  73 ++
 Documentation/media/uapi/jobs/jobs-queue.svg   | 192 ++
 .../media/uapi/v4l/vidioc-g-ext-ctrls.rst  |   6 +
 drivers/media/platform/vim2m.c |  24 +
 drivers/media/platform/vivid/vivid-core.c  |  16 +
 drivers/media/platform/vivid/vivid-core.h  |   2 +
 drivers/media/platform/vivid/vivid-kthread-cap.c   |   5 +
 drivers/media/v4l2-core/Makefile   |   4 +-
 drivers/media/v4l2-core/v4l2-ctrls.c   |  50 +-
 drivers/media/v4l2-core/v4l2-dev.c |  12 +
 drivers/media/v4l2-core/v4l2-job-generic.c | 394 +++