Re: [BRAINSTORM] Controls, matrices and properties
Hi Laurent, On Thu, Jul 18, 2013 at 03:07:26AM +0200, Laurent Pinchart wrote: ... An unrelated thing, which is out of scope for now, but something to think about: when passing around large amounts of (configuration) data the number of times the data is copied really counts. Especially on embedded systems. Memory mapping helps avoiding problems --- what would happen is that the driver would access memory mapped to the device directly and the driver would then get the address to pass to the device as the configuration. Like video buffers, but for control, not data. Would you map that memory to userspace as well ? Yes --- that's the very intent. This way all unnecessary memory copies in the kernel can be avoided. Say, if you're providing (and copying) the device a new LSC table of 128 kiB for every frame while recording video, it will show up in the energy consumption of the system. It does let the user space to do wrong things, too. But the same is equally true for video buffers as well. -- Cheers, Sakari Ailus e-mail: sakari.ai...@iki.fi XMPP: sai...@retiisi.org.uk -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [BRAINSTORM] Controls, matrices and properties
Hi Hans, Thank you for the proposal. On Monday 08 July 2013 13:06:56 Hans Verkuil wrote: Hi all, I have been working on support for passing matrices to/from drivers using a new matrix API. See this earlier thread for more background information: http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/662 00 The basic feedback is that, yes, matrices are useful, but it is yet another control-like API. My problem with using the control API for things like this is that the control API has been designed for use with GUIs: e.g. the controls are elements that the end-user can modify through a GUI. Things like a matrix or some really low-level driver property are either hard to model in a GUI or too advanced and obscure for an end-user. That's true for some controls as well. I think the controls API has evolved over time to accommodate the needs of both user-facing, GUI-friendly controls and low-level controls (especially in the embedded world). The controls APIs (both the userspace API and the in-kernel control framework) were suitable for both (after a bit of tweaking in some cases), so we just haven't realized how conceptually different controls were. There's not much more in common between a brightness level and a flash strobe time then between a contrast level and a gamma table or motion detection matrix. On the other hand, the control framework has all the desirable properties that you would want: atomicity (as far as is allowed by the hardware), the ability to get/set multiple controls in one ioctl, efficient, inheritance of subdev controls in bridge devices, events, etc. I'm wondering whether we cannot tweak the control API a bit to make it possible to use it for matrices and general 'properties' as well. The main requirement for me is that when applications enumerate over controls such properties should never turn up in the enumerations: only controls suitable for a GUI should appear. After all, an application would have no idea what to do with a matrix of e.g. 200x300 elements. While it is possible to extend queryctrl to e.g. enumerate only properties instead of controls, it is probably better to create a new VIDIOC_QUERYPROP ioctl. Also because the v4l2_queryctrl is pretty full and has no support to set the minimum/maximum values of a 64 bit value. In addition, the name field is not needed for a property, I think. Currently the name is there for the GUI, not for identification purposes. Names indeed feel a bit out of place. Even for GUI-related controls the controls API doesn't support localization (not that it should!), so handling control names in a central location in userspace would make sense (names can still be provided by the kernel as hints though, to simplify basic command line applications). Moreover, I've never been really convinced by the GUI-related flags we currently have in the controls API. They feel like a policy decision to me, and should ideally (at least in my ideal view of V4L2) be part of userspace. If an application wants to render controls as knobs instead of sliders I don't see a reason why the kernel should hint otherwise. Maybe we could rethink, while developing the properties API, how userspace should use properties globally, from a userspace point of view. What is it that intrinsically make some of the controls we have today be displayed as sliders for instance ? Is it really a property of the control ? What influences our idea of how controls should be rendered ? At the end of the day, whether we should present a control to a user will depend on the user and his/her use cases. There's no one-size-fits-them-all rule to decide whether all individual controls should or should not be displayed in a panel application. I haven't really thought this topic through, but a binary decision made in the kernel doesn't look like the right solution to me. As Sakari noted, user-facing panel applications could just whitelist controls they want to display. This would work for the most simple cases, but would require updating all those userspace applications when we add a user- facing control, which adds an additional burden to developers and make extensions slower to roll out. One could however argue that such controls are very rarely added. I have no clear answer to this question. For setting/getting controls the existing extended control API can be used, although I would be inclined to go for VIDIOC_G/S/TRY_PROPS ioctls as well. For example, when I set a matrix property it is very desirable to pass only a subset of the matrix along instead of a full matrix. In my original matrix proposal I had a v4l2_rect struct that defined that. But there is no space in struct v4l2_ext_control to store such information. In general, implementing properties requires more variation since the GUI restriction has been lifted. Also, properties can be assigned to specific internal objects (e.g. buffer
Re: [BRAINSTORM] Controls, matrices and properties
Hi Sakari, Just a small question below, I've addressed all the rest in a reply to Hans' original e-mail. On Thursday 11 July 2013 00:30:21 Sakari Ailus wrote: Hans Verkuil wrote: Hi all, I have been working on support for passing matrices to/from drivers using a new matrix API. See this earlier thread for more background information: http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/6 6200 The basic feedback is that, yes, matrices are useful, but it is yet another control-like API. My problem with using the control API for things like this is that the control API has been designed for use with GUIs: e.g. the controls are elements that the end-user can modify through a GUI. Things like a matrix or some really low-level driver property are either hard to model in a GUI or too advanced and obscure for an end-user. We also have a lot of low level controls. GUI implementations can always choose not to show matrix controls. I think a low level control flag has been proposed earlier on, but AFAIR the conclusion that time around was that it's sometimes difficult to define what is actually a low level control and what isn't. IMHO (and according to Unix principles, too), APIs should provide means, not policy. Saying that controls should be used for something that can (or should) be displayed by a GUI, and what isn't displayed in a GUI isn't a control, definitely falls into this category. On the other hand, the control framework has all the desirable properties that you would want: atomicity (as far as is allowed by the hardware), the ability to get/set multiple controls in one ioctl, efficient, inheritance of subdev controls in bridge devices, events, etc. I'm wondering whether we cannot tweak the control API a bit to make it possible to use it for matrices and general 'properties' as well. The main requirement for me is that when applications enumerate over controls such properties should never turn up in the enumerations: only controls suitable for a GUI should appear. After all, an application would have no idea what to do with a matrix of e.g. 200x300 elements. This sounds like the low-level control flag. I'm certainly not against it. I have to admit I'm not someone who'd ever access controls through a GUI, and if it helps, then why not. Alternatively... how about just ignoring control types that aren't supported in GUI? That'd be probably even easier for GUIs than checking a flag --- just ignore what you don't know about. While it is possible to extend queryctrl to e.g. enumerate only properties instead of controls, it is probably better to create a new VIDIOC_QUERYPROP ioctl. Also because the v4l2_queryctrl is pretty full and has no support to set the minimum/maximum values of a 64 bit value. In addition, the name field is not needed for a property, I think. Currently the name is there for the GUI, not for identification purposes. The are drivers that use private controls the ID of which is only defined as a macro in the driver. I wonder if user space programs expect controls under certain names. Alternatively we could require that macro definitions exists for all new controls. Would you intend VIDIOC_QUERYPROP to replace VIDIOC_QUERYCTRL or not? I might just create an extended version of QUERYCTRL which would fix the limits for 64-bit controls, too... it'd be easy to add a wrapper in v4l2-ioctl.c to implement the original VIDIOV_QUERYCTRL for drivers that implement the extended version (like we've done a bunch of time already). For setting/getting controls the existing extended control API can be used, although I would be inclined to go for VIDIOC_G/S/TRY_PROPS ioctls as well. For example, when I set a matrix property it is very desirable to pass only a subset of the matrix along instead of a full matrix. In my original matrix proposal I had a v4l2_rect struct that defined that. But there is no space in struct v4l2_ext_control to store such information. Doe you have a use case for this? An unrelated thing, which is out of scope for now, but something to think about: when passing around large amounts of (configuration) data the number of times the data is copied really counts. Especially on embedded systems. Memory mapping helps avoiding problems --- what would happen is that the driver would access memory mapped to the device directly and the driver would then get the address to pass to the device as the configuration. Like video buffers, but for control, not data. Would you map that memory to userspace as well ? This requires a new RFC (one or more). For later, definitely. In general, implementing properties requires more variation since the GUI restriction has been lifted. Also, properties can be assigned to specific internal objects (e.g. buffer specific properties), so you need fields to tell the kernel with which object
Re: [BRAINSTORM] Controls, matrices and properties
Hi Sakari, Thanks for looking at this! Much appreciated. On Wed 10 July 2013 23:30:21 Sakari Ailus wrote: Hi Hans, Hans Verkuil wrote: Hi all, I have been working on support for passing matrices to/from drivers using a new matrix API. See this earlier thread for more background information: http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/66200 The basic feedback is that, yes, matrices are useful, but it is yet another control-like API. My problem with using the control API for things like this is that the control API has been designed for use with GUIs: e.g. the controls are elements that the end-user can modify through a GUI. Things like a matrix or some really low-level driver property are either hard to model in a GUI or too advanced and obscure for an end-user. We also have a lot of low level controls. GUI implementations can always choose not to show matrix controls. I think a low level control flag has been proposed earlier on, but AFAIR the conclusion that time around was that it's sometimes difficult to define what is actually a low level control and what isn't. That's correct. Another reason was that existing applications don't check for such a flag, so they will show everything anyway. IMHO (and according to Unix principles, too), APIs should provide means, not policy. Saying that controls should be used for something that can (or should) be displayed by a GUI, and what isn't displayed in a GUI isn't a control, definitely falls into this category. Well, yes, but tell that to the original API designers :-) It is what it is, and the whole control API *has* been designed with a GUI in mind. On the other hand, the control framework has all the desirable properties that you would want: atomicity (as far as is allowed by the hardware), the ability to get/set multiple controls in one ioctl, efficient, inheritance of subdev controls in bridge devices, events, etc. I'm wondering whether we cannot tweak the control API a bit to make it possible to use it for matrices and general 'properties' as well. The main requirement for me is that when applications enumerate over controls such properties should never turn up in the enumerations: only controls suitable for a GUI should appear. After all, an application would have no idea what to do with a matrix of e.g. 200x300 elements. This sounds like the low-level control flag. I'm certainly not against it. I have to admit I'm not someone who'd ever access controls through a GUI, and if it helps, then why not. Alternatively... how about just ignoring control types that aren't supported in GUI? That'd be probably even easier for GUIs than checking a flag --- just ignore what you don't know about. It runs into the same problems: properties that you don't want to show up in a GUI and that have a valid control type would then appear. While it is possible to extend queryctrl to e.g. enumerate only properties instead of controls, it is probably better to create a new VIDIOC_QUERYPROP ioctl. Also because the v4l2_queryctrl is pretty full and has no support to set the minimum/maximum values of a 64 bit value. In addition, the name field is not needed for a property, I think. Currently the name is there for the GUI, not for identification purposes. The are drivers that use private controls the ID of which is only defined as a macro in the driver. I wonder if user space programs expect controls under certain names. Occasionally, yes. But in that case the control ID has to be available in a public header. Alternatively we could require that macro definitions exists for all new controls. Would you intend VIDIOC_QUERYPROP to replace VIDIOC_QUERYCTRL or not? I might just create an extended version of QUERYCTRL which would fix the limits for 64-bit controls, too... it'd be easy to add a wrapper in v4l2-ioctl.c to implement the original VIDIOV_QUERYCTRL for drivers that implement the extended version (like we've done a bunch of time already). I haven't decided yet. For setting/getting controls the existing extended control API can be used, although I would be inclined to go for VIDIOC_G/S/TRY_PROPS ioctls as well. For example, when I set a matrix property it is very desirable to pass only a subset of the matrix along instead of a full matrix. In my original matrix proposal I had a v4l2_rect struct that defined that. But there is no space in struct v4l2_ext_control to store such information. Doe you have a use case for this? If you have a large matrix (e.g. motion detection sensitivity for 16x16 pixel squares in an image) and you only want to update a subset, then this is actually quite useful. No need to completely exchange the full matrix if you only are interested in specific parts. An unrelated thing, which is out of scope for now, but something to think about: when passing
Re: [BRAINSTORM] Controls, matrices and properties
Hans Verkuil wrote: If possible I would like to discuss this during the New Orleans conference. I know Laurent and Mauro will be there. I won't be able to attend that one, but I'll be in Edinburgh in October. -- Sakari Ailus sakari.ai...@iki.fi -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [BRAINSTORM] Controls, matrices and properties
Hi Hans, Hans Verkuil wrote: Hi all, I have been working on support for passing matrices to/from drivers using a new matrix API. See this earlier thread for more background information: http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/66200 The basic feedback is that, yes, matrices are useful, but it is yet another control-like API. My problem with using the control API for things like this is that the control API has been designed for use with GUIs: e.g. the controls are elements that the end-user can modify through a GUI. Things like a matrix or some really low-level driver property are either hard to model in a GUI or too advanced and obscure for an end-user. We also have a lot of low level controls. GUI implementations can always choose not to show matrix controls. I think a low level control flag has been proposed earlier on, but AFAIR the conclusion that time around was that it's sometimes difficult to define what is actually a low level control and what isn't. IMHO (and according to Unix principles, too), APIs should provide means, not policy. Saying that controls should be used for something that can (or should) be displayed by a GUI, and what isn't displayed in a GUI isn't a control, definitely falls into this category. On the other hand, the control framework has all the desirable properties that you would want: atomicity (as far as is allowed by the hardware), the ability to get/set multiple controls in one ioctl, efficient, inheritance of subdev controls in bridge devices, events, etc. I'm wondering whether we cannot tweak the control API a bit to make it possible to use it for matrices and general 'properties' as well. The main requirement for me is that when applications enumerate over controls such properties should never turn up in the enumerations: only controls suitable for a GUI should appear. After all, an application would have no idea what to do with a matrix of e.g. 200x300 elements. This sounds like the low-level control flag. I'm certainly not against it. I have to admit I'm not someone who'd ever access controls through a GUI, and if it helps, then why not. Alternatively... how about just ignoring control types that aren't supported in GUI? That'd be probably even easier for GUIs than checking a flag --- just ignore what you don't know about. While it is possible to extend queryctrl to e.g. enumerate only properties instead of controls, it is probably better to create a new VIDIOC_QUERYPROP ioctl. Also because the v4l2_queryctrl is pretty full and has no support to set the minimum/maximum values of a 64 bit value. In addition, the name field is not needed for a property, I think. Currently the name is there for the GUI, not for identification purposes. The are drivers that use private controls the ID of which is only defined as a macro in the driver. I wonder if user space programs expect controls under certain names. Alternatively we could require that macro definitions exists for all new controls. Would you intend VIDIOC_QUERYPROP to replace VIDIOC_QUERYCTRL or not? I might just create an extended version of QUERYCTRL which would fix the limits for 64-bit controls, too... it'd be easy to add a wrapper in v4l2-ioctl.c to implement the original VIDIOV_QUERYCTRL for drivers that implement the extended version (like we've done a bunch of time already). For setting/getting controls the existing extended control API can be used, although I would be inclined to go for VIDIOC_G/S/TRY_PROPS ioctls as well. For example, when I set a matrix property it is very desirable to pass only a subset of the matrix along instead of a full matrix. In my original matrix proposal I had a v4l2_rect struct that defined that. But there is no space in struct v4l2_ext_control to store such information. Doe you have a use case for this? An unrelated thing, which is out of scope for now, but something to think about: when passing around large amounts of (configuration) data the number of times the data is copied really counts. Especially on embedded systems. Memory mapping helps avoiding problems --- what would happen is that the driver would access memory mapped to the device directly and the driver would then get the address to pass to the device as the configuration. Like video buffers, but for control, not data. This requires a new RFC (one or more). For later, definitely. In general, implementing properties requires more variation since the GUI restriction has been lifted. Also, properties can be assigned to specific internal objects (e.g. buffer specific properties), so you need fields to tell the kernel with which object the property is associated. Interesting idea. Definitely. However, although the public API is different from the control API, there is no reason not to use the internal control framework for both. This could be extended ^ 2 controls. For existing controls the scope would always be either the
