Re: [RFC v4] V4L2 API for flash devices
Em 19-05-2011 05:12, Sakari Ailus escreveu:
> Sylwester Nawrocki wrote:
These were mostly fixed point arithmetic numbers in [32-bit numerator/
32-bit denominator] form carrying exposure time, shutter speed, aperture,
brightness, flash, etc. information. The tags could be read from ISP after
it buffered a frame in its memory and processed it.
In case of a JPEG image format the tags can be embedded into the main
image file. But the image processors not always supported that so we used
to have an ioctl for the purpose of retrieving the metadata in user space.
In some cases it is desired to read data directly from the driver rather
than parsing a relatively large buffer.
It would be good to have a uniform interface for passing such data to
applications. I think in that particular use case a control id/value pair
sequences would do.
> - Which formats are your rational numbers in? A kernel interface can't
> really have floating point numbers, so there would need to be a sane way
> to pass these to user space.
The V4L2 API has support for rational numbers. The frame rate is specified as a
rational number. There's a struct for that:
struct v4l2_fract {
__u32 numerator;
__u32 denominator;
};
Cheers,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v4] V4L2 API for flash devices
Sylwester Nawrocki wrote: > Hi Sakari, Hi Sylwester, > On 05/17/2011 10:34 PM, Sakari Ailus wrote: >> Sylwester Nawrocki wrote: >>> On 05/09/2011 12:11 AM, Sakari Ailus wrote: Sylwester Nawrocki wrote: > On 05/07/2011 02:46 PM, Hans Verkuil wrote: >> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote: >>> Hi, >>> >>> This is a fourth proposal for an interface for controlling flash devices >>> on the V4L2/v4l2_subdev APIs. > ... >>> 3. Sensor metadata on frames >>> >>> >>> It'd be useful to be able to read back sensor metadata. If the flash is >>> strobed (on sensor hardware) while streaming, it's difficult to know >>> otherwise which frame in the stream has been exposed with flash. >> >> I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer >> flag? >> That way userspace can tell if that particular frame was taken with >> flash. > > This looks more as a workaround for the problem rather than a good long > term solution. It might be tempting to use the buffer flags which seem > to be be more or less intended for buffer control. > I'd like much more to see a buffer flags to be used to indicate whether > an additional plane of (meta)data is carried by the buffer. > There seem to be many more parameters, than a single flag indicating > whether the frame has been exposed with flash or not, needed to be > carried over to user space. > But then we might need some standard format of the meta data, perhaps > control id/value pairs and possibly a per plane configurable memory > type. There are multiple possible approaches for this. For sensors where metadata is register-value pairs, that is, essentially V4L2 control values, I think this should be parsed by the sensor driver. The ISP (camera bridge) driver does receive the data so it'd have to "ask for help" from the sensor driver. >>> >>> I am inclined to let the ISP drivers parse the data but on the other hand >>> it might be difficult to access same DMA buffers in kernel _and_ user space. >> >> This is just about mapping the buffer to both kernel and user spaces. If >> the ISP has an iommu the kernel mapping might already exist if it comes >> from vmalloc(). > > Yes, I know. I was thinking of possibly required different mapping attributes > for kernel and user space and the problems on ARM with that. As I replied to Laurent, this is not an issue since the memory wouldn't be mapped to user space. The user space would get the result which has been interpreted by the sensor driver. > Also as metadata is supposed to occupy only small part of a frame buffer > perhaps > only one page or so could be mapped in kernel space. > I'm referring here mainly to SMIA++ method of storing metadata. Yes, I agree. > In case of sensors I used to work with it wouldn't be necessary to touch the > main frame buffers as the metadata is transmitted out of band. This depends on link a little, but on CSI-2 this is a separate channel. In other cases the metadata is part of the frame --- but I don't know anyone using metadata in such a case, so it _might_ be possible to just ignore this use case. Another alternative would be to pass binary blobs to user space and let a library to interpret the data. This might be the way to go at least with some more exotic cases. >> As discussed previously, using V4L2 control events shouldn't probably be the way to go, but I think it's obvious that this is _somehow_ bound to controls, at least control ids. > Also as Sakari indicated some sensors adopt custom meta data formats > so maybe we need to introduce standard fourcc like IDs for meta data > formats? I am not sure whether it is possible to create common > description of an image meta data that fits all H/W. I'm not sure either since I know of only one example. That example, i.e. register-value pairs, should be something that I'd assume _some_ other hardware uses as well, but there could exist also hardware which doesn't. This solution might not work on that hardware. >>> >>> Of course it's hard to find a silver bullet for a hardware we do not know ;) >>> If there is metadata which does not associate to V4L2 controls (or ioctls), either existing or not yet existing, then that probably should be parsed by the driver. On the other hand, I can't think of metadata that wouldn't fall under this right now. :-) >>> >>> Some metadata are arrays of length specific to a given attribute, >>> I wonder how to support that with v4l2 controls ? >> >> Is the metadata something which really isn't associated to any V4L2 >> control? Are we now talking about a sensor which is more complex than a >> regular raw bayer sensor? > > I referred to tags defined in EXIF standard, as you may know every tag > is basically an array of spec
Re: [RFC v4] V4L2 API for flash devices
Hi Sakari, On 05/17/2011 10:34 PM, Sakari Ailus wrote: > Sylwester Nawrocki wrote: >> On 05/09/2011 12:11 AM, Sakari Ailus wrote: >>> Sylwester Nawrocki wrote: On 05/07/2011 02:46 PM, Hans Verkuil wrote: > On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote: >> Hi, >> >> This is a fourth proposal for an interface for controlling flash devices >> on the V4L2/v4l2_subdev APIs. ... >> 3. Sensor metadata on frames >> >> >> It'd be useful to be able to read back sensor metadata. If the flash is >> strobed (on sensor hardware) while streaming, it's difficult to know >> otherwise which frame in the stream has been exposed with flash. > > I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer > flag? > That way userspace can tell if that particular frame was taken with > flash. This looks more as a workaround for the problem rather than a good long term solution. It might be tempting to use the buffer flags which seem to be be more or less intended for buffer control. I'd like much more to see a buffer flags to be used to indicate whether an additional plane of (meta)data is carried by the buffer. There seem to be many more parameters, than a single flag indicating whether the frame has been exposed with flash or not, needed to be carried over to user space. But then we might need some standard format of the meta data, perhaps control id/value pairs and possibly a per plane configurable memory type. >>> >>> There are multiple possible approaches for this. >>> >>> For sensors where metadata is register-value pairs, that is, essentially >>> V4L2 control values, I think this should be parsed by the sensor driver. >>> The ISP (camera bridge) driver does receive the data so it'd have to >>> "ask for help" from the sensor driver. >> >> I am inclined to let the ISP drivers parse the data but on the other hand >> it might be difficult to access same DMA buffers in kernel _and_ user space. > > This is just about mapping the buffer to both kernel and user spaces. If > the ISP has an iommu the kernel mapping might already exist if it comes > from vmalloc(). Yes, I know. I was thinking of possibly required different mapping attributes for kernel and user space and the problems on ARM with that. Also as metadata is supposed to occupy only small part of a frame buffer perhaps only one page or so could be mapped in kernel space. I'm referring here mainly to SMIA++ method of storing metadata. In case of sensors I used to work with it wouldn't be necessary to touch the main frame buffers as the metadata is transmitted out of band. > >>> As discussed previously, using V4L2 control events shouldn't probably be >>> the way to go, but I think it's obvious that this is _somehow_ bound to >>> controls, at least control ids. >>> Also as Sakari indicated some sensors adopt custom meta data formats so maybe we need to introduce standard fourcc like IDs for meta data formats? I am not sure whether it is possible to create common description of an image meta data that fits all H/W. >>> >>> I'm not sure either since I know of only one example. That example, i.e. >>> register-value pairs, should be something that I'd assume _some_ other >>> hardware uses as well, but there could exist also hardware which >>> doesn't. This solution might not work on that hardware. >> >> Of course it's hard to find a silver bullet for a hardware we do not know ;) >> >>> >>> If there is metadata which does not associate to V4L2 controls (or >>> ioctls), either existing or not yet existing, then that probably should >>> be parsed by the driver. On the other hand, I can't think of metadata >>> that wouldn't fall under this right now. :-) >> >> Some metadata are arrays of length specific to a given attribute, >> I wonder how to support that with v4l2 controls ? > > Is the metadata something which really isn't associated to any V4L2 > control? Are we now talking about a sensor which is more complex than a > regular raw bayer sensor? I referred to tags defined in EXIF standard, as you may know every tag is basically an array of specific data type. For most tag types the array length is 1 though. Similar problem is solved by the V4L extended string controls API. And yes this kind of tags are mostly produced by more powerful ISPs, with, for instance, 3A or distortion corrections implemented in their firmware. So this is a little bit different situation than a raw sensor with OMAP3 ISP. > >>> Do you know more examples of sensor produced metadata than SMIA++? >> >> The only metadata I've had a bit experience with was regular EXIF tags >> which could be retrieved from ISP through I2C bus. > > That obviously won't map to V4L2 controls. > > This should very likely be just passed to user space as-is as > different... plane? Yes, but I am trying to assess whether it's poss
Re: [RFC v4] V4L2 API for flash devices
Laurent Pinchart wrote: > Hi Sakari, Hi Laurent, > On Tuesday 17 May 2011 22:34:36 Sakari Ailus wrote: >> Sylwester Nawrocki wrote: >>> On 05/09/2011 12:11 AM, Sakari Ailus wrote: Sylwester Nawrocki wrote: > On 05/07/2011 02:46 PM, Hans Verkuil wrote: >> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote: >>> Hi, >>> >>> This is a fourth proposal for an interface for controlling flash >>> devices on the V4L2/v4l2_subdev APIs. >>> >>> I want to thank everyone who have participated to the development of >>> the flash interface. >>> >>> Comments and questions are very, very welcome as always. >>> >>> >>> Changes since v3 [12] >>> = >>> >>> - V4L2_CID_FLASH_STROBE changed to button control, >>> V4L2_CID_FLASH_STROBE_STOP button control added, >>> V4L2_CID_FLASH_STROBE_STATUS boolean control added. >>> >>> - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE. >>> >>> - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE. >>> >>> - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY >>> and V4L2_CID_INDICATOR_INTENSITY renamed to >>> V4L2_CID_FLASH_INDICATOR_INTENSITY. >>> >>> - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future >>> extensions". > > [snip] > >>> 3. Sensor metadata on frames >>> >>> >>> It'd be useful to be able to read back sensor metadata. If the flash >>> is strobed (on sensor hardware) while streaming, it's difficult to >>> know otherwise which frame in the stream has been exposed with >>> flash. >> >> I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer >> flag? >> That way userspace can tell if that particular frame was taken with >> flash. > > This looks more as a workaround for the problem rather than a good long > term solution. It might be tempting to use the buffer flags which seem > to be be more or less intended for buffer control. > I'd like much more to see a buffer flags to be used to indicate whether > an additional plane of (meta)data is carried by the buffer. > There seem to be many more parameters, than a single flag indicating > whether the frame has been exposed with flash or not, needed to be > carried over to user space. > But then we might need some standard format of the meta data, perhaps > control id/value pairs and possibly a per plane configurable memory > type. There are multiple possible approaches for this. For sensors where metadata is register-value pairs, that is, essentially V4L2 control values, I think this should be parsed by the sensor driver. The ISP (camera bridge) driver does receive the data so it'd have to "ask for help" from the sensor driver. >>> >>> I am inclined to let the ISP drivers parse the data but on the other hand >>> it might be difficult to access same DMA buffers in kernel _and_ user >>> space. >> >> This is just about mapping the buffer to both kernel and user spaces. If >> the ISP has an iommu the kernel mapping might already exist if it comes >> from vmalloc(). > > And we're also trying to get rid of that mapping to facilitate cache > management. Any API we create for metadata parsing will need to take > potential > cache-related performances issues into account at the design stage. In this case, it's not necessary to map this memory to user space at all so the kernel mapping would be the only one. -- Sakari Ailus sakari.ai...@maxwell.research.nokia.com -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v4] V4L2 API for flash devices
Hi Sakari, On Tuesday 17 May 2011 22:34:36 Sakari Ailus wrote: > Sylwester Nawrocki wrote: > > On 05/09/2011 12:11 AM, Sakari Ailus wrote: > >> Sylwester Nawrocki wrote: > >>> On 05/07/2011 02:46 PM, Hans Verkuil wrote: > On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote: > > Hi, > > > > This is a fourth proposal for an interface for controlling flash > > devices on the V4L2/v4l2_subdev APIs. > > > > I want to thank everyone who have participated to the development of > > the flash interface. > > > > Comments and questions are very, very welcome as always. > > > > > > Changes since v3 [12] > > = > > > > - V4L2_CID_FLASH_STROBE changed to button control, > > V4L2_CID_FLASH_STROBE_STOP button control added, > > V4L2_CID_FLASH_STROBE_STATUS boolean control added. > > > > - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE. > > > > - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE. > > > > - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY > > and V4L2_CID_INDICATOR_INTENSITY renamed to > > V4L2_CID_FLASH_INDICATOR_INTENSITY. > > > > - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future > > extensions". > >>> > >>> [snip] > >>> > > 3. Sensor metadata on frames > > > > > > It'd be useful to be able to read back sensor metadata. If the flash > > is strobed (on sensor hardware) while streaming, it's difficult to > > know otherwise which frame in the stream has been exposed with > > flash. > > I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer > flag? > That way userspace can tell if that particular frame was taken with > flash. > >>> > >>> This looks more as a workaround for the problem rather than a good long > >>> term solution. It might be tempting to use the buffer flags which seem > >>> to be be more or less intended for buffer control. > >>> I'd like much more to see a buffer flags to be used to indicate whether > >>> an additional plane of (meta)data is carried by the buffer. > >>> There seem to be many more parameters, than a single flag indicating > >>> whether the frame has been exposed with flash or not, needed to be > >>> carried over to user space. > >>> But then we might need some standard format of the meta data, perhaps > >>> control id/value pairs and possibly a per plane configurable memory > >>> type. > >> > >> There are multiple possible approaches for this. > >> > >> For sensors where metadata is register-value pairs, that is, essentially > >> V4L2 control values, I think this should be parsed by the sensor driver. > >> The ISP (camera bridge) driver does receive the data so it'd have to > >> "ask for help" from the sensor driver. > > > > I am inclined to let the ISP drivers parse the data but on the other hand > > it might be difficult to access same DMA buffers in kernel _and_ user > > space. > > This is just about mapping the buffer to both kernel and user spaces. If > the ISP has an iommu the kernel mapping might already exist if it comes > from vmalloc(). And we're also trying to get rid of that mapping to facilitate cache management. Any API we create for metadata parsing will need to take potential cache-related performances issues into account at the design stage. > >> As discussed previously, using V4L2 control events shouldn't probably be > >> the way to go, but I think it's obvious that this is _somehow_ bound to > >> controls, at least control ids. > >> > >>> Also as Sakari indicated some sensors adopt custom meta data formats > >>> so maybe we need to introduce standard fourcc like IDs for meta data > >>> formats? I am not sure whether it is possible to create common > >>> description of an image meta data that fits all H/W. > >> > >> I'm not sure either since I know of only one example. That example, i.e. > >> register-value pairs, should be something that I'd assume _some_ other > >> hardware uses as well, but there could exist also hardware which > >> doesn't. This solution might not work on that hardware. > > > > Of course it's hard to find a silver bullet for a hardware we do not know > > ;) > > > >> If there is metadata which does not associate to V4L2 controls (or > >> ioctls), either existing or not yet existing, then that probably should > >> be parsed by the driver. On the other hand, I can't think of metadata > >> that wouldn't fall under this right now. :-) > > > > Some metadata are arrays of length specific to a given attribute, > > I wonder how to support that with v4l2 controls ? > > Is the metadata something which really isn't associated to any V4L2 > control? Are we now talking about a sensor which is more complex than a > regular raw bayer sensor? > > >> Do you know more examples of sensor produced metadata than SMIA++? > > > > The
Re: [RFC v4] V4L2 API for flash devices
Sylwester Nawrocki wrote: > Hi Sakari, Hi Sylwester, > On 05/09/2011 12:11 AM, Sakari Ailus wrote: >> Sylwester Nawrocki wrote: >>> On 05/07/2011 02:46 PM, Hans Verkuil wrote: On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote: > Hi, > > This is a fourth proposal for an interface for controlling flash devices > on the V4L2/v4l2_subdev APIs. > > I want to thank everyone who have participated to the development of the > flash interface. > > Comments and questions are very, very welcome as always. > > > Changes since v3 [12] > = > > - V4L2_CID_FLASH_STROBE changed to button control, > V4L2_CID_FLASH_STROBE_STOP button control added, > V4L2_CID_FLASH_STROBE_STATUS boolean control added. > > - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE. > > - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE. > > - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY and > V4L2_CID_INDICATOR_INTENSITY renamed to > V4L2_CID_FLASH_INDICATOR_INTENSITY. > > - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future > extensions". > >>> >>> [snip] >>> > > 3. Sensor metadata on frames > > > It'd be useful to be able to read back sensor metadata. If the flash is > strobed (on sensor hardware) while streaming, it's difficult to know > otherwise which frame in the stream has been exposed with flash. I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer flag? That way userspace can tell if that particular frame was taken with flash. >>> >>> This looks more as a workaround for the problem rather than a good long >>> term solution. It might be tempting to use the buffer flags which seem >>> to be be more or less intended for buffer control. >>> I'd like much more to see a buffer flags to be used to indicate whether >>> an additional plane of (meta)data is carried by the buffer. >>> There seem to be many more parameters, than a single flag indicating >>> whether the frame has been exposed with flash or not, needed to be >>> carried over to user space. >>> But then we might need some standard format of the meta data, perhaps >>> control id/value pairs and possibly a per plane configurable memory >>> type. >> >> There are multiple possible approaches for this. >> >> For sensors where metadata is register-value pairs, that is, essentially >> V4L2 control values, I think this should be parsed by the sensor driver. >> The ISP (camera bridge) driver does receive the data so it'd have to >> "ask for help" from the sensor driver. > > I am inclined to let the ISP drivers parse the data but on the other hand > it might be difficult to access same DMA buffers in kernel _and_ user space. This is just about mapping the buffer to both kernel and user spaces. If the ISP has an iommu the kernel mapping might already exist if it comes from vmalloc(). >> As discussed previously, using V4L2 control events shouldn't probably be >> the way to go, but I think it's obvious that this is _somehow_ bound to >> controls, at least control ids. >> >>> Also as Sakari indicated some sensors adopt custom meta data formats >>> so maybe we need to introduce standard fourcc like IDs for meta data >>> formats? I am not sure whether it is possible to create common >>> description of an image meta data that fits all H/W. >> >> I'm not sure either since I know of only one example. That example, i.e. >> register-value pairs, should be something that I'd assume _some_ other >> hardware uses as well, but there could exist also hardware which >> doesn't. This solution might not work on that hardware. > > Of course it's hard to find a silver bullet for a hardware we do not know ;) > >> >> If there is metadata which does not associate to V4L2 controls (or >> ioctls), either existing or not yet existing, then that probably should >> be parsed by the driver. On the other hand, I can't think of metadata >> that wouldn't fall under this right now. :-) > > Some metadata are arrays of length specific to a given attribute, > I wonder how to support that with v4l2 controls ? Is the metadata something which really isn't associated to any V4L2 control? Are we now talking about a sensor which is more complex than a regular raw bayer sensor? >> Do you know more examples of sensor produced metadata than SMIA++? > > The only metadata I've had a bit experience with was regular EXIF tags > which could be retrieved from ISP through I2C bus. That obviously won't map to V4L2 controls. This should very likely be just passed to user space as-is as different... plane? In some cases it's time critical to pass data to user space that otherwise could be associated with a video buffer. I wonder if this case is time critical or not. > These were mostly fixed point arithmetic numbers in [32-bit numerato
Re: [RFC v4] V4L2 API for flash devices
Andy Walls wrote:
> On Sat, 2011-05-07 at 14:46 +0200, Hans Verkuil wrote:
>> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote:
>>>
>>>
>>> enum v4l2_flash_strobe_whence {
>>> V4L2_FLASH_STROBE_WHENCE_SOFTWARE,
>>> V4L2_FLASH_STROBE_WHENCE_EXTERNAL,
>>> };
>>
>> Perhaps use 'type' instead of 'whence'? English isn't my native language,
>> but it sounds pretty archaic to me.
>
> "SOURCE" is better than "WHENCE" here.
>
>
> "whence" is certainly very formal and used very little. "whence" likely
> still gets some use in English, simply because a terse synonym doesn't
> exist.
>
> The problem with using whence is that many English speakers won't know
> its correct definition.
>
> "whence" means "from what source, origin, or place"
>
> In your use here, the implicit "from" in the definition of whence is
> essential. However, most (American) English speakers that I know think
> "whence" simply means "where".
Thanks for the feedback, Andy!
WHENCE has since changed to MODE (or at least should have been), but I
think SOURCE is even better. I'll switch to that.
Regards,
--
Sakari Ailus
sakari.ai...@maxwell.research.nokia.com
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v4] V4L2 API for flash devices
On Sat, 2011-05-07 at 14:46 +0200, Hans Verkuil wrote:
> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote:
> >
> >
> > enum v4l2_flash_strobe_whence {
> > V4L2_FLASH_STROBE_WHENCE_SOFTWARE,
> > V4L2_FLASH_STROBE_WHENCE_EXTERNAL,
> > };
>
> Perhaps use 'type' instead of 'whence'? English isn't my native language,
> but it sounds pretty archaic to me.
"SOURCE" is better than "WHENCE" here.
"whence" is certainly very formal and used very little. "whence" likely
still gets some use in English, simply because a terse synonym doesn't
exist.
The problem with using whence is that many English speakers won't know
its correct definition.
"whence" means "from what source, origin, or place"
In your use here, the implicit "from" in the definition of whence is
essential. However, most (American) English speakers that I know think
"whence" simply means "where".
Regards,
Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v4] V4L2 API for flash devices
Hi Sakari, On 05/09/2011 12:11 AM, Sakari Ailus wrote: > Sylwester Nawrocki wrote: >> On 05/07/2011 02:46 PM, Hans Verkuil wrote: >>> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote: Hi, This is a fourth proposal for an interface for controlling flash devices on the V4L2/v4l2_subdev APIs. I want to thank everyone who have participated to the development of the flash interface. Comments and questions are very, very welcome as always. Changes since v3 [12] = - V4L2_CID_FLASH_STROBE changed to button control, V4L2_CID_FLASH_STROBE_STOP button control added, V4L2_CID_FLASH_STROBE_STATUS boolean control added. - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE. - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE. - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY and V4L2_CID_INDICATOR_INTENSITY renamed to V4L2_CID_FLASH_INDICATOR_INTENSITY. - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future extensions". >> >> [snip] >> 3. Sensor metadata on frames It'd be useful to be able to read back sensor metadata. If the flash is strobed (on sensor hardware) while streaming, it's difficult to know otherwise which frame in the stream has been exposed with flash. >>> >>> I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer >>> flag? >>> That way userspace can tell if that particular frame was taken with >>> flash. >> >> This looks more as a workaround for the problem rather than a good long >> term solution. It might be tempting to use the buffer flags which seem >> to be be more or less intended for buffer control. >> I'd like much more to see a buffer flags to be used to indicate whether >> an additional plane of (meta)data is carried by the buffer. >> There seem to be many more parameters, than a single flag indicating >> whether the frame has been exposed with flash or not, needed to be >> carried over to user space. >> But then we might need some standard format of the meta data, perhaps >> control id/value pairs and possibly a per plane configurable memory >> type. > > There are multiple possible approaches for this. > > For sensors where metadata is register-value pairs, that is, essentially > V4L2 control values, I think this should be parsed by the sensor driver. > The ISP (camera bridge) driver does receive the data so it'd have to > "ask for help" from the sensor driver. I am inclined to let the ISP drivers parse the data but on the other hand it might be difficult to access same DMA buffers in kernel _and_ user space. > > As discussed previously, using V4L2 control events shouldn't probably be > the way to go, but I think it's obvious that this is _somehow_ bound to > controls, at least control ids. > >> Also as Sakari indicated some sensors adopt custom meta data formats >> so maybe we need to introduce standard fourcc like IDs for meta data >> formats? I am not sure whether it is possible to create common >> description of an image meta data that fits all H/W. > > I'm not sure either since I know of only one example. That example, i.e. > register-value pairs, should be something that I'd assume _some_ other > hardware uses as well, but there could exist also hardware which > doesn't. This solution might not work on that hardware. Of course it's hard to find a silver bullet for a hardware we do not know ;) > > If there is metadata which does not associate to V4L2 controls (or > ioctls), either existing or not yet existing, then that probably should > be parsed by the driver. On the other hand, I can't think of metadata > that wouldn't fall under this right now. :-) Some metadata are arrays of length specific to a given attribute, I wonder how to support that with v4l2 controls ? > > Do you know more examples of sensor produced metadata than SMIA++? The only metadata I've had a bit experience with was regular EXIF tags which could be retrieved from ISP through I2C bus. These were mostly fixed point arithmetic numbers in [32-bit numerator/ 32-bit denominator] form carrying exposure time, shutter speed, aperture, brightness, flash, etc. information. The tags could be read from ISP after it buffered a frame in its memory and processed it. In case of a JPEG image format the tags can be embedded into the main image file. But the image processors not always supported that so we used to have an ioctl for the purpose of retrieving the metadata in user space. In some cases it is desired to read data directly from the driver rather than parsing a relatively large buffer. It would be good to have a uniform interface for passing such data to applications. I think in that particular use case a control id/value pair sequences would do. -- Regards, Sylwester -- To unsubscribe from this list: send the
Re: [RFC v4] V4L2 API for flash devices
Sylwester Nawrocki wrote: > Hi Hans, Hi Sylwester, Hans, > On 05/07/2011 02:46 PM, Hans Verkuil wrote: >> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote: >>> Hi, >>> >>> This is a fourth proposal for an interface for controlling flash devices >>> on the V4L2/v4l2_subdev APIs. >>> >>> I want to thank everyone who have participated to the development of the >>> flash interface. >>> >>> Comments and questions are very, very welcome as always. >>> >>> >>> Changes since v3 [12] >>> = >>> >>> - V4L2_CID_FLASH_STROBE changed to button control, >>> V4L2_CID_FLASH_STROBE_STOP button control added, >>> V4L2_CID_FLASH_STROBE_STATUS boolean control added. >>> >>> - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE. >>> >>> - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE. >>> >>> - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY and >>> V4L2_CID_INDICATOR_INTENSITY renamed to >>> V4L2_CID_FLASH_INDICATOR_INTENSITY. >>> >>> - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future >>> extensions". >>> > > [snip] > >>> >>> 3. Sensor metadata on frames >>> >>> >>> It'd be useful to be able to read back sensor metadata. If the flash is >>> strobed (on sensor hardware) while streaming, it's difficult to know >>> otherwise which frame in the stream has been exposed with flash. >> >> I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer >> flag? >> That way userspace can tell if that particular frame was taken with >> flash. > > This looks more as a workaround for the problem rather than a good long > term solution. It might be tempting to use the buffer flags which seem > to be be more or less intended for buffer control. > I'd like much more to see a buffer flags to be used to indicate whether > an additional plane of (meta)data is carried by the buffer. > There seem to be many more parameters, than a single flag indicating > whether the frame has been exposed with flash or not, needed to be > carried over to user space. > But then we might need some standard format of the meta data, perhaps > control id/value pairs and possibly a per plane configurable memory > type. There are multiple possible approaches for this. For sensors where metadata is register-value pairs, that is, essentially V4L2 control values, I think this should be parsed by the sensor driver. The ISP (camera bridge) driver does receive the data so it'd have to "ask for help" from the sensor driver. As discussed previously, using V4L2 control events shouldn't probably be the way to go, but I think it's obvious that this is _somehow_ bound to controls, at least control ids. > Also as Sakari indicated some sensors adopt custom meta data formats > so maybe we need to introduce standard fourcc like IDs for meta data > formats? I am not sure whether it is possible to create common > description of an image meta data that fits all H/W. I'm not sure either since I know of only one example. That example, i.e. register-value pairs, should be something that I'd assume _some_ other hardware uses as well, but there could exist also hardware which doesn't. This solution might not work on that hardware. If there is metadata which does not associate to V4L2 controls (or ioctls), either existing or not yet existing, then that probably should be parsed by the driver. On the other hand, I can't think of metadata that wouldn't fall under this right now. :-) Do you know more examples of sensor produced metadata than SMIA++? Regards, -- Sakari Ailus sakari.ai...@maxwell.research.nokia.com -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v4] V4L2 API for flash devices
Hans Verkuil wrote:
[clip]
>> V4L2_CID_FLASH_STROBE (button; wo; LED, xenon)
>>
>> Strobe the flash using software strobe from the host, typically over I2C
>> or a GPIO. The flash is NOT synchronised to sensor pixel are exposure
>> since the command is given asynchronously. Alternatively, if the flash
>> controller is a master in the system, the sensor exposure may be
>> triggered based on software strobe.
>
> I would like a comment here that the driver will always stop the strobe
> to prevent hardware damage, but that STROBE_STOP can be called to shut
> down the strobe earlier.
>
> Also, how to protect against nastiness like this:
>
> struct v4l2_control ctrl = { V4L2_CID_FLASH_STROBE, 0 };
>
> for (;;)
> ioctl(fd, VIDIOC_S_CTRL, &ctrl);
>
> Return -EBUSY as long as either a strobe is in progress or if it is unsafe to
> start a new one?
This is not implemented in the current adp1653 driver but yes, it should
be done. However, it isn't simple to implement in a way that it would
still allow the optimal use of the flash.
I'll add that to the todo list for the driver.
Modelling of the flash LED cooldown period should be relatively
independent of the board, actual LED and the flash chip, so this might
make a nice library in the future.
>>
>>
>> V4L2_CID_FLASH_STROBE_STOP (button; wo; LED)
>>
>> This control may be used to shut down the strobe immediately.
>>
>>
>> V4L2_CID_FLASH_STROBE_STATUS (boolean; ro; LED)
>>
>> This contol may be used to query the status of the strobe (on/off).
>>
>>
>> V4L2_CID_FLASH_STROBE_MODE (menu; rw; LED)
>>
>> Use hardware or software strobe. If hardware strobe is selected, the
>> flash controller is a slave in the system where the sensor produces the
>> strobe signal to the flash.
>>
>> In this case the flash controller setup is limited to programming strobe
>> timeout and power (LED flash) and the sensor controls the timing and
>> length of the strobe.
>>
>> enum v4l2_flash_strobe_whence {
>> V4L2_FLASH_STROBE_WHENCE_SOFTWARE,
>> V4L2_FLASH_STROBE_WHENCE_EXTERNAL,
>> };
>
> Perhaps use 'type' instead of 'whence'? English isn't my native language,
> but it sounds pretty archaic to me.
Neither it is mine. I actually forgot to change the name of the enum to
v4l2_flash_strobe_mode so it would match the control.
What do you think of that?
>>
>>
>> V4L2_CID_FLASH_TIMEOUT (integer; rw; LED)
>>
>> The flash controller provides timeout functionality to shut down the led
>> in case the host fails to do that. For hardware strobe, this is the
>> maximum amount of time the flash should stay on, and the purpose of the
>> setting is to prevent the LED from catching fire.
>
> It is a bit ambiguous. I assume you mean to say that the application can
> set the timeout to any value less than the maximum of the control (which
> is the maximum allowed by the hardware).
Yes; that's what I intended. I'll attempt to formulate it better in the
forthcoming patches. :-)
[clip]
>> Open questions
>> ==
>>
>> 1. Flash LED mode control
>> -
>>
>> Should the flash led mode control be a single control or a set of
>> controls? A single control would make it easier for application to
>> choose between different modes, but on the other hand limits future
>> extensibility if there would have to be further splitting of the modes. [8]
>
> I think I like the current proposal best.
I agree.
>> 3. Units
>>
>>
>> Which units should e.g. V4L2_CID_FLASH_TIMEOUT be using? It'd be very
>> useful to have standard unit on this control, like ms or µs.
>
> This should be standardized. The question is how flexible we want this.
> I would definitely go for µs or perhaps even ns. But what should the
> maximum duration be? Are there extreme cases where you can have a really
> long flash in special types of photography? Perhaps this should be a 64-bit
> number?
On xenon flashes the strobe strength is controlled by strobe duration,
as far as I understand. I think it would be better to hide that under
the V4L2_CID_FLASH_INTENSITY, though.
The timeout on adp1653 the timeout is really coarse, there are 16
different values for it.
On other chips more precise control could be possible.
>> Some controls, like V4L2_CID_FLASH_INTENSITY, can't easily use a
>> standard unit. The luminous output depends on the LED connected (you
>> could use an incandescent lightbulb too, I suppose?) to the flash
>> controller. These controls typically control the current supplied by the
>> flash controller.
>>
>> The timing controls on the sensor could use pixels (or lines) since
>> these are the units the sensor uses itself. Converting between pixels
>> (or lines) and SI units is trivial since the pixel clock is known in the
>> user space.
>>
>> The flash chip, on the other hand, has no knowledge of sensor pixel
>> clock, so it should use SI units. Also, the timing control currently in
>> the flash chip itself is very coarse.
>>
>> PROPOSAL: If the co
Re: [RFC v4] V4L2 API for flash devices
Hi Hans, On 05/07/2011 02:46 PM, Hans Verkuil wrote: On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote: Hi, This is a fourth proposal for an interface for controlling flash devices on the V4L2/v4l2_subdev APIs. I want to thank everyone who have participated to the development of the flash interface. Comments and questions are very, very welcome as always. Changes since v3 [12] = - V4L2_CID_FLASH_STROBE changed to button control, V4L2_CID_FLASH_STROBE_STOP button control added, V4L2_CID_FLASH_STROBE_STATUS boolean control added. - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE. - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE. - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY and V4L2_CID_INDICATOR_INTENSITY renamed to V4L2_CID_FLASH_INDICATOR_INTENSITY. - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future extensions". [snip] 3. Sensor metadata on frames It'd be useful to be able to read back sensor metadata. If the flash is strobed (on sensor hardware) while streaming, it's difficult to know otherwise which frame in the stream has been exposed with flash. I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer flag? That way userspace can tell if that particular frame was taken with flash. This looks more as a workaround for the problem rather than a good long term solution. It might be tempting to use the buffer flags which seem to be be more or less intended for buffer control. I'd like much more to see a buffer flags to be used to indicate whether an additional plane of (meta)data is carried by the buffer. There seem to be many more parameters, than a single flag indicating whether the frame has been exposed with flash or not, needed to be carried over to user space. But then we might need some standard format of the meta data, perhaps control id/value pairs and possibly a per plane configurable memory type. Also as Sakari indicated some sensors adopt custom meta data formats so maybe we need to introduce standard fourcc like IDs for meta data formats? I am not sure whether it is possible to create common description of an image meta data that fits all H/W. -- Regards, Sylwester -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v4] V4L2 API for flash devices
On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote: > Hi, > > This is a fourth proposal for an interface for controlling flash devices > on the V4L2/v4l2_subdev APIs. > > I want to thank everyone who have participated to the development of the > flash interface. > > Comments and questions are very, very welcome as always. > > > Changes since v3 [12] > = > > - V4L2_CID_FLASH_STROBE changed to button control, > V4L2_CID_FLASH_STROBE_STOP button control added, > V4L2_CID_FLASH_STROBE_STATUS boolean control added. > > - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE. > > - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE. > > - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY and > V4L2_CID_INDICATOR_INTENSITY renamed to V4L2_CID_FLASH_INDICATOR_INTENSITY. > > - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future > extensions". > > > Changes since v2 [9] > > > - Rearranged proposed controls. V4L2_CID_FLASH_LED_MODE is now the first > control. > > - Added an open question on naming of indicator and torch controls. > > - V4L2_CID_FLASH_STROBE_MODE renamed to V4L2_CID_FLASH_STROBE_WHENCE. > V4L2_CID_FLASH_EXTERNAL_STROBE_WHENCE renamed to > V4L2_CID_FLASH_EXTERNAL_STROBE_MODE. > > - Removed CID_ from V4L2_CID_FLASH_EXTERNAL_STROBE_MODE values. > > - Added a new use case based on [11]: Synchronised LED flash (hardware > strobe, timed exposure). > > - Added section on possible future extensions. > > - Complemented the open question on units. > > > Changes since v1 [7] > > > - V4L2_FLASH_STROBE_MODE_EXT_STROBE renamed to > V4L2_FLASH_STROBE_MODE_EXTERNAL. > > - V4L2_CID_FLASH_STROBE control changed from button to bool. > > - Removed suggestion of adding V4L2_CID_FLASH_DURATION. > V4L2_CID_FLASH_TIMEOUT is used as hardware timeout. > > - Added control access info (ro/rw). > > - V4L2_FLASH_MODE_NONE added, V4L2_FLASH_LED_MODE_FLASH no longer forced > as 1 in enum. > > - Bits use (1 << x) instead of 0x00... format. > > - Added an open question on flash LED mode controls. > > - Added an open question on a new control: > V4L2_CID_FLASH_EXTERNAL_STROBE_WHENCE. > > - Added an open question on control units. > > > Scope > = > > This RFC is focused mostly on the ADP1653 [1] and similar chips [2, 3] > which provides following functionality. [2, 3] mostly differ on the > available faults --- for example, there are faults also for the > indicator LED. > > - High power LED output (flash or torch modes) > - Low power indicator LED output (a.k.a. privacy light) > - Programmable flash timeout > - Software and hardware strobe > - Fault detection > - Overvoltage > - Overtemperature > - Short circuit > - Timeout > - Programmable current (both high-power and indicator LEDs) > > If anyone else is aware of hardware which significantly differs from > these and does not get served well under the proposed interface, please > tell about it. > > This RFC does NOT address the synchronisation of the flash to a given > frame since this task is typically performed by the sensor through a > strobe signal. The host does not have enough information for this --- > exact timing information on the exposure of the sensor pixel array. In > this case the flash synchronisation is visible to the flash controller > as the hardware strobe originating from the sensor. > > Flash synchronisation requires > > 1) flash control capability from the sensor including a strobe output, > 2) strobe input in the flash controller, > 3) (optionally) ability to program sensor parameters at given frame, > such as flash strobe, and > 4) ability to read back metadata produced by the sensor related to a > given frame. This should include whether the frame is exposed with > flash, i.e. the sensor's flash strobe output. > > Since we have little examples of both in terms of hardware support, > which is in practice required, it was decided to postpone the interface > specification for now. [6] > > Xenon flash controllers exist but I don't have a specific example of > those. Typically the interface is quite simple. Gpio pins for charge and > strobe. The length of the strobe signal determines the strength of the > flash pulse. The strobe is controlled by the sensor as for LED flash if > it is hardware based. > > See "Possible future extensions" section below for more. > > > Known use cases > === > > The use case listed below concentrate on using a flash in a mobile > device, for example in a mobile phone. The use cases could be somewhat > different in devices the primary use of which is camera. > > Unsynchronised LED flash (software strobe) > -- > > Unsynchronised LED flash is controlled directly by the host as the > sensor. The flash must be enabled by the host before the exposure of the > image starts and disabled once it ends. The
[RFC v4] V4L2 API for flash devices
Hi, This is a fourth proposal for an interface for controlling flash devices on the V4L2/v4l2_subdev APIs. I want to thank everyone who have participated to the development of the flash interface. Comments and questions are very, very welcome as always. Changes since v3 [12] = - V4L2_CID_FLASH_STROBE changed to button control, V4L2_CID_FLASH_STROBE_STOP button control added, V4L2_CID_FLASH_STROBE_STATUS boolean control added. - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE. - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE. - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY and V4L2_CID_INDICATOR_INTENSITY renamed to V4L2_CID_FLASH_INDICATOR_INTENSITY. - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future extensions". Changes since v2 [9] - Rearranged proposed controls. V4L2_CID_FLASH_LED_MODE is now the first control. - Added an open question on naming of indicator and torch controls. - V4L2_CID_FLASH_STROBE_MODE renamed to V4L2_CID_FLASH_STROBE_WHENCE. V4L2_CID_FLASH_EXTERNAL_STROBE_WHENCE renamed to V4L2_CID_FLASH_EXTERNAL_STROBE_MODE. - Removed CID_ from V4L2_CID_FLASH_EXTERNAL_STROBE_MODE values. - Added a new use case based on [11]: Synchronised LED flash (hardware strobe, timed exposure). - Added section on possible future extensions. - Complemented the open question on units. Changes since v1 [7] - V4L2_FLASH_STROBE_MODE_EXT_STROBE renamed to V4L2_FLASH_STROBE_MODE_EXTERNAL. - V4L2_CID_FLASH_STROBE control changed from button to bool. - Removed suggestion of adding V4L2_CID_FLASH_DURATION. V4L2_CID_FLASH_TIMEOUT is used as hardware timeout. - Added control access info (ro/rw). - V4L2_FLASH_MODE_NONE added, V4L2_FLASH_LED_MODE_FLASH no longer forced as 1 in enum. - Bits use (1 << x) instead of 0x00... format. - Added an open question on flash LED mode controls. - Added an open question on a new control: V4L2_CID_FLASH_EXTERNAL_STROBE_WHENCE. - Added an open question on control units. Scope = This RFC is focused mostly on the ADP1653 [1] and similar chips [2, 3] which provides following functionality. [2, 3] mostly differ on the available faults --- for example, there are faults also for the indicator LED. - High power LED output (flash or torch modes) - Low power indicator LED output (a.k.a. privacy light) - Programmable flash timeout - Software and hardware strobe - Fault detection - Overvoltage - Overtemperature - Short circuit - Timeout - Programmable current (both high-power and indicator LEDs) If anyone else is aware of hardware which significantly differs from these and does not get served well under the proposed interface, please tell about it. This RFC does NOT address the synchronisation of the flash to a given frame since this task is typically performed by the sensor through a strobe signal. The host does not have enough information for this --- exact timing information on the exposure of the sensor pixel array. In this case the flash synchronisation is visible to the flash controller as the hardware strobe originating from the sensor. Flash synchronisation requires 1) flash control capability from the sensor including a strobe output, 2) strobe input in the flash controller, 3) (optionally) ability to program sensor parameters at given frame, such as flash strobe, and 4) ability to read back metadata produced by the sensor related to a given frame. This should include whether the frame is exposed with flash, i.e. the sensor's flash strobe output. Since we have little examples of both in terms of hardware support, which is in practice required, it was decided to postpone the interface specification for now. [6] Xenon flash controllers exist but I don't have a specific example of those. Typically the interface is quite simple. Gpio pins for charge and strobe. The length of the strobe signal determines the strength of the flash pulse. The strobe is controlled by the sensor as for LED flash if it is hardware based. See "Possible future extensions" section below for more. Known use cases === The use case listed below concentrate on using a flash in a mobile device, for example in a mobile phone. The use cases could be somewhat different in devices the primary use of which is camera. Unsynchronised LED flash (software strobe) -- Unsynchronised LED flash is controlled directly by the host as the sensor. The flash must be enabled by the host before the exposure of the image starts and disabled once it ends. The host is fully responsible for the timing of the flash. Example of such device: Nokia N900. Synchronised LED flash (hardware strobe) The synchronised LED flash is pre-programmed by the host (power and timeout) but controlled by the sensor through a strobe signal from the sensor to the fla
