Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
Hi, On Mon, Dec 12, 2011 at 1:43 AM, Sylwester Nawrocki snj...@gmail.com For OMAP4 FD, it is not needed to include FD into MC framework since a intermediate buffer is always required. If your HW doesn't belong to this case, what is the output of your HW FD in the link? Also sounds FD results may not be needed at all for use space application in the case. The result data is similar to OMAP4 one, plus a few other attributes. User buffers may be filled by other than FD device driver. OK. Could you provide some practical use cases about these? As above, and any device with a camera that controls something and makes decision according to presence of human face in his view. Sounds a reasonable case, :-) If FD result is associated with a frame, how can user space get the frame seq if no v4l2 buffer is involved? Without a frame sequence, it is a bit difficult to retrieve FD results from user space. If you pass image data in memory buffers from user space, yes, it could be impossible. It is easy to get the frame sequence from v4l2_buffer for the case too, :-) Not really, still v4l2_buffer may be used by other (sub)driver within same video processing pipeline. OK. A related question: how can we make one application to support the two kinds of devices(input from user space data as OMAP4, input from SoC bus as Samsung) at the same time? Maybe some capability info is to be exported to user space? or other suggestions? And will your Samsung FD HW support to detect faces from memory? or just only detect from SoC bus? It will be included in the FD result... or in a dedicated v4l2 event data structure. More important, at the end of the day, we'll be getting buffers with image data at some stage of a video pipeline, which would contain same frame identifier (I think we can ignore v4l2_buffer.field for FD purpose). OK, I will associate FD result with frame identifier, and not invent a dedicated v4l2 event for query frame seq now until a specific requirement for it is proposed. I will convert/integrate recent discussions into patches of v2 for further review, and sub device support will be provided. But before starting to do it, I am still not clear how to integrate FD into MC framework. I understand FD sub device is only a media entity, so how can FD sub device find the media device(struct media_device)? or just needn't to care about it now? thanks, -- Ming Lei -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
Hi Ming, It's maybe late, but I want to suggest one thing about FD API. This OMAP FD block looks detection ability of just face. But, It's possible to occur another device which can detect specific object or patterns. Moreover, this API can expand object recognition area. So, I think it's good to change the API name like v4l2_recog. Actually, I'm preparing similar control class for mainline with m5mols camera sensor driver. The m5mols camera sensor has the function about face detection. But, I has experienced about Robot Recognition, and I remember the image processing chip which can detect spefic pattern. So, I hesitated naming the API(control or ioctl whatever) with face. It can be possible to provide just object or pattern, not face. Even user library on windows, there is famous OpenCV. And this is also support not only face, but also object. The function of OMAP FDIF looks like m5mols ISP's one. please understand I don't have experience about OMAP AP. But, I can tell you it's better to use the name object recognition, not the face detection, for any other device or driver. In a few days, I'll share the CIDs I have thought for m5mols driver. And, I hope to discuss about this with OMAP FDIF. Thank you. Regards, Heungjun Kim -Original Message- From: linux-media-ow...@vger.kernel.org [mailto:linux-media- ow...@vger.kernel.org] On Behalf Of Ming Lei Sent: Monday, December 12, 2011 6:50 PM To: Sylwester Nawrocki Cc: linux-o...@vger.kernel.org; linux-arm-ker...@lists.infradead.org; linux- ker...@vger.kernel.org; linux-media@vger.kernel.org Subject: Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module Hi, On Mon, Dec 12, 2011 at 1:43 AM, Sylwester Nawrocki snj...@gmail.com For OMAP4 FD, it is not needed to include FD into MC framework since a intermediate buffer is always required. If your HW doesn't belong to this case, what is the output of your HW FD in the link? Also sounds FD results may not be needed at all for use space application in the case. The result data is similar to OMAP4 one, plus a few other attributes. User buffers may be filled by other than FD device driver. OK. Could you provide some practical use cases about these? As above, and any device with a camera that controls something and makes decision according to presence of human face in his view. Sounds a reasonable case, :-) If FD result is associated with a frame, how can user space get the frame seq if no v4l2 buffer is involved? Without a frame sequence, it is a bit difficult to retrieve FD results from user space. If you pass image data in memory buffers from user space, yes, it could be impossible. It is easy to get the frame sequence from v4l2_buffer for the case too, :-) Not really, still v4l2_buffer may be used by other (sub)driver within same video processing pipeline. OK. A related question: how can we make one application to support the two kinds of devices(input from user space data as OMAP4, input from SoC bus as Samsung) at the same time? Maybe some capability info is to be exported to user space? or other suggestions? And will your Samsung FD HW support to detect faces from memory? or just only detect from SoC bus? It will be included in the FD result... or in a dedicated v4l2 event data structure. More important, at the end of the day, we'll be getting buffers with image data at some stage of a video pipeline, which would contain same frame identifier (I think we can ignore v4l2_buffer.field for FD purpose). OK, I will associate FD result with frame identifier, and not invent a dedicated v4l2 event for query frame seq now until a specific requirement for it is proposed. I will convert/integrate recent discussions into patches of v2 for further review, and sub device support will be provided. But before starting to do it, I am still not clear how to integrate FD into MC framework. I understand FD sub device is only a media entity, so how can FD sub device find the media device(struct media_device)? or just needn't to care about it now? thanks, -- Ming Lei -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
Hi, On 12/12/2011 10:49 AM, Ming Lei wrote: If FD result is associated with a frame, how can user space get the frame seq if no v4l2 buffer is involved? Without a frame sequence, it is a bit difficult to retrieve FD results from user space. If you pass image data in memory buffers from user space, yes, it could be impossible. It is easy to get the frame sequence from v4l2_buffer for the case too, :-) Oops, have mixed something up ;) Not really, still v4l2_buffer may be used by other (sub)driver within same video processing pipeline. OK. A related question: how can we make one application to support the two kinds of devices(input from user space data as OMAP4, input from SoC bus as Samsung) at the same time? Maybe some capability info is to be exported to user space? or other suggestions? Good question. To let applications know that a video device is not just an ordinary video output device I suppose we'll need a new object detection/recognition capability flag. V4L2_CAPS_OBJECT_DETECTION, V4L2_CAP_OBJECT_RECOGNITION or something similar. It's probably safe to assume the SoC will support either input method at time, not both simultaneously. Then it could be, for example, modelled with a video node and a subdev: user image data video capture for FDstream +-+ +-+ | /dev/video0 | | /dev/video0 | | OUTPUT| | CAPTURE| +--+--+ +--+--+ || v^ ..--++--+--+--+-+| image | link0 | pad | face | pad || sensor +---+ 0 | detector | 1 || sub-dev +---+ | | sub-dev | || ..--+| +-+--+-+| | | | +--+--++-+ | | | pad | image | pad | | +---+ 0 | processing | 1 +--+ link1 | | sub-dev| | +-++-+ User space can control state of link0. If the link is active (streaming) then access to /dev/video0 would be blocked by the driver, e.g. with EBUSY errno. This means that only one data source can be attached to an input pad (pad0). These are intrinsic properties of Media Controller/v4l2 subdev API. And will your Samsung FD HW support to detect faces from memory? or just only detect from SoC bus? I think we should be prepared for both configurations, as on a diagram above. [...] OK, I will associate FD result with frame identifier, and not invent a dedicated v4l2 event for query frame seq now until a specific requirement for it is proposed. I will convert/integrate recent discussions into patches of v2 for further Sure, sounds like a good idea. review, and sub device support will be provided. But before starting to do it, I am still not clear how to integrate FD into MC framework. I understand FD sub device is only a media entity, so how can FD sub device find the media device(struct media_device)? or just needn't to care about it now? The media device driver will register all entities that belong to it and will create relevant links between entities' pads, which then can be activated by applications. How the entities are registered is another topic, that we don't need to be concerned about at the moment. If you're curious see drivers/media/video/omap3isp or driver/media/video/s5p-fimc for example media device drivers. -- Regards, Sylwester -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
Hi, On Mon, Dec 12, 2011 at 8:08 PM, HeungJun, Kim riverful@samsung.com wrote: Hi Ming, It's maybe late, but I want to suggest one thing about FD API. This OMAP FD block looks detection ability of just face. But, It's possible to occur another device which can detect specific object or patterns. Moreover, this API can expand object recognition area. So, I think it's good to change the API name like v4l2_recog. IMO, object detection is better, at least now OMAP4 and samsung has face detection IP module, and face recognition is often done on results of face detection and more complicated interfaces will be involved. Actually, I'm preparing similar control class for mainline with m5mols camera sensor driver. The m5mols camera sensor has the function about face detection. But, I has experienced about Robot Recognition, and I remember the image processing chip which can detect spefic pattern. So, I hesitated naming the API(control or ioctl whatever) with face. It can be possible to provide just object or pattern, not face. Even user library on windows, there is famous OpenCV. And this is also support not only face, but also object. Yes, object is better than face, and we can use enum flag to describe that the objects detected are which kind of objects. In fact, I plan to rename the face detection generic driver as object detection generic driver and let hardware driver to handle the object detection details. The function of OMAP FDIF looks like m5mols ISP's one. please understand I don't have experience about OMAP AP. But, I can tell you it's better to use the name object recognition, not the face detection, for any other device or driver. In a few days, I'll share the CIDs I have thought for m5mols driver. And, I hope to discuss about this with OMAP FDIF. You have been doing it already, :-) thanks, -- Ming Lei Thank you. Regards, Heungjun Kim -Original Message- From: linux-media-ow...@vger.kernel.org [mailto:linux-media- ow...@vger.kernel.org] On Behalf Of Ming Lei Sent: Monday, December 12, 2011 6:50 PM To: Sylwester Nawrocki Cc: linux-o...@vger.kernel.org; linux-arm-ker...@lists.infradead.org; linux- ker...@vger.kernel.org; linux-media@vger.kernel.org Subject: Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module Hi, On Mon, Dec 12, 2011 at 1:43 AM, Sylwester Nawrocki snj...@gmail.com For OMAP4 FD, it is not needed to include FD into MC framework since a intermediate buffer is always required. If your HW doesn't belong to this case, what is the output of your HW FD in the link? Also sounds FD results may not be needed at all for use space application in the case. The result data is similar to OMAP4 one, plus a few other attributes. User buffers may be filled by other than FD device driver. OK. Could you provide some practical use cases about these? As above, and any device with a camera that controls something and makes decision according to presence of human face in his view. Sounds a reasonable case, :-) If FD result is associated with a frame, how can user space get the frame seq if no v4l2 buffer is involved? Without a frame sequence, it is a bit difficult to retrieve FD results from user space. If you pass image data in memory buffers from user space, yes, it could be impossible. It is easy to get the frame sequence from v4l2_buffer for the case too, :-) Not really, still v4l2_buffer may be used by other (sub)driver within same video processing pipeline. OK. A related question: how can we make one application to support the two kinds of devices(input from user space data as OMAP4, input from SoC bus as Samsung) at the same time? Maybe some capability info is to be exported to user space? or other suggestions? And will your Samsung FD HW support to detect faces from memory? or just only detect from SoC bus? It will be included in the FD result... or in a dedicated v4l2 event data structure. More important, at the end of the day, we'll be getting buffers with image data at some stage of a video pipeline, which would contain same frame identifier (I think we can ignore v4l2_buffer.field for FD purpose). OK, I will associate FD result with frame identifier, and not invent a dedicated v4l2 event for query frame seq now until a specific requirement for it is proposed. I will convert/integrate recent discussions into patches of v2 for further review, and sub device support will be provided. But before starting to do it, I am still not clear how to integrate FD into MC framework. I understand FD sub device is only a media entity, so how can FD sub device find the media device(struct media_device)? or just needn't to care about it now? thanks, -- Ming Lei -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http
RE: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
Hi Ming and Sylwester, Thanks for the reply. -Original Message- From: Ming Lei [mailto:ming@canonical.com] Sent: Tuesday, December 13, 2011 1:02 PM To: HeungJun, Kim Cc: Sylwester Nawrocki; linux-o...@vger.kernel.org; linux-arm- ker...@lists.infradead.org; linux-ker...@vger.kernel.org; linux- me...@vger.kernel.org Subject: Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module Hi, On Mon, Dec 12, 2011 at 8:08 PM, HeungJun, Kim riverful@samsung.com wrote: Hi Ming, It's maybe late, but I want to suggest one thing about FD API. This OMAP FD block looks detection ability of just face. But, It's possible to occur another device which can detect specific object or patterns. Moreover, this API can expand object recognition area. So, I think it's good to change the API name like v4l2_recog. IMO, object detection is better, at least now OMAP4 and samsung has face detection IP module, and face recognition is often done on results of face detection and more complicated interfaces will be involved. Actually, the detection point is different, I guess. The OMAP has the detection block separately, named FDIF. But, Samsung Exynos doing detection process with externel sensor - m5mols, in our case. This sensor(m5mols) has ISP function, and these ISP can process detection. The expert of FIMC is Sylwester. Probably, he can tell the differences between both in more details. :) Actually, I'm preparing similar control class for mainline with m5mols camera sensor driver. The m5mols camera sensor has the function about face detection. But, I has experienced about Robot Recognition, and I remember the image processing chip which can detect spefic pattern. So, I hesitated naming the API(control or ioctl whatever) with face. It can be possible to provide just object or pattern, not face. Even user library on windows, there is famous OpenCV. And this is also support not only face, but also object. Yes, object is better than face, and we can use enum flag to describe that the objects detected are which kind of objects. In fact, I plan to rename the face detection generic driver as object detection generic driver and let hardware driver to handle the object detection details. The function of OMAP FDIF looks like m5mols ISP's one. please understand I don't have experience about OMAP AP. But, I can tell you it's better to use the name object recognition, not the face detection, for any other device or driver. In a few days, I'll share the CIDs I have thought for m5mols driver. And, I hope to discuss about this with OMAP FDIF. You have been doing it already, :-) Yeah, actually I did. :) But, until I see OMAP FDIF case, I did not recognize AP(like OMAP) can have detection capability. :) So, although I did not think about at that time, I also think we should re-consider this case for now. As I look around your patch quickly, the functions is very similar with ours. (even detection of left/right eye) So, the problem is there are two ways to proceed recognition. - Does it handle at the level of IOCTLs? or CIDs? If the detection algorithm is proceeded at the level of HW block, it's right the platform driver provide APIs as IOCTLs(as you're FDIF patches). On the other hand, if it is proceeded at the sensor(subdevice) level, I think it's more right to control using CIDs. We need the solution including those two cases. And the name - object detection? object recognition? So, do you have any good ideas? ps. There can be another not-matched HW block level issues. But, the problem which I can check is just above issue for now. Thanks, Heungjun Kim thanks, -- Ming Lei Thank you. Regards, Heungjun Kim -Original Message- From: linux-media-ow...@vger.kernel.org [mailto:linux-media- ow...@vger.kernel.org] On Behalf Of Ming Lei Sent: Monday, December 12, 2011 6:50 PM To: Sylwester Nawrocki Cc: linux-o...@vger.kernel.org; linux-arm-ker...@lists.infradead.org; linux- ker...@vger.kernel.org; linux-media@vger.kernel.org Subject: Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module Hi, On Mon, Dec 12, 2011 at 1:43 AM, Sylwester Nawrocki snj...@gmail.com For OMAP4 FD, it is not needed to include FD into MC framework since a intermediate buffer is always required. If your HW doesn't belong to this case, what is the output of your HW FD in the link? Also sounds FD results may not be needed at all for use space application in the case. The result data is similar to OMAP4 one, plus a few other attributes. User buffers may be filled by other than FD device driver. OK. Could you provide some practical use cases about these? As above, and any device with a camera that controls something and makes decision according to presence of human face in his view. Sounds a reasonable case, :-) If FD result
Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
On Tue, Dec 13, 2011 at 1:59 PM, HeungJun, Kim riverful@samsung.com wrote: Hi Ming and Sylwester, Thanks for the reply. -Original Message- From: Ming Lei [mailto:ming@canonical.com] Sent: Tuesday, December 13, 2011 1:02 PM To: HeungJun, Kim Cc: Sylwester Nawrocki; linux-o...@vger.kernel.org; linux-arm- ker...@lists.infradead.org; linux-ker...@vger.kernel.org; linux- me...@vger.kernel.org Subject: Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module Hi, On Mon, Dec 12, 2011 at 8:08 PM, HeungJun, Kim riverful@samsung.com wrote: Hi Ming, It's maybe late, but I want to suggest one thing about FD API. This OMAP FD block looks detection ability of just face. But, It's possible to occur another device which can detect specific object or patterns. Moreover, this API can expand object recognition area. So, I think it's good to change the API name like v4l2_recog. IMO, object detection is better, at least now OMAP4 and samsung has face detection IP module, and face recognition is often done on results of face detection and more complicated interfaces will be involved. Actually, the detection point is different, I guess. The OMAP has the detection block separately, named FDIF. But, Samsung Exynos doing detection process with externel sensor - m5mols, in our case. This sensor(m5mols) has ISP function, and these ISP can process detection. The expert of FIMC is Sylwester. Probably, he can tell the differences between both in more details. :) Actually, I'm preparing similar control class for mainline with m5mols camera sensor driver. The m5mols camera sensor has the function about face detection. But, I has experienced about Robot Recognition, and I remember the image processing chip which can detect spefic pattern. So, I hesitated naming the API(control or ioctl whatever) with face. It can be possible to provide just object or pattern, not face. Even user library on windows, there is famous OpenCV. And this is also support not only face, but also object. Yes, object is better than face, and we can use enum flag to describe that the objects detected are which kind of objects. In fact, I plan to rename the face detection generic driver as object detection generic driver and let hardware driver to handle the object detection details. The function of OMAP FDIF looks like m5mols ISP's one. please understand I don't have experience about OMAP AP. But, I can tell you it's better to use the name object recognition, not the face detection, for any other device or driver. In a few days, I'll share the CIDs I have thought for m5mols driver. And, I hope to discuss about this with OMAP FDIF. You have been doing it already, :-) Yeah, actually I did. :) But, until I see OMAP FDIF case, I did not recognize AP(like OMAP) can have detection capability. :) So, although I did not think about at that time, I also think we should re-consider this case for now. As I look around your patch quickly, the functions is very similar with ours. (even detection of left/right eye) So, the problem is there are two ways to proceed recognition. - Does it handle at the level of IOCTLs? or CIDs? The patch introduces two IOCTL to retrieve object detection result. I think it should be handled by IOCTL. The interface is a little complex, so it is not easy to handle it by CIDs, IMO. If the detection algorithm is proceeded at the level of HW block, it's right the platform driver provide APIs as IOCTLs(as you're FDIF patches). On the other hand, if it is proceeded at the sensor(subdevice) level, I think it's more right to control using CIDs. Certainly, some generic CIDs for object detection will be introduced later and will be handled in the object detection(the current fdif generic driver, patch 6/7) generic driver. We need the solution including those two cases. And the name - object detection? object recognition? I think object detection is better. For example, face detection is very different with face recognition, isn't it? thanks, -- Ming Lei So, do you have any good ideas? ps. There can be another not-matched HW block level issues. But, the problem which I can check is just above issue for now. Thanks, Heungjun Kim thanks, -- Ming Lei Thank you. Regards, Heungjun Kim -Original Message- From: linux-media-ow...@vger.kernel.org [mailto:linux-media- ow...@vger.kernel.org] On Behalf Of Ming Lei Sent: Monday, December 12, 2011 6:50 PM To: Sylwester Nawrocki Cc: linux-o...@vger.kernel.org; linux-arm-ker...@lists.infradead.org; linux- ker...@vger.kernel.org; linux-media@vger.kernel.org Subject: Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module Hi, On Mon, Dec 12, 2011 at 1:43 AM, Sylwester Nawrocki snj...@gmail.com For OMAP4 FD, it is not needed to include FD into MC framework since a intermediate buffer
Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
On 12/09/2011 04:10 PM, Ming Lei wrote: On Fri, Dec 9, 2011 at 7:25 AM, Sylwester Nawrocki snj...@gmail.com wrote: On 12/07/2011 02:40 PM, Ming Lei wrote: Yes, that is the motivation of the generic FD module. I think we can focus on two use cases for the generic FD now: - one is to detect faces from user space image data - another one is to detect faces in image data generated from HW(SoC internal bus, resize hardware) For OMAP4 hardware, input data is always from physically continuous memory directly, so it is very easy to support the two cases. For the use case 2, if buffer copy is to be avoided, we can use the coming shared dma-buf[1] to pass the image buffer produced by other HW to FD hw directly. Some H/W uses direct data buses to exchange data between processing blocks, and there is no need for additional memory. We only need to manage the data links, for which MC has been designed. For OMAP4 FD, it is not needed to include FD into MC framework since a intermediate buffer is always required. If your HW doesn't belong to this case, what is the output of your HW FD in the link? Also sounds FD results may not be needed at all for use space application in the case. The result data is similar to OMAP4 one, plus a few other attributes. User buffers may be filled by other than FD device driver. For other FD hardware, if it supports to detect faces in image data from physically continuous memory, I think the patch is OK to support it. If the FD hw doesn't support to detect faces from physically continuous memory, I have some questions: how does user space app to parse the FD result if application can't get the input image data? If user space can Do we need the map of detected objects on a input image in all cases ? For normal cases, I think we need, :-) If an application needs only coordinates of detected object on a video signal to for example, focus on it, trigger some action, or just count detected faces, etc. Perhaps there are more practical similar use cases. Could you provide some practical use cases about these? As above, and any device with a camera that controls something and makes decision according to presence of human face in his view. get image data, how does it connect the image data with FD result? and If hardware provides frame sequence numbers the FD result can be associated with a frame, whether it's passing through H/W interconnect or is located in memory. If FD result is associated with a frame, how can user space get the frame seq if no v4l2 buffer is involved? Without a frame sequence, it is a bit difficult to retrieve FD results from user space. If you pass image data in memory buffers from user space, yes, it could be impossible. If there is no buffers you don't need to associate FD result with particular image data. There will be just ascending frame identifiers in reported fd result data... what standard v4l2 ways(v4l2_buffer?) can the app use to describe the image data? We have USERPTR and MMAP memeory buffer for streaming IO, those use v4l2_buffer 1). read()/write() is also used 2), mostly for compressed formats. Except that there are works on shared buffers. If the input image data is from other HW(SoC bus, resizer HW, ...), is the v4l2_buffer needed for the FD HW driver or application? Not really, still v4l2_buffer may be used by other (sub)driver within same video processing pipeline. How long it approximately takes to process single image for OMAP4 FDIF ? See the link[2], and my test result is basically consistent with the data. Thanks. The processing time is rather low, looks like we could easily detect objects in each frame with 30..50 fps. The detection time on image or frame data may be changed a lot, are you sure that your FD HW can handle the data flow correctly? I understand you FD HW has to integrate at least two buffers for coping with the issue, so it should introduce some extra HW cost. I'm not absolutely sure, untill I write the driver and test it myself.. ;) [...] No, I still insist on using frame sequence number rather than buffer index :-) As I mentioned above, how does user space get frame sequence number for retrieving FD results if no v4l2_buffer is involved for FD driver and application? It will be included in the FD result... or in a dedicated v4l2 event data structure. More important, at the end of the day, we'll be getting buffers with image data at some stage of a video pipeline, which would contain same frame identifier (I think we can ignore v4l2_buffer.field for FD purpose). -- Regards, Sylwester -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
Hi Ming,
On 12/02/2011 04:02 PM, Ming Lei wrote:
This patch introduces one driver for face detection purpose.
The driver is responsible for all v4l2 stuff, buffer management
and other general things, and doesn't touch face detection hardware
directly. Several interfaces are exported to low level drivers
(such as the coming omap4 FD driver)which will communicate with
face detection hw module.
So the driver will make driving face detection hw modules more
easy.
TODO:
- implement FD setting interfaces with v4l2 controls or
ext controls
Signed-off-by: Ming Lei ming@canonical.com
---
...
+static int vidioc_g_fd_result(struct file *file, void *priv,
+ struct v4l2_fd_result *f)
+{
+ struct fdif_dev *dev = video_drvdata(file);
+ unsigned long flags;
+ struct v4l2_fdif_result *tmp;
+ struct v4l2_fdif_result *fr = NULL;
+ unsigned int cnt = 0;
+ int ret = -EINVAL;
+
+ spin_lock_irqsave(dev-lock, flags);
+ list_for_each_entry(tmp, dev-fdif_dq.complete, list)
+ if (tmp-index == f-buf_index) {
+ fr = tmp;
+ list_del(tmp-list);
+ break;
+ }
+ spin_unlock_irqrestore(dev-lock, flags);
+
+ if (fr) {
+ ret = 0;
+ cnt = min(f-face_cnt, fr-face_cnt);
+ if (cnt)
+ memcpy(f-fd, fr-faces,
+ sizeof(struct v4l2_fd_detection) * cnt);
+ f-face_cnt = cnt;
+ kfree(fr-faces);
+ kfree(fr);
struct v4l2_fdif_result is allocated in HW driver at the time when FD result
is read out and it is freed in generic module here. Not sure if it is a good
practice to split memory management like this. Also IMHO it would be much
better to pre-allocate memory for FD output data, according to maximum number
of detected faces per frame. It could be more reliable than allocating memory
in interrupt context per each frame.
+ }
+ return ret;
+}
--
Regards,
Sylwester
--
To unsubscribe from this list: send the line unsubscribe linux-media in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
On Fri, Dec 9, 2011 at 7:25 AM, Sylwester Nawrocki snj...@gmail.com wrote: On 12/07/2011 02:40 PM, Ming Lei wrote: I understand the API you mentioned here should belong to kernel internal API, correct me if it is wrong. Yes, I meant the in kernel design, i.e. generic face detection kernel module and an OMAP4 FDIF driver. It makes lots of sense to separate common code in this way, maybe even when there would be only OMAP devices using it. Yes, that is the motivation of the generic FD module. I think we can focus on two use cases for the generic FD now: - one is to detect faces from user space image data - another one is to detect faces in image data generated from HW(SoC internal bus, resize hardware) For OMAP4 hardware, input data is always from physically continuous memory directly, so it is very easy to support the two cases. For the use case 2, if buffer copy is to be avoided, we can use the coming shared dma-buf[1] to pass the image buffer produced by other HW to FD hw directly. Some H/W uses direct data buses to exchange data between processing blocks, and there is no need for additional memory. We only need to manage the data links, for which MC has been designed. For OMAP4 FD, it is not needed to include FD into MC framework since a intermediate buffer is always required. If your HW doesn't belong to this case, what is the output of your HW FD in the link? Also sounds FD results may not be needed at all for use space application in the case. For other FD hardware, if it supports to detect faces in image data from physically continuous memory, I think the patch is OK to support it. If the FD hw doesn't support to detect faces from physically continuous memory, I have some questions: how does user space app to parse the FD result if application can't get the input image data? If user space can Do we need the map of detected objects on a input image in all cases ? For normal cases, I think we need, :-) If an application needs only coordinates of detected object on a video signal to for example, focus on it, trigger some action, or just count detected faces, etc. Perhaps there are more practical similar use cases. Could you provide some practical use cases about these? get image data, how does it connect the image data with FD result? and If hardware provides frame sequence numbers the FD result can be associated with a frame, whether it's passing through H/W interconnect or is located in memory. If FD result is associated with a frame, how can user space get the frame seq if no v4l2 buffer is involved? Without a frame sequence, it is a bit difficult to retrieve FD results from user space. what standard v4l2 ways(v4l2_buffer?) can the app use to describe the image data? We have USERPTR and MMAP memeory buffer for streaming IO, those use v4l2_buffer 1). read()/write() is also used 2), mostly for compressed formats. Except that there are works on shared buffers. If the input image data is from other HW(SoC bus, resizer HW, ...), is the v4l2_buffer needed for the FD HW driver or application? I'm sure now the Samsung devices won't fit in video output node based driver design. They read image data in different ways and also the FD result format is totally different. I think user space will need the FD result, so it is very important to define API to describe the FD result format to user space. And the input about your FD HW result format is certainly helpful to define the API. I'll post exact attributes generated by our FD detection H/W soon. Good news, :-) AFAICS OMAP4 FDIF processes only data stored in memory, thus it seems reasonable to use the videodev interface for passing data to the kernel from user space. But there might be face detection devices that accept data from other H/W modules, e.g. transferred through SoC internal data buses between image processing pipeline blocks. Thus any new interfaces need to be designed with such devices in mind. Also the face detection hardware block might now have an input DMA engine in it, the data could be fed from memory through some other subsystem (e.g. resize/colour converter). Then the driver for that subsystem would implement a video node. I think the direct input image or frame data to FD should be from memory no matter the actual data is from external H/W modules or input DMA because FD will take lot of time to detect faces in one image or frame and FD can't have so much memory to cache several images or frames data. Sorry, I cannot provide much details at the moment, but there exists hardware that reads data from internal SoC buses and even if it uses some sort of cache memory it doesn't necessarily have to be available for the user. Without some hardware background, it is not easy to give a generic FD module design. Yes, please give me some time so I can prepare the list of requirements. Still the FD result is associated with an image frame for such
Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
On 12/07/2011 02:40 PM, Ming Lei wrote: I understand the API you mentioned here should belong to kernel internal API, correct me if it is wrong. Yes, I meant the in kernel design, i.e. generic face detection kernel module and an OMAP4 FDIF driver. It makes lots of sense to separate common code in this way, maybe even when there would be only OMAP devices using it. Yes, that is the motivation of the generic FD module. I think we can focus on two use cases for the generic FD now: - one is to detect faces from user space image data - another one is to detect faces in image data generated from HW(SoC internal bus, resize hardware) For OMAP4 hardware, input data is always from physically continuous memory directly, so it is very easy to support the two cases. For the use case 2, if buffer copy is to be avoided, we can use the coming shared dma-buf[1] to pass the image buffer produced by other HW to FD hw directly. Some H/W uses direct data buses to exchange data between processing blocks, and there is no need for additional memory. We only need to manage the data links, for which MC has been designed. For other FD hardware, if it supports to detect faces in image data from physically continuous memory, I think the patch is OK to support it. If the FD hw doesn't support to detect faces from physically continuous memory, I have some questions: how does user space app to parse the FD result if application can't get the input image data? If user space can Do we need the map of detected objects on a input image in all cases ? If an application needs only coordinates of detected object on a video signal to for example, focus on it, trigger some action, or just count detected faces, etc. Perhaps there are more practical similar use cases. get image data, how does it connect the image data with FD result? and If hardware provides frame sequence numbers the FD result can be associated with a frame, whether it's passing through H/W interconnect or is located in memory. what standard v4l2 ways(v4l2_buffer?) can the app use to describe the image data? We have USERPTR and MMAP memeory buffer for streaming IO, those use v4l2_buffer 1). read()/write() is also used 2), mostly for compressed formats. Except that there are works on shared buffers. I'm sure now the Samsung devices won't fit in video output node based driver design. They read image data in different ways and also the FD result format is totally different. I think user space will need the FD result, so it is very important to define API to describe the FD result format to user space. And the input about your FD HW result format is certainly helpful to define the API. I'll post exact attributes generated by our FD detection H/W soon. AFAICS OMAP4 FDIF processes only data stored in memory, thus it seems reasonable to use the videodev interface for passing data to the kernel from user space. But there might be face detection devices that accept data from other H/W modules, e.g. transferred through SoC internal data buses between image processing pipeline blocks. Thus any new interfaces need to be designed with such devices in mind. Also the face detection hardware block might now have an input DMA engine in it, the data could be fed from memory through some other subsystem (e.g. resize/colour converter). Then the driver for that subsystem would implement a video node. I think the direct input image or frame data to FD should be from memory no matter the actual data is from external H/W modules or input DMA because FD will take lot of time to detect faces in one image or frame and FD can't have so much memory to cache several images or frames data. Sorry, I cannot provide much details at the moment, but there exists hardware that reads data from internal SoC buses and even if it uses some sort of cache memory it doesn't necessarily have to be available for the user. Without some hardware background, it is not easy to give a generic FD module design. Yes, please give me some time so I can prepare the list of requirements. Still the FD result is associated with an image frame for such H/W, but not necessarily with a memory buffer queued by a user application. For user space application, it doesn't make sense to handle FD results only without image data. Even though there are other ways of input image data to FD, user space still need to know the image data, so it makes sense to associate FD result with a memory buffer. How long it approximately takes to process single image for OMAP4 FDIF ? See the link[2], and my test result is basically consistent with the data. Thanks. The processing time is rather low, looks like we could easily detect objects in each frame with 30..50 fps. If you have seen this kind of FD hardware design, please let me know. I'm for leaving the buffer handling details for individual drivers and focusing on a standard interface for
Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
Hi, On Wed, Dec 7, 2011 at 6:01 AM, Sylwester Nawrocki snj...@gmail.com wrote: On 12/06/2011 03:07 PM, Ming Lei wrote: Hi, Thanks for your review. On Tue, Dec 6, 2011 at 5:55 AM, Sylwester Nawrocki snj...@gmail.com wrote: Hi Ming, (I've pruned the Cc list, leaving just the mailing lists) On 12/02/2011 04:02 PM, Ming Lei wrote: This patch introduces one driver for face detection purpose. The driver is responsible for all v4l2 stuff, buffer management and other general things, and doesn't touch face detection hardware directly. Several interfaces are exported to low level drivers (such as the coming omap4 FD driver)which will communicate with face detection hw module. So the driver will make driving face detection hw modules more easy. I would hold on for a moment on implementing generic face detection module which is based on the V4L2 video device interface. We need to first define an API that would be also usable at sub-device interface level (http://linuxtv.org/downloads/v4l-dvb-apis/subdev.html). If we can define a good/stable enough APIs between kernel and user space, I think the patches can be merged first. For internal kernel APIs, we should allow it to evolve as new hardware comes or new features are to be introduced. I also don't see a problem in discussing it a bit more;) OK, fair enough, let's discuss it, :-) I understand the API you mentioned here should belong to kernel internal API, correct me if it is wrong. Yes, I meant the in kernel design, i.e. generic face detection kernel module and an OMAP4 FDIF driver. It makes lots of sense to separate common code in this way, maybe even when there would be only OMAP devices using it. Yes, that is the motivation of the generic FD module. I think we can focus on two use cases for the generic FD now: - one is to detect faces from user space image data - another one is to detect faces in image data generated from HW(SoC internal bus, resize hardware) For OMAP4 hardware, input data is always from physically continuous memory directly, so it is very easy to support the two cases. For the use case 2, if buffer copy is to be avoided, we can use the coming shared dma-buf[1] to pass the image buffer produced by other HW to FD hw directly. For other FD hardware, if it supports to detect faces in image data from physically continuous memory, I think the patch is OK to support it. If the FD hw doesn't support to detect faces from physically continuous memory, I have some questions: how does user space app to parse the FD result if application can't get the input image data? If user space can get image data, how does it connect the image data with FD result? and what standard v4l2 ways(v4l2_buffer?) can the app use to describe the image data? I'm sure now the Samsung devices won't fit in video output node based driver design. They read image data in different ways and also the FD result format is totally different. I think user space will need the FD result, so it is very important to define API to describe the FD result format to user space. And the input about your FD HW result format is certainly helpful to define the API. AFAICS OMAP4 FDIF processes only data stored in memory, thus it seems reasonable to use the videodev interface for passing data to the kernel from user space. But there might be face detection devices that accept data from other H/W modules, e.g. transferred through SoC internal data buses between image processing pipeline blocks. Thus any new interfaces need to be designed with such devices in mind. Also the face detection hardware block might now have an input DMA engine in it, the data could be fed from memory through some other subsystem (e.g. resize/colour converter). Then the driver for that subsystem would implement a video node. I think the direct input image or frame data to FD should be from memory no matter the actual data is from external H/W modules or input DMA because FD will take lot of time to detect faces in one image or frame and FD can't have so much memory to cache several images or frames data. Sorry, I cannot provide much details at the moment, but there exists hardware that reads data from internal SoC buses and even if it uses some sort of cache memory it doesn't necessarily have to be available for the user. Without some hardware background, it is not easy to give a generic FD module design. Still the FD result is associated with an image frame for such H/W, but not necessarily with a memory buffer queued by a user application. For user space application, it doesn't make sense to handle FD results only without image data. Even though there are other ways of input image data to FD, user space still need to know the image data, so it makes sense to associate FD result with a memory buffer. How long it approximately takes to process single image for OMAP4 FDIF ? See the link[2], and my test result is basically consistent with the
Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
Hi,
Thanks for your review.
On Tue, Dec 6, 2011 at 5:55 AM, Sylwester Nawrocki snj...@gmail.com wrote:
Hi Ming,
(I've pruned the Cc list, leaving just the mailing lists)
On 12/02/2011 04:02 PM, Ming Lei wrote:
This patch introduces one driver for face detection purpose.
The driver is responsible for all v4l2 stuff, buffer management
and other general things, and doesn't touch face detection hardware
directly. Several interfaces are exported to low level drivers
(such as the coming omap4 FD driver)which will communicate with
face detection hw module.
So the driver will make driving face detection hw modules more
easy.
I would hold on for a moment on implementing generic face detection
module which is based on the V4L2 video device interface. We need to
first define an API that would be also usable at sub-device interface
level (http://linuxtv.org/downloads/v4l-dvb-apis/subdev.html).
If we can define a good/stable enough APIs between kernel and user space,
I think the patches can be merged first. For internal kernel APIs, we should
allow it to evolve as new hardware comes or new features are to be introduced.
I understand the API you mentioned here should belong to kernel internal
API, correct me if it is wrong.
AFAICS OMAP4 FDIF processes only data stored in memory, thus it seems
reasonable to use the videodev interface for passing data to the kernel
from user space.
But there might be face detection devices that accept data from other
H/W modules, e.g. transferred through SoC internal data buses between
image processing pipeline blocks. Thus any new interfaces need to be
designed with such devices in mind.
Also the face detection hardware block might now have an input DMA
engine in it, the data could be fed from memory through some other
subsystem (e.g. resize/colour converter). Then the driver for that
subsystem would implement a video node.
I think the direct input image or frame data to FD should be from memory
no matter the actual data is from external H/W modules or input DMA because
FD will take lot of time to detect faces in one image or frame and FD can't
have so much memory to cache several images or frames data.
If you have seen this kind of FD hardware design, please let me know.
I'm for leaving the buffer handling details for individual drivers
and focusing on a standard interface for applications, i.e. new
I think leaving buffer handling details in generic FD driver or
individual drivers
doesn't matter now, since it don't have effect on interfaces between kernel
and user space.
ioctl(s) and controls.
TODO:
- implement FD setting interfaces with v4l2 controls or
ext controls
Signed-off-by: Ming Lei ming@canonical.com
---
drivers/media/video/Kconfig | 2 +
drivers/media/video/Makefile | 1 +
drivers/media/video/fdif/Kconfig | 7 +
drivers/media/video/fdif/Makefile | 1 +
drivers/media/video/fdif/fdif.c | 645
+
drivers/media/video/fdif/fdif.h | 114 +++
6 files changed, 770 insertions(+), 0 deletions(-)
create mode 100644 drivers/media/video/fdif/Kconfig
create mode 100644 drivers/media/video/fdif/Makefile
create mode 100644 drivers/media/video/fdif/fdif.c
create mode 100644 drivers/media/video/fdif/fdif.h
[...]
diff --git a/drivers/media/video/fdif/fdif.h
b/drivers/media/video/fdif/fdif.h
new file mode 100644
index 000..ae37ab8
--- /dev/null
+++ b/drivers/media/video/fdif/fdif.h
@@ -0,0 +1,114 @@
+#ifndef _LINUX_FDIF_H
+#define _LINUX_FDIF_H
+
+#include linux/types.h
+#include linux/magic.h
+#include linux/errno.h
+#include linux/kref.h
+#include linux/kernel.h
+#include linux/videodev2.h
+#include media/videobuf2-page.h
+#include media/v4l2-device.h
+#include media/v4l2-ioctl.h
+#include media/v4l2-ctrls.h
+#include media/v4l2-fh.h
+#include media/v4l2-event.h
+#include media/v4l2-common.h
+
+#define MAX_FACE_COUNT 40
+
+#define FACE_SIZE_20_PIXELS 0
+#define FACE_SIZE_25_PIXELS 1
+#define FACE_SIZE_32_PIXELS 2
+#define FACE_SIZE_40_PIXELS 3
This is still OMAP4 FDIF specific, we need to think about v4l2 controls
for this. An ideal would be a menu control type that supports pixel size
(width/height), but unfortunately something like this isn't available
in v4l2 yet.
Yes, it is on TODO list, :-)
+
+#define FACE_DIR_UP 0
+#define FACE_DIR_RIGHT 1
+#define FACE_DIR_LIFT 2
+
+struct fdif_fmt {
+ char *name;
+ u32 fourcc; /* v4l2 format id */
+ int depth;
+ int width, height;
Could width/height be negative ? I don't think it's the case for pixel
resolution. The more proper data type would be u32.
Yes, they should be, will fix it in next version.
Please refer to struct v4l2_pix_format or struct v4l2_rect.
+};
+
+struct fdif_setting {
+ struct fdif_fmt
Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
On 12/06/2011 03:07 PM, Ming Lei wrote: Hi, Thanks for your review. On Tue, Dec 6, 2011 at 5:55 AM, Sylwester Nawrocki snj...@gmail.com wrote: Hi Ming, (I've pruned the Cc list, leaving just the mailing lists) On 12/02/2011 04:02 PM, Ming Lei wrote: This patch introduces one driver for face detection purpose. The driver is responsible for all v4l2 stuff, buffer management and other general things, and doesn't touch face detection hardware directly. Several interfaces are exported to low level drivers (such as the coming omap4 FD driver)which will communicate with face detection hw module. So the driver will make driving face detection hw modules more easy. I would hold on for a moment on implementing generic face detection module which is based on the V4L2 video device interface. We need to first define an API that would be also usable at sub-device interface level (http://linuxtv.org/downloads/v4l-dvb-apis/subdev.html). If we can define a good/stable enough APIs between kernel and user space, I think the patches can be merged first. For internal kernel APIs, we should allow it to evolve as new hardware comes or new features are to be introduced. I also don't see a problem in discussing it a bit more;) I understand the API you mentioned here should belong to kernel internal API, correct me if it is wrong. Yes, I meant the in kernel design, i.e. generic face detection kernel module and an OMAP4 FDIF driver. It makes lots of sense to separate common code in this way, maybe even when there would be only OMAP devices using it. I'm sure now the Samsung devices won't fit in video output node based driver design. They read image data in different ways and also the FD result format is totally different. AFAICS OMAP4 FDIF processes only data stored in memory, thus it seems reasonable to use the videodev interface for passing data to the kernel from user space. But there might be face detection devices that accept data from other H/W modules, e.g. transferred through SoC internal data buses between image processing pipeline blocks. Thus any new interfaces need to be designed with such devices in mind. Also the face detection hardware block might now have an input DMA engine in it, the data could be fed from memory through some other subsystem (e.g. resize/colour converter). Then the driver for that subsystem would implement a video node. I think the direct input image or frame data to FD should be from memory no matter the actual data is from external H/W modules or input DMA because FD will take lot of time to detect faces in one image or frame and FD can't have so much memory to cache several images or frames data. Sorry, I cannot provide much details at the moment, but there exists hardware that reads data from internal SoC buses and even if it uses some sort of cache memory it doesn't necessarily have to be available for the user. Still the FD result is associated with an image frame for such H/W, but not necessarily with a memory buffer queued by a user application. How long it approximately takes to process single image for OMAP4 FDIF ? If you have seen this kind of FD hardware design, please let me know. I'm for leaving the buffer handling details for individual drivers and focusing on a standard interface for applications, i.e. new I think leaving buffer handling details in generic FD driver or individual drivers doesn't matter now, since it don't have effect on interfaces between kernel and user space. I think you misunderstood me. I wasn't talking about core/driver module split, I meant we should not be making the user interface video node centric. I think for Samsung devices I'll need a capture video node for passing the result to the user. So instead of associating FD result with a buffer index we could try to use the frame sequence number (struct v4l2_buffer.sequence, http://linuxtv.org/downloads/v4l-dvb-apis/buffer.html#v4l2-buffer). It might be much better as the v4l2 events are associated with the frame sequence. And if we use controls then you get control events for free, and each event carries a frame sequence number int it (http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-dqevent.html). -- Regards, Sylwester -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
On 12/06/2011 11:01 PM, Sylwester Nawrocki wrote: [...] It might be much better as the v4l2 events are associated with the frame sequence. And if we use controls then you get control events for free, and each event carries a frame sequence number int it (http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-dqevent.html). Oops, just ignore that. The frame sequence number is only available for V4L2_EVENT_FRAME_SYNC events. -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH v1 6/7] media: video: introduce face detection driver module
Hi Ming,
(I've pruned the Cc list, leaving just the mailing lists)
On 12/02/2011 04:02 PM, Ming Lei wrote:
This patch introduces one driver for face detection purpose.
The driver is responsible for all v4l2 stuff, buffer management
and other general things, and doesn't touch face detection hardware
directly. Several interfaces are exported to low level drivers
(such as the coming omap4 FD driver)which will communicate with
face detection hw module.
So the driver will make driving face detection hw modules more
easy.
I would hold on for a moment on implementing generic face detection
module which is based on the V4L2 video device interface. We need to
first define an API that would be also usable at sub-device interface
level (http://linuxtv.org/downloads/v4l-dvb-apis/subdev.html).
AFAICS OMAP4 FDIF processes only data stored in memory, thus it seems
reasonable to use the videodev interface for passing data to the kernel
from user space.
But there might be face detection devices that accept data from other
H/W modules, e.g. transferred through SoC internal data buses between
image processing pipeline blocks. Thus any new interfaces need to be
designed with such devices in mind.
Also the face detection hardware block might now have an input DMA
engine in it, the data could be fed from memory through some other
subsystem (e.g. resize/colour converter). Then the driver for that
subsystem would implement a video node.
I'm for leaving the buffer handling details for individual drivers
and focusing on a standard interface for applications, i.e. new
ioctl(s) and controls.
TODO:
- implement FD setting interfaces with v4l2 controls or
ext controls
Signed-off-by: Ming Lei ming@canonical.com
---
drivers/media/video/Kconfig |2 +
drivers/media/video/Makefile |1 +
drivers/media/video/fdif/Kconfig |7 +
drivers/media/video/fdif/Makefile |1 +
drivers/media/video/fdif/fdif.c | 645
+
drivers/media/video/fdif/fdif.h | 114 +++
6 files changed, 770 insertions(+), 0 deletions(-)
create mode 100644 drivers/media/video/fdif/Kconfig
create mode 100644 drivers/media/video/fdif/Makefile
create mode 100644 drivers/media/video/fdif/fdif.c
create mode 100644 drivers/media/video/fdif/fdif.h
[...]
diff --git a/drivers/media/video/fdif/fdif.h b/drivers/media/video/fdif/fdif.h
new file mode 100644
index 000..ae37ab8
--- /dev/null
+++ b/drivers/media/video/fdif/fdif.h
@@ -0,0 +1,114 @@
+#ifndef _LINUX_FDIF_H
+#define _LINUX_FDIF_H
+
+#include linux/types.h
+#include linux/magic.h
+#include linux/errno.h
+#include linux/kref.h
+#include linux/kernel.h
+#include linux/videodev2.h
+#include media/videobuf2-page.h
+#include media/v4l2-device.h
+#include media/v4l2-ioctl.h
+#include media/v4l2-ctrls.h
+#include media/v4l2-fh.h
+#include media/v4l2-event.h
+#include media/v4l2-common.h
+
+#define MAX_FACE_COUNT 40
+
+#define FACE_SIZE_20_PIXELS 0
+#define FACE_SIZE_25_PIXELS 1
+#define FACE_SIZE_32_PIXELS 2
+#define FACE_SIZE_40_PIXELS 3
This is still OMAP4 FDIF specific, we need to think about v4l2 controls
for this. An ideal would be a menu control type that supports pixel size
(width/height), but unfortunately something like this isn't available
in v4l2 yet.
+
+#define FACE_DIR_UP 0
+#define FACE_DIR_RIGHT 1
+#define FACE_DIR_LIFT2
+
+struct fdif_fmt {
+ char *name;
+ u32 fourcc; /* v4l2 format id */
+ int depth;
+ int width, height;
Could width/height be negative ? I don't think it's the case for pixel
resolution. The more proper data type would be u32.
Please refer to struct v4l2_pix_format or struct v4l2_rect.
+};
+
+struct fdif_setting {
+ struct fdif_fmt*fmt;
+ enum v4l2_fieldfield;
+
+ int min_face_size;
+ int face_dir;
+
+ int startx, starty;
s32
+ int sizex, sizey;
u32
+ int lhit;
+
+ int width, height;
u32
+};
+
+/* buffer for one video frame */
+struct fdif_buffer {
+ /* common v4l buffer stuff -- must be first */
+ struct vb2_buffer vb;
+ struct list_headlist;
+};
+
+
+struct v4l2_fdif_result {
+ struct list_headlist;
+ unsigned intface_cnt;
+ struct v4l2_fd_detection*faces;
+
+ /*v4l2 buffer index*/
+ __u32 index;
+};
+
+struct fdif_dmaqueue {
+ struct list_headcomplete;
+ struct list_headactive;
+ wait_queue_head_t wq;
+};
+
+
+struct fdif_dev {
+ struct kref ref;
+ struct device *dev;
+
+ struct list_head
[RFC PATCH v1 6/7] media: video: introduce face detection driver module
This patch introduces one driver for face detection purpose.
The driver is responsible for all v4l2 stuff, buffer management
and other general things, and doesn't touch face detection hardware
directly. Several interfaces are exported to low level drivers
(such as the coming omap4 FD driver)which will communicate with
face detection hw module.
So the driver will make driving face detection hw modules more
easy.
TODO:
- implement FD setting interfaces with v4l2 controls or
ext controls
Signed-off-by: Ming Lei ming@canonical.com
---
drivers/media/video/Kconfig |2 +
drivers/media/video/Makefile |1 +
drivers/media/video/fdif/Kconfig |7 +
drivers/media/video/fdif/Makefile |1 +
drivers/media/video/fdif/fdif.c | 645 +
drivers/media/video/fdif/fdif.h | 114 +++
6 files changed, 770 insertions(+), 0 deletions(-)
create mode 100644 drivers/media/video/fdif/Kconfig
create mode 100644 drivers/media/video/fdif/Makefile
create mode 100644 drivers/media/video/fdif/fdif.c
create mode 100644 drivers/media/video/fdif/fdif.h
diff --git a/drivers/media/video/Kconfig b/drivers/media/video/Kconfig
index 5684a00..2b01402 100644
--- a/drivers/media/video/Kconfig
+++ b/drivers/media/video/Kconfig
@@ -1166,3 +1166,5 @@ config VIDEO_SAMSUNG_S5P_MFC
MFC 5.1 driver for V4L2.
endif # V4L_MEM2MEM_DRIVERS
+
+source drivers/media/video/fdif/Kconfig
diff --git a/drivers/media/video/Makefile b/drivers/media/video/Makefile
index bc797f2..fdf6b1a 100644
--- a/drivers/media/video/Makefile
+++ b/drivers/media/video/Makefile
@@ -197,6 +197,7 @@ obj-$(CONFIG_VIDEO_IR_I2C) += ir-kbd-i2c.o
obj-y += davinci/
obj-$(CONFIG_ARCH_OMAP)+= omap/
+obj-$(CONFIG_FDIF) += fdif/
ccflags-y += -Idrivers/media/dvb/dvb-core
ccflags-y += -Idrivers/media/dvb/frontends
diff --git a/drivers/media/video/fdif/Kconfig b/drivers/media/video/fdif/Kconfig
new file mode 100644
index 000..e214cb4
--- /dev/null
+++ b/drivers/media/video/fdif/Kconfig
@@ -0,0 +1,7 @@
+config FDIF
+ depends on VIDEO_DEV VIDEO_V4L2
+ select VIDEOBUF2_PAGE
+ tristate Face Detection module
+ help
+ The FDIF is a face detection module, which can be integrated into
+ some SoCs to detect the location of faces in one image or video.
diff --git a/drivers/media/video/fdif/Makefile
b/drivers/media/video/fdif/Makefile
new file mode 100644
index 000..ba1e4c8
--- /dev/null
+++ b/drivers/media/video/fdif/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_FDIF) += fdif.o
diff --git a/drivers/media/video/fdif/fdif.c b/drivers/media/video/fdif/fdif.c
new file mode 100644
index 000..84522d6
--- /dev/null
+++ b/drivers/media/video/fdif/fdif.c
@@ -0,0 +1,645 @@
+/*
+ * fdif.c -- face detection module driver
+ *
+ * Copyright (C) 2011 Ming Lei (ming@canonical.com)
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ */
+
+/*/
+
+#include linux/module.h
+#include linux/fs.h
+#include linux/mm.h
+#include linux/signal.h
+#include linux/wait.h
+#include linux/poll.h
+#include linux/mman.h
+#include linux/pm_runtime.h
+#include linux/delay.h
+#include linux/platform_device.h
+#include linux/interrupt.h
+#include asm/uaccess.h
+#include asm/byteorder.h
+#include asm/io.h
+#include fdif.h
+
+static unsigned debug = 0;
+module_param(debug, uint, 0644);
+MODULE_PARM_DESC(debug, activates debug info);
+
+static LIST_HEAD(fdif_devlist);
+static unsigned video_nr = -1;
+
+int fdif_open(struct file *file)
+{
+ struct fdif_dev *dev = video_drvdata(file);
+
+ kref_get(dev-ref);
+ return v4l2_fh_open(file);
+}
+
+static unsigned int
+fdif_poll(struct file *file, struct poll_table_struct *wait)
+{
+ struct fdif_dev *dev = video_drvdata(file);
+ unsigned int mask = 0;
+ unsigned long flags;
+
+ poll_wait(file, dev-fdif_dq.wq, wait);
+
+ spin_lock_irqsave(dev-lock, flags);
+ if ((file-f_mode FMODE_READ)
+ !list_empty(dev-fdif_dq.complete))
+ mask |= POLLIN | POLLWRNORM;
+ spin_unlock_irqrestore(dev-lock, flags);
+ return mask;
+}
+
+static int
