RE: RFC: bus and data format negotiation
Video timings - Once the bus is configured and the data format is set it is finally possible to determine what resolutions and framerates are supported. Here we run into a problem, though. The current V4L2 API is not clear on how that should be done. We have three enumeration ioctls involved in this: either VIDIOC_ENUMSTD to enumerate the supported video standards for analog TV video encoders and decoders, and VIDIOC_ENUM_FRAMESIZES/FRAMEINTERVALS to enumerate native sensor resolutions and framerates. Unfortunately the spec is not clear whether ENUM_FRAMESIZES/INTERVALS really refers to the sensor (or more general, the component that is the initial source of the video stream) or to the sizes and framerates that you can capture on the video node. What makes this worse is that there is an essential ioctl missing: you need to have the equivalent of VIDIOC_S_STD to setup a sensor to a specific resolution/framerate. Right now VIDIOC_S_FMT is used for that, but that is really meant to take an input resolution and scale it up or down to the desired resolution. It is not meant to setup a sensor (or video source or sink in general) to a particular resolution. To fix this we need to 1) specify that the framesize/interval enumeration relates to the video source and not to the output of a possible scaler, and 2) add support for setting up a sensor to a specific size/framerate. Murali Karicheri from TI is working on something similar. Using S_FMT to select a particular resolution never felt right, and now I realize why. Luckily I don't think any of the webcam drivers we have currently do any scaling, so using S_FMT will still work for those and applications do not need to be modified. To do the sensor setup in the new-style we can either introduce a new ioctl to specify the size and use VIDIOC_S_PARM (yuck!) to setup the framerate, or we use the new video timings ioctl that Murali is working on. As yet one more example for your collection: I have an configuration with a sensor, that has a DSP in it. The sensor itself can perform scaling, the DSP has three (!) resizing algorithms, and the SoC can scale too... Data format negotiation --- Depending on the bus configuration a component supports a list of data formats. The next step is to somehow coordinate both sides of the bus to select compatible data formats. In many cases you can just go through the supported data formats and find matches. But this is not true in general. A generic camera framework like soc-camera either needs an optional mapping table that is supplied by the board configuration, or it should be given a callback function that can be implemented at the board level to do the matching for soc-camera (and if no function is supplied it can fallback to a default matching function). A simple example where matching would fail is if the sensor is sending on 8 pins and the SoC receives it on 12 pins (with the lowest 4 pins pulled down). So the data format from the sensor is some 8 bit Bayer format, while the SoC receives a 12 bit Bayer format. You can try all sorts of fancy heuristics to solve such problems, but in my opinion that will never be able to fully solve this data format negotiation. In the end the board designer knows exactly how to match it up, and so it is best to place the matching code at the level where that information is actually available: the platform code. A better example is the one, that I've already mentioned several times: the sensor can only send 10 bits, they are connected directly to an SoC, that can sample 8, 9, or 10 data lines, but it always samples least significant ones. So, in this configuration only 10-bit data would be functional. But there's also an i2c multiplexer in between, thatcan switch D9..D2 sensor outputs to D7..D0 SoC inputs, thus letting it also support 8-bit data. So, in this case: 1. the sensor shall honestly say I can only send 10-bit data. 2. the platform callback shall make this to you can get 8- or 10-bit data from the sensor. 3. when actually configuring that 8-bit format, the platform code shall operate the switch and further issue a call to the sensor to set up the 10-bit data format. Exactly. Using a callback is probably the most flexible way of doing this. This type of data format matching works well in simple situations (e.g. a sensor connected to a SoC), but when you have a network of components all talking to one another it becomes impossible to manage that in a driver. Using the media controller it should be possible to setup the data format for a sub-device directly. Obviously, this only makes sense for SoCs. In theory one could also setup the bus configuration that way, but I feel uncomfortable doing that. This really belongs at the platform level. Note that when dealing with a SoC that e.g. connects a
RFC: bus and data format negotiation
(I promised that I would analyze this. Sorry that it took so long, but I had a lot of other things going on and this was one topic that I needed to really sit down for and think about carefully.) RFC: bus and data format negotiation Version 1.0 Background == As media boards become more complex it is getting harder to enumerate and setup the busses that connect the various components on the board and to map that to actual pixelformats (i.e. how the data will end up in memory). This is a particular problem for sensors that can often be connected in many different ways. Several attempts have been made to find a good internal API for this, but none were satisfactory. In this RFC I'll analyze how such connections are made, what the core problems are and I'll present a solution that hopefully solves these problems in a well-defined and not too complex way. This RFC is directly related to open issue #3 in the media controller RFC. Analysis In general you have two components, A and B, and some sort of a bus between them (i.e. the physical wires). This can be straightforward, e.g. 10 pins on either side are directly connected, or there can be additional components in between like level converters or perhaps even complex FPGAs that are not under our control but that nevertheless can make changes in how the data is presented. While information on such components may not be available at the driver level, it is available at platform (board) level. More complex boards have more than two components and multiple busses, but many of the basics remain the same. Bus configuration - In order to setup a component you will need to supply a bus configuration that sets up the physical properties of the bus: signal polarities, how the data should be sampled, etc. In many cases there is only one possible bus configuration. But especially sensors have more configurations. This configuration should come from the board specification and not be autonegotiated. Depending on the board layout a wrong bus configuration can have quite unpredictable results. Even though both sides of the bus may seemingly support a specific configuration additional board-specific factors may prevent that configuration from working reliably. Data formats For a given bus configuration a component can support one or more data formats. A data format is similar, but not identical, to a pixel format. A pixel format defines the format of the video in memory. A data format defines the format of the video on a bus. The component's driver will know which data formats it supports given the bus config. Note that changing the bus config on the fly will also change the list of supported data formats. Normally a bus config is setup once and not changed, but this is not necessarily always the case. Video timings - Once the bus is configured and the data format is set it is finally possible to determine what resolutions and framerates are supported. Here we run into a problem, though. The current V4L2 API is not clear on how that should be done. We have three enumeration ioctls involved in this: either VIDIOC_ENUMSTD to enumerate the supported video standards for analog TV video encoders and decoders, and VIDIOC_ENUM_FRAMESIZES/FRAMEINTERVALS to enumerate native sensor resolutions and framerates. Unfortunately the spec is not clear whether ENUM_FRAMESIZES/INTERVALS really refers to the sensor (or more general, the component that is the initial source of the video stream) or to the sizes and framerates that you can capture on the video node. What makes this worse is that there is an essential ioctl missing: you need to have the equivalent of VIDIOC_S_STD to setup a sensor to a specific resolution/framerate. Right now VIDIOC_S_FMT is used for that, but that is really meant to take an input resolution and scale it up or down to the desired resolution. It is not meant to setup a sensor (or video source or sink in general) to a particular resolution. To fix this we need to 1) specify that the framesize/interval enumeration relates to the video source and not to the output of a possible scaler, and 2) add support for setting up a sensor to a specific size/framerate. Murali Karicheri from TI is working on something similar. Using S_FMT to select a particular resolution never felt right, and now I realize why. Luckily I don't think any of the webcam drivers we have currently do any scaling, so using S_FMT will still work for those and applications do not need to be modified. To do the sensor setup in the new-style we can either introduce a new ioctl to specify the size and use VIDIOC_S_PARM (yuck!) to setup the framerate, or we use the new video timings ioctl that Murali is working on. Data format negotiation --- Depending on the bus configuration a component supports a list of data formats. The next step is to somehow coordinate both sides of the bus to select compatible
Re: RFC: bus and data format negotiation
On Sun, 13 Sep 2009, Hans Verkuil wrote: (I promised that I would analyze this. Sorry that it took so long, but I had a lot of other things going on and this was one topic that I needed to really sit down for and think about carefully.) RFC: bus and data format negotiation Version 1.0 Background == As media boards become more complex it is getting harder to enumerate and setup the busses that connect the various components on the board and to map that to actual pixelformats (i.e. how the data will end up in memory). This is a particular problem for sensors that can often be connected in many different ways. Several attempts have been made to find a good internal API for this, but none were satisfactory. In this RFC I'll analyze how such connections are made, what the core problems are and I'll present a solution that hopefully solves these problems in a well-defined and not too complex way. This RFC is directly related to open issue #3 in the media controller RFC. Analysis In general you have two components, A and B, and some sort of a bus between them (i.e. the physical wires). This can be straightforward, e.g. 10 pins on either side are directly connected, or there can be additional components in between like level converters or perhaps even complex FPGAs that are not under our control but that nevertheless can make changes in how the data is presented. Or a multiplexer. While information on such components may not be available at the driver level, it is available at platform (board) level. More complex boards have more than two components and multiple busses, but many of the basics remain the same. Bus configuration - In order to setup a component you will need to supply a bus configuration that sets up the physical properties of the bus: signal polarities, how the data should be sampled, etc. In many cases there is only one possible bus configuration. But especially sensors have more configurations. This configuration should come from the board specification and not be autonegotiated. Depending on the board layout a wrong bus configuration can have quite unpredictable results. Even though both sides of the bus may seemingly support a specific configuration additional board-specific factors may prevent that configuration from working reliably. Data formats For a given bus configuration a component can support one or more data formats. A data format is similar, but not identical, to a pixel format. A pixel format defines the format of the video in memory. A data format defines the format of the video on a bus. The component's driver will know which data formats it supports given the bus config. Would be good to mention here, that a new enumeration will be created for those data formats. Note that changing the bus config on the fly will also change the list of supported data formats. Normally a bus config is setup once and not changed, but this is not necessarily always the case. Video timings - Once the bus is configured and the data format is set it is finally possible to determine what resolutions and framerates are supported. Here we run into a problem, though. The current V4L2 API is not clear on how that should be done. We have three enumeration ioctls involved in this: either VIDIOC_ENUMSTD to enumerate the supported video standards for analog TV video encoders and decoders, and VIDIOC_ENUM_FRAMESIZES/FRAMEINTERVALS to enumerate native sensor resolutions and framerates. Unfortunately the spec is not clear whether ENUM_FRAMESIZES/INTERVALS really refers to the sensor (or more general, the component that is the initial source of the video stream) or to the sizes and framerates that you can capture on the video node. What makes this worse is that there is an essential ioctl missing: you need to have the equivalent of VIDIOC_S_STD to setup a sensor to a specific resolution/framerate. Right now VIDIOC_S_FMT is used for that, but that is really meant to take an input resolution and scale it up or down to the desired resolution. It is not meant to setup a sensor (or video source or sink in general) to a particular resolution. To fix this we need to 1) specify that the framesize/interval enumeration relates to the video source and not to the output of a possible scaler, and 2) add support for setting up a sensor to a specific size/framerate. Murali Karicheri from TI is working on something similar. Using S_FMT to select a particular resolution never felt right, and now I realize why. Luckily I don't think any of the webcam drivers we have currently do any scaling, so using S_FMT will still work for those and applications do not need to be modified. To do the sensor setup in the new-style we can either introduce a new ioctl to specify the size and use VIDIOC_S_PARM (yuck!) to setup the framerate, or we use
Re: RFC: bus and data format negotiation
On Sunday 13 September 2009 23:26:54 Guennadi Liakhovetski wrote: On Sun, 13 Sep 2009, Hans Verkuil wrote: (I promised that I would analyze this. Sorry that it took so long, but I had a lot of other things going on and this was one topic that I needed to really sit down for and think about carefully.) RFC: bus and data format negotiation Version 1.0 Background == As media boards become more complex it is getting harder to enumerate and setup the busses that connect the various components on the board and to map that to actual pixelformats (i.e. how the data will end up in memory). This is a particular problem for sensors that can often be connected in many different ways. Several attempts have been made to find a good internal API for this, but none were satisfactory. In this RFC I'll analyze how such connections are made, what the core problems are and I'll present a solution that hopefully solves these problems in a well-defined and not too complex way. This RFC is directly related to open issue #3 in the media controller RFC. Analysis In general you have two components, A and B, and some sort of a bus between them (i.e. the physical wires). This can be straightforward, e.g. 10 pins on either side are directly connected, or there can be additional components in between like level converters or perhaps even complex FPGAs that are not under our control but that nevertheless can make changes in how the data is presented. Or a multiplexer. While information on such components may not be available at the driver level, it is available at platform (board) level. More complex boards have more than two components and multiple busses, but many of the basics remain the same. Bus configuration - In order to setup a component you will need to supply a bus configuration that sets up the physical properties of the bus: signal polarities, how the data should be sampled, etc. In many cases there is only one possible bus configuration. But especially sensors have more configurations. This configuration should come from the board specification and not be autonegotiated. Depending on the board layout a wrong bus configuration can have quite unpredictable results. Even though both sides of the bus may seemingly support a specific configuration additional board-specific factors may prevent that configuration from working reliably. Data formats For a given bus configuration a component can support one or more data formats. A data format is similar, but not identical, to a pixel format. A pixel format defines the format of the video in memory. A data format defines the format of the video on a bus. The component's driver will know which data formats it supports given the bus config. Would be good to mention here, that a new enumeration will be created for those data formats. I'm not sure. It can be as simple as an array of data formats that is accessed from the v4l2_subdev struct. In that case no enumeration is needed. Although to be realistic I suspect we need one in the v4l2_subdev ops. What I don't know yet is whether we also need something like that in userspace for use with the media controller. I'd say we don't for now. Note that changing the bus config on the fly will also change the list of supported data formats. Normally a bus config is setup once and not changed, but this is not necessarily always the case. Video timings - Once the bus is configured and the data format is set it is finally possible to determine what resolutions and framerates are supported. Here we run into a problem, though. The current V4L2 API is not clear on how that should be done. We have three enumeration ioctls involved in this: either VIDIOC_ENUMSTD to enumerate the supported video standards for analog TV video encoders and decoders, and VIDIOC_ENUM_FRAMESIZES/FRAMEINTERVALS to enumerate native sensor resolutions and framerates. Unfortunately the spec is not clear whether ENUM_FRAMESIZES/INTERVALS really refers to the sensor (or more general, the component that is the initial source of the video stream) or to the sizes and framerates that you can capture on the video node. What makes this worse is that there is an essential ioctl missing: you need to have the equivalent of VIDIOC_S_STD to setup a sensor to a specific resolution/framerate. Right now VIDIOC_S_FMT is used for that, but that is really meant to take an input resolution and scale it up or down to the desired resolution. It is not meant to setup a sensor (or video source or sink in general) to a particular resolution. To fix this we need to 1) specify that the framesize/interval enumeration relates to the video source and not to the output of a possible scaler, and 2) add
