On Wed, 2011-05-18 at 00:07 +0200, Laurent Pinchart wrote: > Hi everybody, > > I need to implement support for a YUV frame buffer in an fbdev driver. As the > fbdev API doesn't support this out of the box, I've spent a couple of days > reading fbdev (and KMS) code and thinking about how we could cleanly add YUV > support to the API. I'd like to share my findings and thoughts, and hopefully > receive some comments back.
I haven't looked at anything below, but I'll mention that the ivtv driver presents an fbdev interface for the YUV output stream of the CX23415. It may be worth a look and asking Hans what are the short-comings. -Andy > The terms 'format', 'pixel format', 'frame buffer format' and 'data format' > will be used interchangeably in this e-mail. They all refer to the way pixels > are stored in memory, including both the representation of a pixel as integer > values and the layout of those integer values in memory. > > > History and current situation > ----------------------------- > > The fbdev API predates YUV frame buffers. In those old days developers only > cared (and probably thought) about RGB frame buffers, and they developed an > API that could express all kind of weird RGB layout in memory (such as R- > GGGGGGGGGGG-BBBB for instance, I can't imagine hardware implementing that). > This resulted in individual bit field description for the red, green, blue > and > alpha components: > > struct fb_bitfield { > __u32 offset; /* beginning of bitfield */ > __u32 length; /* length of bitfield */ > __u32 msb_right; /* != 0 : Most significant bit is */ > /* right */ > }; > > Grayscale formats were pretty common, so a grayscale field tells color > formats > (grayscale == 0) from grayscale formats (grayscale != 0). > > People already realized that hardware developers were crazily inventive (the > word to remember here is crazily), and that non-standard formats would be > needed at some point. The fb_var_screeninfo ended up containing the following > format-related fields. > > struct fb_var_screeninfo { > ... > __u32 bits_per_pixel; /* guess what */ > __u32 grayscale; /* != 0 Graylevels instead of colors */ > > struct fb_bitfield red; /* bitfield in fb mem if true color, */ > struct fb_bitfield green; /* else only length is significant */ > struct fb_bitfield blue; > struct fb_bitfield transp; /* transparency */ > > __u32 nonstd; /* != 0 Non standard pixel format */ > ... > }; > > Two bits have been specified for the nonstd field: > > #define FB_NONSTD_HAM 1 /* Hold-And-Modify (HAM) */ > #define FB_NONSTD_REV_PIX_IN_B 2 /* order of pixels in each byte is > reversed > */ > > The FB_NONSTD_HAM bit is used by the video/amifb.c driver only to enable HAM > mode[1]. I very much doubt that any new hardware will implement HAM mode (and > I certainly hope none will). > > The FB_NONSTD_REV_PIX_IN_B is used in video/fb_draw.h by the generic bitblit, > fillrect and copy area implementations, not directly by drivers. > > A couple of drivers hardcode the nonstd field to 1 for some reason. Those are > video/arcfb.c (1bpp gray display), video/hecubafb.c (1bpp gray display) and > video/metronomefb.c (8bpp gray display). > > The following drivers use nonstd for various other (and sometimes weird) > purposes: > > video/arkfb.c > Used in 4bpp mode only, to control fb_setcolreg operation > video/fsl-diu-fb.c > Set var->nonstd bits into var->sync (why?) > video/pxafb.c > Encode frame buffer xpos and ypos in the nonstd field > video/s3fb.c > Used in 4bpp mode only, to control fb_setcolreg operation > video/vga16fb.c > When panning in non-8bpp, non-text mode, decrement xoffset > Do some other weird stuff when not 0 > video/i810/i810_main.c > Select direct color mode when set to 1 (truecolor otherwise) > > Fast forward a couple of years, hardware provides support for YUV frame > buffers. Several drivers had to provide YUV format selection to applications, > without any standard API to do so. All of them used the nonstd field for that > purpose: > > media/video/ivtv/ > Enable YUV mode when set to 1 > video/pxafb.c > Encode pixel format in the nonstd field > video/sh_mobile_lcdfb.c > If bpp == 12 and nonstd != 0, enable NV12 mode > If bpp == 16 or bpp == 24, ? > video/omap/omapfb_main.c > Select direct color mode when set to 1 (depend on bpp otherwise) > Used as a pixel format identifier (YUV422, YUV420 or YUY422) > video/omap2/omapfb/omapfb-main.c > Select direct color mode when set to 1 (depend on bpp otherwise) > Used as a pixel format identifier (YUV422 or YUY422) > > Those drivers use the nonstd field in different, incompatible ways. > > > Other related APIs > ------------------ > > Beside the fbdev API, two other kernel/userspace APIs deal with video-related > modes and formats. > > - Kernel Mode Setting (KMS) > > The KMS API is similar in purpose to XRandR. Its main purpose is to provide a > kernel API to configure output video modes. As such, it doesn't care about > frame buffer formats, as they are irrelevant at the CRTC output. > > In addition to handling video modes, the KMS API also provides support for > creating (and managing) frame buffer devices, as well as dumb scan-out > buffers. In-memory data format is relevant there, but KMS only handles width, > height, pitch, depth and bit-per-pixel information. The API doesn't care > whether the frame buffer or the dumb scan-out buffer contains RGB or YUV data. > > An RFC[2] has recently been posted to the dri-devel mailing list to "add > overlays as first class KMS objects". The proposal includes explicit overlay > format support, and discussions have so far pushed towards reusing V4L format > codes for the KMS API. > > - Video 4 Linux (V4L) > > The V4L API version 2 (usually called V4L2) has since the beginning included > explicit support for data format, referred to as pixel formats. > > Pixel formats are identified by a 32-bit number in the form of a four > characters code (4CC or FCC[3]). The list of FCCs defined by V4L2 is > available > in [4]. > > A pixel format uniquely defines the layout of pixel data in memory, including > the format type (RGB, YUV, ...), number of bits per components, components > order and subsampling. It doesn't define the range of acceptable values for > pixel components (such as full-range YUV vs. BT.601[5]), which is carried > through a separate colorspace identifier[6]. > > > YUV support in the fbdev API > ---------------------------- > > Experience with the V4L2 API, both for desktop and embedded devices, and the > KMS API, suggests that recent hardware tend to standardize on a relatively > small number of pixel formats that don't require bitfield-level descriptions. > A pixel format definition based on identifiers should thus fullfill the > hardware needs for the foreseeable future. > > Given the overlap between the KMS, V4L2 and fbdev APIs, the need to share > data > and buffers between those subsystems, and the planned use of V4L2 FCCs in the > KMS overlay API, I believe using V4L2 FCCs to identify fbdev formats would be > a wise decision. > > To select a frame buffer YUV format, the fb_var_screeninfo structure will > need > to be extended with a format field. The fbdev API and ABI must not be broken, > which prevents us from changing the current structure layout and replacing > the > existing format selection mechanism (through the red, green, blue and alpha > bitfields) completely. > > Several solutions exist to introduce a format field in the fb_var_screeninfo > structure. > > - Use the nonstd field as a format identifier. Existing drivers that use the > nonstd field for other purposes wouldn't be able to implement the new API > while keeping their existing API. This isn't a show stopper for drivers using > the FB_NONSTD_HAM and FB_NONSTD_REV_PIX_IN_B bits, as they don't need to > support any non-RGB format. > > Existing drivers that support YUV formats through the nonstd field would have > to select between the current and the new API, without being able to > implement > both. > > - Use one of the reserved fields as a format identifier. Applications are > supposed to zero the fb_var_screeninfo structure before passing it to the > kernel, but experience showed that many applications forget to do so. Drivers > would then be unable to find out whether applications request a particular > format, or just forgot to initialize the field. > > - Add a new FB_NONSTD_FORMAT bit to the nonstd field, and pass the format > through a separate field. If an application sets the FB_NONSTD_FORMAT bit in > the nonstd field, drivers will ignore the red, green, blue, transp and > grayscale fields, and use the format identifier to configure the frame buffer > format. The grayscale field would then be unneeded and could be reused as a > format identifier. Alternatively, one of the reserved fields could be used > for > that purpose. > > Existing drivers that support YUV formats through the nonstd field don't use > the field's most significant bits. They could implement both their current > API > and the new API if the FB_NONSTD_FORMAT bit is stored in one of the nonstd > MSBs. > > - Other solutions are possible, such as adding new ioctls. Those solutions > are > more intrusive, and require larger changes to both userspace and kernelspace > code. > > > The third solution has my preference. Comments and feedback will be > appreciated. I will then work on a proof of concept and submit patches. > > > [1] http://en.wikipedia.org/wiki/Hold_And_Modify > [2] http://lwn.net/Articles/440192/ > [3] http://www.fourcc.org/ > [4] http://lxr.linux.no/linux+v2.6.38/include/linux/videodev2.h#L271 > [5] http://en.wikipedia.org/wiki/Rec._601 > [6] http://lxr.linux.no/linux+v2.6.38/include/linux/videodev2.h#L175 > _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/dri-devel