Video Hackfest conclusions
Hi, As you may be aware, we held a video hackfest last week in Barcelona. Developers met to discuss how best to improve GPU support for video applications. See http://gstreamer.freedesktop.org/wiki/VideoHackfest for more details. In particular, you might be interested in the notes some people took while the hackfest was going on. These can be found at http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes What follows is the results we arrived at after the hackfest. They are taken from http://gstreamer.freedesktop.org/wiki/VideoHackfest/Conclusions but certainly deserve wide circulation. We're interested in any comments (or questions) you might have about them, so please don't heistate to reply. If you do reply, please make sure to reduce the recipient list appropriately. Cheers, Benjamin, on behalf of all hackfest paticipants. == YUV in pixman == It has been a goal for a while to make video a first-class citizen in the image stack. A concrete API proposal on how to integrate video into pixman was reached and reviewed. Links: * http://gstreamer.freedesktop.org/wiki/VideoHackfest/YCbCr_Formats * http://gstreamer.freedesktop.org/wiki/VideoHackfest/pixman Actions: * GStreamer developers to provide a specification of important YUV formats * Write implementation == YUV in Cairo == To make it easy for applications to use these different formats, the Cairo API needs to be extended to allow them. A proposal does exist and has been reviewed, details need to be finalized. Links: * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo Actions: * Finalize proposal * Write implementation == locking in Cairo == GStreamer takes a lot of freedom in deciding which threads to schedule elements in. Most hardware backends however require proper serialization of commands. The current gst-plugins-cairo and gst-plugins-gl code provide different, but ugly ways to achieve this. Cairo's internal thread safety guarantees match these requirements very well. But the different Cairo backends don't always keep these guarantees. Interaction with these locking mechanisms from outside applications is not possible yet either. Links: * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo Actions: * Improve Cairo backend implementations, in particular X11 and GL * Expose Cairo's locking API to allow interaction with it * Write testcases to squash bugs == locking in Mesa == The GLX specification does not allow binding the same GLX context in multiple threads at the same time. This is a requirement for both Cairo's and GStreamer's threading model. An extension was proposed and initial code developed to support this requirement in the same way as Apple's GL does by default. Windows does not support this and would require potentially expensive fallback code. Links: * http://people.freedesktop.org/~anholt/MESA_multithread_makecurrent.spec Actions: * Get review for suggested extension and include it in future Mesa releases * Make Windows developers investigate the situation == switch GStreamer to Cairo as default video transport model == The current approach to handling video in GStreamer is very outdated. It does not allow hw-accelerated buffers nor does it provide a unified API to modify video buffers, which leads to fragmentation and duplication. It was agreed that using Cairo to provide an abstraction as a drawing API and abstraction over different backends was a good idea. Links: * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes * http://cgit.freedesktop.org/~company/gst-plugins-cairo Actions: * Rework gst-plugins-cairo to match improvements listed in previous points * Get more review on API and to avoid regressions * Merge into gst-plugins-base * Switch plugins gradually to use new Cairo code == gst-plugins-gl == The general consensus was that gst-plugins-gl is a hack. It was necessary in the past to get things to work, but is not a good way forward. However, the functionality or performance provided by the current elements needs to continue working. Developers shared the opinion that gst-plugins-cairo with cairo-gl surfaces is the best way to achieve this. Links: * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes Actions: * Get review of cairo-gl/gst-plugins-cairo code from gst-plugins-gl developers * Port gst-plugins-gl elements to use Cairo early to ensure required features are available == XRenderPutImage == Currently there is no way to upload video data to the X server for later use. The XV extension is not sufficient for anything more complicated than a simple video player. Links: * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo Actions: * Figure out the best way to move YUV data into hw-accelerated Cairo surfaces (GL vs X) * Possibly draft an extension to XRender to handle this in the no-GL case == JIT in Pixman == ORC was investigated as a potential JIT for pixman. A lot of talk has happened, but no clear conclusions were reached. The idea to share a
[ANNOUNCE] Video hackfest
Hi, I'd like to announce the Video hackfest.[1] Its goal is to improve the Linux desktop video experience. The hackfest will take place in Collabora's Barcelona offices on the 4 days from Thursday, November 19th to Sunday, November 22nd, 2009. See the wiki for details[1]. A big thanks goes to the X Foundation for funding and Collabora Multimedia for providing the location. The idea to this hackfest originated in my idea to bridge X, Cairo and GStreamer and make them work together better. I prototyped some code touching every part of the stack, the result of which can be seen on my blog[2][3]. If you are interested in technical details, I sent them to the gstreamer and cairo lists before[4][5]. While doing this work we realized that there is a lack of knowledge about the challenges that the other projects face. Which is when we decided to do a hackfest focussing on improving the Linux desktop video experience. We are currently in the process of collecting topics for the hackfest. The current list is at [6]. If you think there are issues that should be discussed and are missing from the list, just add them. You don't have to be a participant to do that. Just edit the wiki. Also, if you think this hackfest covers exactly the stuff you are hacking on and you should participate in the hackfest, it's most likely a sign you are right and I forgot you. In that case, please mail me at o...@gnome.org or poke me on IRC (my nick is Company on freenode) and we'll figure something out. We might even be able to pay for you, if you are a poor student. Cheers, Benjamin [1]: http://gstreamer.freedesktop.org/wiki/VideoHackfest [2]: http://blogs.gnome.org/otte/2009/10/05/cairo-is-slow/ [3]: http://blogs.gnome.org/otte/2009/10/14/video-hackfest/ [4]: http://lists.cairographics.org/archives/cairo/2009-September/018121.html [5]: http://lists.cairographics.org/archives/cairo/2009-September/018221.html [6]: http://gstreamer.freedesktop.org/wiki/VideoHackfest/Agenda ___ xorg mailing list xorg@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/xorg
Re: Blend modes take 3
On Tue, Nov 25, 2008 at 10:10 PM, Soeren Sandmann [EMAIL PROTECTED] wrote:
- When mask is 0, there is no reason to read the source.
- There is no reason to read the destination if the inverse
combined src/mask is 0
Don't compilers take care of that?
The code looks better to me if the variable assignments all occur in one place.
- The continue is incorrect; it needs to write out 0 in that case,
since the intermediate buffer is not zero-filled.
Oh, the code is supposed to modify the _source_ parameter, too?
Someone really needs to document what component alpha does properly so
I really grok it.
Fixed.
- In FlashSubtractC, the m = ~m is not used. And fbCombineMaskC()
seems like it could just be fbCombineMaskValueC.
Fixed.
if (Max (tmp) sa) while (1);
if (Max (dest) 255 * 255) while (1);
Are these supposed to never happen?
Yeah, It's my poor man's choice at an assertion statement while
debugging. Unfortunately I always forget removing them after I'm done
debugging.
Fixed.
if (min 0) {
tmp[0] = l + (tmp[0] - l) / 4 * l / (l - min) * 4;
tmp[1] = l + (tmp[1] - l) / 4 * l / (l - min) * 4;
tmp[2] = l + (tmp[2] - l) / 4 * l / (l - min) * 4;
}
if (max a) {
tmp[0] = l + (tmp[0] - l) / 4 * (a - l) / (max - l) * 4;
tmp[1] = l + (tmp[1] - l) / 4 * (a - l) / (max - l) * 4;
tmp[2] = l + (tmp[2] - l) / 4 * (a - l) / (max - l) * 4;
}
Where do those 4s come from?
Overflow protection. The tmp value can range from -COMP2_T_MAX .. 2 *
COMP2_T_MAX (I think), so I took the shortcut to just divide by a high
enough number to avoid it. Also, this code will break on 64bit cases
as I'm using ints there for lack of a signed comp4_t type.
It's one of the cases I asked about previously on IRC I think as I was
unsure if this is a case for doubles or how it should best be handled.
Benjamin
___
xorg mailing list
xorg@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/xorg
Re: Blend modes take 3
On Wed, Nov 19, 2008 at 7:00 PM, Soeren Sandmann [EMAIL PROTECTED] wrote: Hi Benjamin, Overall, the separable blend mode stuff looks good, except for the component alpha versions. New update pushed to my git branches of pixman/cairo. I've updated the separable blend modes according to your explanation. However, I couldn't figure out how to compute luminance or saturation from component-alpha versions. So I left the non-separable versions alone, as I'd rather not do something stupid. - I'd like to see one comment that explains how PDF blending works in general, then for each blend function just have the comment describe the blending, not the whole operation. - There is a spurious while() in BlendOverlay(). - Separable is spelled like that, not Seperable - There is a semicolon after while(0) in the Subtract macro - I think the Subtract blend mode should be called something like FLASH_SUBTRACT since it is so weird and is not actually the inverse of Add. All of them applied. I did not yet look at the non-separable blend modes, except that I'm wondering whether the Lum macro has sufficient precision when blending in 64 bit mode? I checked that the Lum macro takes at most a value of the size of a comp2_t but is only called with values of size comp4_t, so I'm very sure that it never overflows. Cheers, Benjamin ___ xorg mailing list xorg@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/xorg
Re: Blend modes take 3
Hey, I've updated the patch again here at the hackfest, and as far as I'm concerned the patch can go in now. But I'd like you to ok it before I push it. We decided to prefix the non-seperable blend-modes with HSL to avoid the confusion with the SATURATE operator. Otherwise I've used the exact naming that is used in the ISO32000 PDF spec. On Thu, Oct 23, 2008 at 11:11 PM, Soeren Sandmann [EMAIL PROTECTED] wrote: So far as pixman and Render are concerned, it seems like the INVERT operator can be trivially implemented with (white IN src) DIFFERENCE dest. If so, there doesn't seem to be a good reason for pixman to export the INVERT operator. Whether cairo should is another matter - personally, I wouldn't add it, but I'll let other people answer that question. I switched swfdec to use it and it worked fine, so I removed INVERT from Pixman/Cairo. If it would be really expensive, say involving reading pixels back from the X server, then I'd say add it, but if there is a fairly simple workaround, then maybe we can leave it out and revisit the question later. I've asked people here at the hackfest and nobody knew a solution to it. So I left it in. Here are the things that I think should be fixed before merging: [...] All the blend loops could then be defined by simply FbBlendFuncU(Overlay); FbBlendFuncU(Hardlight); ... I've done that and saved roughly 50% of the code. The code looks way cleaner to me, too. Also, I think we need component-alpha versions of the blend modes. As you noted on IRC, this should be a mostly a matter of writing an FbBlendFuncC macro. I did this, too. Could we just use floating point here? The integer math above is quite incomprehensible. [...] I'll look at the separable blend modes later, but from a quick look it seems like it could benefit from using floating point in various places too. I've switched SoftLight to use doubles now, but left non-seperable ones alone. Cheers, Benjamin ___ xorg mailing list xorg@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/xorg
