RE: [PATCH] unstable: add HDR Mastering Meta-data Protocol
Regards Shashank > -Original Message- > From: Graeme Gill [mailto:grae...@argyllcms.com] > Sent: Thursday, March 7, 2019 7:05 AM > To: Sharma, Shashank ; gra...@argyllcms.com; > Pekka Paalanen ; Nautiyal, Ankit K > > Cc: e.bur...@gmail.com; Kps, Harish Krupo ; > niels_...@salscheider-online.de; sebast...@sebastianwick.net; wayland- > de...@lists.freedesktop.org > Subject: Re: [PATCH] unstable: add HDR Mastering Meta-data Protocol > > Sharma, Shashank wrote: > > >> From: Graeme Gill [mailto:grae...@argyllcms.com] > > >>>> From: Ankit Nautiyal > >>>> > >>>> This protcol enables a client to send the hdr meta-data: MAX-CLL, > >>>> MAX-FALL, Max Luminance and Min Luminance as defined by SMPTE ST.2086. > >> > >> Hmm. I'm wondering what the intent of this is. > > > The main reason is to pass it to the compositor so that it can do the > > tone mapping properly. As you know, there could be cases where we are > > dealing with one HDR and multiple SDR frames. Now, the HDR content > > comes with its own metadata information, which needs to be passed to > > the display using the AVI infoframes, for the correct HDR experience, > > but we need to make others non-HDR frames also compatible to this brightness > range of metadata, so we do tone mapping. > > I've been thinking a bit about that, and currently my view is that this is > not the best way > of arranging things. MAX-CLL and MAX-FALL are specific to certain current > source of > HD imagery, and not all HDR sources will have those specific numbers, and > video > standards may change in the future (i.e. frame by frame meta-data). Exactly > how they > can be used is unclear - i.e. they are observations from a video stream, not > parameters > for a specific tone mapping. So punting them to the Compositor doesn't seem > such a > good approach. I kindof agree on this point, that, the usage model is not very clear right now, and this stack and usage model will mature with time. But I would like to still keep this here, for following reasons: - If you closely see the HDMI AVI infoframes specified in CEA-861-G spec, the HDR metadata section is expecting MAX-CLL and MAX-FALL, this means many monitor might be using this already to provide you better viewing experience. So if a content can set these values, no harm in hand carrying that. - The tone mapping API implemented by libVA expects this data, as they are using it to weight the brightness values in frame. So the compositor will pass these values to libVA too. > > Instead what I'd suggest is providing some concrete tone mapping operators > for HDR > handling (at least for V1), with well defined behavior and parameters. The > video > playback application can do the conversion from source format specific > observations > like MAX-CLL and MAX-FALL into the specific compositor tone mapping > parameters, > or it has the option of determining those parameters some other way (user > setting > perhaps, or it's own analysis of the video stream), or it can even do the > adaptation of > the source to the display itself, since it has access to the display > characteristics via the > ICC profile. > > I think it's quite possible to have the Compositor do sane HDR conversions > with no > other extra information than the HDR nominal diffuse white brightness for > each HDR > profile, and perhaps this could be a base Compositor tone mapping behavior, > with > other tone mapping algorithms (and their associated parameters) added as they > are > identified and defined or standardized. > Yep, this is very close to what we are doing in our compositor code. > >> If the idea is that somehow the compositor is to do dynamic HDR > >> rendering, then I have my doubts about its suitability and > >> practicality, unless you are prepared to spell out in detail the algorithm > >> it should > use for performing this. > >> > > The GL tone mapping algorithms will be opensource methods, and will be > > available for the code review. HW tone mapping is using the libVA interface > > and > media-driver handler. > > Right, but there is a distinction between a particular implementation and a > Compositor > specification. > > Can you write a specification of how the tone curves you are proposing to use > work, in > such a way that someone else can implement them, and that someone dealing with > other HDR sources can use them ? Sure, in fact, soon we will publish the code, and you can have a look and provide us feedback. > (i.e. HDR photographs or synthetic imagery from games etc.) > > > Right now we are not
Re: [PATCH] unstable: add HDR Mastering Meta-data Protocol
Pekka Paalanen wrote: > The only question is, do color management and HDR support conceptually > make sense as independent features, or would implementations always > support both with essentially the same effort? In my view, you can certainly add an HDR extension independently of a color management extension, but it would either be a hack to get things working (i.e. make all sorts of general display colorspace assumptions or approximations) or if less hacky, would start to encompass general color management concerns. A Color Management extension on the other hand adds a bunch of non-HDR related stuff, but HDR then slots in as a sub-case of handling differences in display characteristics. > "Supporting HDR" here means only that the compositor is able to process > HDR content from clients, it does not include the capability to drive > HDR monitors. IOW, a compositor that only converts HDR content to SDR > is a valid HDR implementation from protocol perspective. It merely > advertises all outputs as SDR. True, but once you allow for a display to be labelled HDR and have an associated color profile, there isn't that much to handling HDR like any other display. > The big question here is: does a HDR video, with reasonable defaults if > not explicitly defined, allow to programmatically manufacture a custom > ICC profile with the HDR primaries? Yes, this works. Some HDR TV calibration already takes this approach. > What I mean is, if a compositor implements color management, is there > any good reason to not implement HDR luminance mapping as well? At > least the implementation would not differ, just the parameters or > tables. That is why I suspect supporting HDR will be easy on top of a > good color conversion implementation. I agree. > Harish Krupo wrote: >> Out-of-gamut pixels are handled completely separately compared to >> out-of-dynamic-range pixels. Out of gamut can be solved either clamping >> the color after conversion or by using a nearest approximation of the >> color available within the colorspace. >> Out-of-dynamic-range OTOH, requires applying tone mapping to adjust the >> luminance ranges. Adapting the luminance range is just another aspect of mapping one color space to another. One mechanism for achieving this is to use a relative colorspace description of the spaces and then use an orthogonal luminance mapping (such as tone mapping for HDR -> SDR). > I thought that some of the rendering intents of color management do the > same as HDR luminance mapping. Do they not? Only few parameters more to > adjust the ranges of the mapping curves. Standard ICC intents generally do not, because most of them work around a Relative Colorimetric assumption. Absolute Colorimetric in theory could support this, but in practice is not a mechanism that would work the way you want for HDR handling (i.e. it would work for SDR->HDR, but would clip HDR->SDR, and would have the side effect of not aligning the white points, which you probably don't want it to do.) But I think it should be relatively straight forward to augment the standard ICC CMM linking parameters with some HDR options. (In terms of implementation I would hope that lcms's plugin architecture would help facilitate this, or at worst it could be patches and the changes fed back upstream to Marti). The two options I would imagine adding would be SDR->HDR brightness (i.e. what the SDR white brightness should be), and for HDR->SDR what the HDR diffuse white is and possibly a tone mapping strategy. (The technical details of this can be guided by something like the ITU-R BT.2390-2 EETF). > It is also likely that color management will need a wl_output extension > object of its own, so if HDR information can be integrated in that, it > would be simpler to use. Yep. > There is no need to ensure at the protocol level. If the compositor > advertises HDR support and there exists at least one HDR output, the > player should always produce HDR frames if possible, especially if it > is cheaper for the player than SDR. The compositor will take care of the > SDR conversion if necessary, and there won't be glitches if the window > moves between HDR and SDR monitors as there is no need for the client > to catch up. In a color managed implementation, the player just need mark the source with the video ICC profile and HDR intent information for default handling by the Compositor. If it wanted to do the HDR handling itself, it would download the output ICC profile to figure out its conversion from video source to output space, and then mark the buffer as being in the primary output colorspace so that the Compositor knows it doesn't have to do anything. Cheers, Graeme Gill. ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/wayland-devel
RE: [PATCH] unstable: add HDR Mastering Meta-data Protocol
Regards Shashank > -Original Message- > From: Graeme Gill [mailto:grae...@argyllcms.com] > Sent: Tuesday, March 5, 2019 9:35 AM > To: Pekka Paalanen ; Nautiyal, Ankit K > > Cc: e.bur...@gmail.com; niels_...@salscheider-online.de; wayland- > de...@lists.freedesktop.org; sebast...@sebastianwick.net; Kps, Harish Krupo > ; Sharma, Shashank > Subject: Re: [PATCH] unstable: add HDR Mastering Meta-data Protocol > > Pekka Paalanen wrote: > > On Wed, 27 Feb 2019 10:27:16 +0530 > > "Nautiyal, Ankit K" wrote: > > > >> From: Ankit Nautiyal > >> > >> This protcol enables a client to send the hdr meta-data: > >> MAX-CLL, MAX-FALL, Max Luminance and Min Luminance as defined by > >> SMPTE ST.2086. > > Hmm. I'm wondering what the intent of this is. > > If it is to feed these values through to the display itself so that it can do > dynamic HDR > rendering, then this tends to cut across color management, since color > management > as it is currently formulated depends on reproducability in color devices. If > really > desired, I guess this would have to be some kind of special mode that > bypasses the > usual rendering pipeline. > The main reason is to pass it to the compositor so that it can do the tone mapping properly. As you know, there could be cases where we are dealing with one HDR and multiple SDR frames. Now, the HDR content comes with its own metadata information, which needs to be passed to the display using the AVI infoframes, for the correct HDR experience, but we need to make others non-HDR frames also compatible to this brightness range of metadata, so we do tone mapping. We are writing compositor code, which will take a call on target outputs content brightness level too (apart with colorspace) by applying appropriate tone mapping (SDR to HDR, HDR to HDR, or HDR to SDR) using libVA or GL extensions. This will make sure that everything being shown on the display falls into same brightness range. > If the idea is that somehow the compositor is to do dynamic HDR rendering, > then I > have my doubts about its suitability and practicality, unless you are > prepared to spell > out in detail the algorithm it should use for performing this. > The GL tone mapping algorithms will be opensource methods, and will be available for the code review. HW tone mapping is using the libVA interface and media-driver handler. > My assumption was that the compositor would only be expected to do well > understood fallback color conversions, and that anything that needed special > functionality or that was going to do experimental color handling (such as > dynamic > HDR rendering) would do so in the client. > Yes, we are trying our best here to make compositor fair and intelligent, and I believe with proper discussions and code reviews we should be able to reach that level. > For the purposes of mixing SDR and HDR in the compositor as a fallback, a much > simpler set of transforms should be sufficient, namely a brightness for > converting > SDR into HDR, and a tone mapping curve or equivalent settings for compressing > HDR > into SDR. > Agree. > > Is there a case for compositors that implement color management but > > not HDR support? Would it be unreasonable to make HDR parameters a > > part of the color management extension? Such that it would be optional > > for a client to set the HDR parameters. > Right now we are not targeting this, but eventually we want to be there. As we all know, it's a huge area to cover in an single attempt, and gets very cumbersome. So the plan is to first implement a modular limited capability HDR video playback, and use that as a driving and testing vehicle for color management, and then later enhance it with adding more and more modules. > I think that at this point in time it is entirely reasonable that a > compositing system have > some level of HDR awareness. > Yes, the idea is to make it completely aware of HDR scenarios over the period, over a slowly maturing stack. - Shashank > > Btw. do HDR videos contain ICC profiles? Where would a client get the > > ICC profile for a video it wants to show? Do we need to require the > > compositor to expose some commonly used specific profiles? > > As I understand it, no they don't. They either rely on a stated encoding > (i.e. REC2020 > based), or sometimes have simple meta-data such as primary coordinates & > transfer > curve specifications. A client wanting to support such meta-data can simply > create a > matching ICC profile on the fly if it wishes. > > Cheers, > Graeme Gill. ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/wayland-devel
RE: [PATCH] unstable: add HDR Mastering Meta-data Protocol
> From: Pekka Paalanen [mailto:ppaala...@gmail.com] > Sent: Friday, March 1, 2019 7:49 PM > To: Nautiyal, Ankit K > Cc: wayland-devel@lists.freedesktop.org; sebast...@sebastianwick.net; > Sharma, Shashank ; e.bur...@gmail.com; > niels_...@salscheider-online.de; Kps, Harish Krupo > > Subject: Re: [PATCH] unstable: add HDR Mastering Meta-data Protocol > > On Fri, 1 Mar 2019 14:40:16 +0530 > "Nautiyal, Ankit K" wrote: > > > Hi Pekka, > > > > Thanks for the comments and suggestions. > > > > Please my responses inline: > > > > On 2/28/2019 7:51 PM, Pekka Paalanen wrote: > > > On Wed, 27 Feb 2019 10:27:16 +0530 > > > "Nautiyal, Ankit K" wrote: > > > > > >> From: Ankit Nautiyal > > >> > > >> This protcol enables a client to send the hdr meta-data: > > >> MAX-CLL, MAX-FALL, Max Luminance and Min Luminance as defined by > > >> SMPTE ST.2086. > > >> The clients get these values for an HDR video, encoded for a video > > >> stream/file. MAX-CLL (Maximum Content Light Level) tells the > > >> brightest pixel in the entire stream/file in nits. > > >> MAX-FALL (Maximum Frame Average Light Level) tells the highest > > >> frame average brightness in nits for a single frame. Max and Min > > >> Luminance tells the max/min Luminance for the mastering display. > > >> These values give an idea of the brightness of the video which can > > >> be used by displays, so that they can adjust themselves for a > > >> better viewing experience. > > >> > > >> The protocol depends on the Color Management Protocol which is > > >> still under review. There are couple of propsed protocols by Neils > > >> Ole [1], and Sebastian Wick [2], which allow a client to select a > > >> color-space for a surface, via ICC color profile files. > > >> The client is expected to set the color space using the icc files > > >> and the color-management protocol. > > >> > > >> [1]https://patchwork.freedesktop.org/patch/134570/ > > >> [2]https://patchwork.freedesktop.org/patch/286062/ > > >> > > >> Co-authored-by: Harish Krupo > > >> Signed-off-by: Ankit Nautiyal > > > >> diff --git > > >> a/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v > > >> 1.xml > > >> b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v > > >> 1.xml > > >> new file mode 100644 > > >> index 000..aeddf39 > > >> --- /dev/null > > >> +++ b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstab > > >> +++ le-v1.xml > > > Could it be named hdr-mastering rather than hdr-mastering-metadata? > > > Shorter C function names wouldn't hurt. > > Yes I guess, thanks for the suggestion, or perhaps "hdr-metadata" ? > > Hi Ankit, > > to me, the word "metadata" is the redundant one. "HDR" is significant, > "mastering" is probably significant as well, but since we are talking about > Wayland, half of the protocol is essentially metadata. > > But "hdr-metadata" is not bad either. > > Then we have hdr_surface.set_hdr_mastering_metadata. That could be simply > hdr_surface.set_metadata... or hdr_surface.set_luminance since all the > arguments are about luminance somehow. > > Does mastering metadata include more than just luminance parameters? > The primaries and white point maybe? > Yes, it does. It contains the primaries and white point along with luminance of the display used for mastering the content. ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/wayland-devel
Re: [PATCH] unstable: add HDR Mastering Meta-data Protocol
On Fri, 1 Mar 2019 14:40:16 +0530 "Nautiyal, Ankit K" wrote: > Hi Pekka, > > Thanks for the comments and suggestions. > > Please my responses inline: > > On 2/28/2019 7:51 PM, Pekka Paalanen wrote: > > On Wed, 27 Feb 2019 10:27:16 +0530 > > "Nautiyal, Ankit K" wrote: > > > >> From: Ankit Nautiyal > >> > >> This protcol enables a client to send the hdr meta-data: > >> MAX-CLL, MAX-FALL, Max Luminance and Min Luminance as defined by > >> SMPTE ST.2086. > >> The clients get these values for an HDR video, encoded for a video > >> stream/file. MAX-CLL (Maximum Content Light Level) tells the brightest > >> pixel in the entire stream/file in nits. > >> MAX-FALL (Maximum Frame Average Light Level) tells the highest frame > >> average brightness in nits for a single frame. Max and Min Luminance > >> tells the max/min Luminance for the mastering display. > >> These values give an idea of the brightness of the video which can be > >> used by displays, so that they can adjust themselves for a better > >> viewing experience. > >> > >> The protocol depends on the Color Management Protocol which is still > >> under review. There are couple of propsed protocols by Neils Ole [1], > >> and Sebastian Wick [2], which allow a client to select a color-space > >> for a surface, via ICC color profile files. > >> The client is expected to set the color space using the icc files and > >> the color-management protocol. > >> > >> [1]https://patchwork.freedesktop.org/patch/134570/ > >> [2]https://patchwork.freedesktop.org/patch/286062/ > >> > >> Co-authored-by: Harish Krupo > >> Signed-off-by: Ankit Nautiyal > >> diff --git > >> a/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml > >> b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml > >> new file mode 100644 > >> index 000..aeddf39 > >> --- /dev/null > >> +++ > >> b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml > > Could it be named hdr-mastering rather than hdr-mastering-metadata? > > Shorter C function names wouldn't hurt. > Yes I guess, thanks for the suggestion, or perhaps "hdr-metadata" ? Hi Ankit, to me, the word "metadata" is the redundant one. "HDR" is significant, "mastering" is probably significant as well, but since we are talking about Wayland, half of the protocol is essentially metadata. But "hdr-metadata" is not bad either. Then we have hdr_surface.set_hdr_mastering_metadata. That could be simply hdr_surface.set_metadata... or hdr_surface.set_luminance since all the arguments are about luminance somehow. Does mastering metadata include more than just luminance parameters? The primaries and white point maybe? Thanks, pq pgp4STe3LHuz4.pgp Description: OpenPGP digital signature ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/wayland-devel
Re: [PATCH] unstable: add HDR Mastering Meta-data Protocol
On Fri, 01 Mar 2019 15:22:17 +0530 Harish Krupo wrote: > Hi Pekka, > > I have some comments to add to Ankit's. Please find them inline. > > Nautiyal, Ankit K writes: Hi Ankit and Harish, comments below. > > Hi Pekka, > > > > Thanks for the comments and suggestions. > > > > Please my responses inline: > > > > On 2/28/2019 7:51 PM, Pekka Paalanen wrote: > >> On Wed, 27 Feb 2019 10:27:16 +0530 > >> "Nautiyal, Ankit K" wrote: > >> > >>> From: Ankit Nautiyal > >>> > >>> This protcol enables a client to send the hdr meta-data: > >>> MAX-CLL, MAX-FALL, Max Luminance and Min Luminance as defined by > >>> SMPTE ST.2086. > >>> The clients get these values for an HDR video, encoded for a video > >>> stream/file. MAX-CLL (Maximum Content Light Level) tells the brightest > >>> pixel in the entire stream/file in nits. > >>> MAX-FALL (Maximum Frame Average Light Level) tells the highest frame > >>> average brightness in nits for a single frame. Max and Min Luminance > >>> tells the max/min Luminance for the mastering display. > >>> These values give an idea of the brightness of the video which can be > >>> used by displays, so that they can adjust themselves for a better > >>> viewing experience. > >>> > >>> The protocol depends on the Color Management Protocol which is still > >>> under review. There are couple of propsed protocols by Neils Ole [1], > >>> and Sebastian Wick [2], which allow a client to select a color-space > >>> for a surface, via ICC color profile files. > >>> The client is expected to set the color space using the icc files and > >>> the color-management protocol. > >>> > > I believe this part should be removed as the hdr_surface object wraps > over the wl_surface and doesn't interact with colorspace objects. > More on this below. > > >>> [1]https://patchwork.freedesktop.org/patch/134570/ > >>> [2]https://patchwork.freedesktop.org/patch/286062/ > >>> > >>> Co-authored-by: Harish Krupo > >>> Signed-off-by: Ankit Nautiyal > >> Hi Ankit, > >> > >> thanks for working on this, comments inline. > >> > >> I do wonder if this should just be baked into a color management > >> extension directly. More on that below. > > > > We had this in mind initially with this line of thought in couple of early > > interactions with Niels and Sebastian. > > Later I had a discussion in #wayland with Sebastian, and it seemed that the > > brightness values should be handled separately, > > as these do not directly come under ICC profiles as far as I understand. > > As we know HDR comprises of : > > * Transfer functions > > * Color primaries > > * Mastering meta-data : MAX_CLL, MAX_FALL etc. > > Out of these, the first two, can be handled by color-manager and would not > > change often. > > HDR mastering meta-data, as is discussed is coming from the video stream, > > and > > with the dynamic HDR mastering metadata, > > these brightness/luminance values might change frame by frame. > > So it makes sense to have a separate protocol for this, where a client can > > tell > > the compositor about the color-space once with the color-manager protocol, > > and these luminance values, can be sent to the compositor as per the video > > frames. We could have a separate request in the color management protocol instead of having this whole another protocol extension, that is what I meant. If ICC profile does not contain all the parameters HDR needs, then add a request for those that were missed. Being a separate request, a client can choose to send them or not. The per-frame change is not a problem. Both color management proposals already support changing the color profile for each frame. Not changing the color profile while changing the HDR mastering parameters is not a problem either. The only question is, do color management and HDR support conceptually make sense as independent features, or would implementations always support both with essentially the same effort? "Supporting HDR" here means only that the compositor is able to process HDR content from clients, it does not include the capability to drive HDR monitors. IOW, a compositor that only converts HDR content to SDR is a valid HDR implementation from protocol perspective. It merely advertises all outputs as SDR. > > Agreed. The two protocols do not interact with each other and can be used > independently. > > >>> --- > >>> Makefile.am| 1 + > >>> unstable/hdr-mastering-metadata/README | 5 ++ > >>> .../hdr-mastering-metadata-unstable-v1.xml | 95 > >>> ++ > >>> 3 files changed, 101 insertions(+) > >>> create mode 100644 unstable/hdr-mastering-metadata/README > >>> create mode 100644 > >>> unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml > >>> diff --git > >>> a/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml > >>> b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml > >>> new file mode
Re: [PATCH] unstable: add HDR Mastering Meta-data Protocol
Hi Pekka, I have some comments to add to Ankit's. Please find them inline. Nautiyal, Ankit K writes: > Hi Pekka, > > Thanks for the comments and suggestions. > > Please my responses inline: > > On 2/28/2019 7:51 PM, Pekka Paalanen wrote: >> On Wed, 27 Feb 2019 10:27:16 +0530 >> "Nautiyal, Ankit K" wrote: >> >>> From: Ankit Nautiyal >>> >>> This protcol enables a client to send the hdr meta-data: >>> MAX-CLL, MAX-FALL, Max Luminance and Min Luminance as defined by >>> SMPTE ST.2086. >>> The clients get these values for an HDR video, encoded for a video >>> stream/file. MAX-CLL (Maximum Content Light Level) tells the brightest >>> pixel in the entire stream/file in nits. >>> MAX-FALL (Maximum Frame Average Light Level) tells the highest frame >>> average brightness in nits for a single frame. Max and Min Luminance >>> tells the max/min Luminance for the mastering display. >>> These values give an idea of the brightness of the video which can be >>> used by displays, so that they can adjust themselves for a better >>> viewing experience. >>> >>> The protocol depends on the Color Management Protocol which is still >>> under review. There are couple of propsed protocols by Neils Ole [1], >>> and Sebastian Wick [2], which allow a client to select a color-space >>> for a surface, via ICC color profile files. >>> The client is expected to set the color space using the icc files and >>> the color-management protocol. >>> I believe this part should be removed as the hdr_surface object wraps over the wl_surface and doesn't interact with colorspace objects. More on this below. >>> [1]https://patchwork.freedesktop.org/patch/134570/ >>> [2]https://patchwork.freedesktop.org/patch/286062/ >>> >>> Co-authored-by: Harish Krupo >>> Signed-off-by: Ankit Nautiyal >> Hi Ankit, >> >> thanks for working on this, comments inline. >> >> I do wonder if this should just be baked into a color management >> extension directly. More on that below. > > We had this in mind initially with this line of thought in couple of early > interactions with Niels and Sebastian. > Later I had a discussion in #wayland with Sebastian, and it seemed that the > brightness values should be handled separately, > as these do not directly come under ICC profiles as far as I understand. > As we know HDR comprises of : > * Transfer functions > * Color primaries > * Mastering meta-data : MAX_CLL, MAX_FALL etc. > Out of these, the first two, can be handled by color-manager and would not > change often. > HDR mastering meta-data, as is discussed is coming from the video stream, and > with the dynamic HDR mastering metadata, > these brightness/luminance values might change frame by frame. > So it makes sense to have a separate protocol for this, where a client can > tell > the compositor about the color-space once with the color-manager protocol, > and these luminance values, can be sent to the compositor as per the video > frames. > Agreed. The two protocols do not interact with each other and can be used independently. >>> --- >>> Makefile.am| 1 + >>> unstable/hdr-mastering-metadata/README | 5 ++ >>> .../hdr-mastering-metadata-unstable-v1.xml | 95 >>> ++ >>> 3 files changed, 101 insertions(+) >>> create mode 100644 unstable/hdr-mastering-metadata/README >>> create mode 100644 >>> unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml >>> >>> diff --git a/Makefile.am b/Makefile.am >>> index 345ae6a..c097080 100644 >>> --- a/Makefile.am >>> +++ b/Makefile.am >>> @@ -23,6 +23,7 @@ unstable_protocols = >>> \ >>> unstable/xdg-decoration/xdg-decoration-unstable-v1.xml \ >>> >>> unstable/linux-explicit-synchronization/linux-explicit-synchronization-unstable-v1.xml >>> \ >>> unstable/primary-selection/primary-selection-unstable-v1.xml >>> \ >>> + unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml >>> \ >>> $(NULL) >>> stable_protocols = >>> \ >>> diff --git a/unstable/hdr-mastering-metadata/README >>> b/unstable/hdr-mastering-metadata/README >>> new file mode 100644 >>> index 000..b567860 >>> --- /dev/null >>> +++ b/unstable/hdr-mastering-metadata/README >>> @@ -0,0 +1,5 @@ >>> +HDR-MASTERING-META-DATA-PROTOCOL >>> + >>> +Maintainers: >>> +Ankit Nautiyal >>> +Harish Krupo >>> diff --git >>> a/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml >>> b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml >>> new file mode 100644 >>> index 000..aeddf39 >>> --- /dev/null >>> +++ b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml >> Could it be named hdr-mastering rather than hdr-mastering-metadata? >> Shorter C function names wouldn't hurt. > Yes I guess, thanks for the suggestion, or perhaps "hdr-metadata" ? > >>> @@ -0,0 +1,95 @@ >>> + >>> + >>> + >>> + >>> +
Re: [PATCH] unstable: add HDR Mastering Meta-data Protocol
Hi Pekka, Thanks for the comments and suggestions. Please my responses inline: On 2/28/2019 7:51 PM, Pekka Paalanen wrote: On Wed, 27 Feb 2019 10:27:16 +0530 "Nautiyal, Ankit K" wrote: From: Ankit Nautiyal This protcol enables a client to send the hdr meta-data: MAX-CLL, MAX-FALL, Max Luminance and Min Luminance as defined by SMPTE ST.2086. The clients get these values for an HDR video, encoded for a video stream/file. MAX-CLL (Maximum Content Light Level) tells the brightest pixel in the entire stream/file in nits. MAX-FALL (Maximum Frame Average Light Level) tells the highest frame average brightness in nits for a single frame. Max and Min Luminance tells the max/min Luminance for the mastering display. These values give an idea of the brightness of the video which can be used by displays, so that they can adjust themselves for a better viewing experience. The protocol depends on the Color Management Protocol which is still under review. There are couple of propsed protocols by Neils Ole [1], and Sebastian Wick [2], which allow a client to select a color-space for a surface, via ICC color profile files. The client is expected to set the color space using the icc files and the color-management protocol. [1]https://patchwork.freedesktop.org/patch/134570/ [2]https://patchwork.freedesktop.org/patch/286062/ Co-authored-by: Harish Krupo Signed-off-by: Ankit Nautiyal Hi Ankit, thanks for working on this, comments inline. I do wonder if this should just be baked into a color management extension directly. More on that below. We had this in mind initially with this line of thought in couple of early interactions with Niels and Sebastian. Later I had a discussion in #wayland with Sebastian, and it seemed that the brightness values should be handled separately, as these do not directly come under ICC profiles as far as I understand. As we know HDR comprises of : * Transfer functions * Color primaries * Mastering meta-data : MAX_CLL, MAX_FALL etc. Out of these, the first two, can be handled by color-manager and would not change often. HDR mastering meta-data, as is discussed is coming from the video stream, and with the dynamic HDR mastering metadata, these brightness/luminance values might change frame by frame. So it makes sense to have a separate protocol for this, where a client can tell the compositor about the color-space once with the color-manager protocol, and these luminance values, can be sent to the compositor as per the video frames. --- Makefile.am| 1 + unstable/hdr-mastering-metadata/README | 5 ++ .../hdr-mastering-metadata-unstable-v1.xml | 95 ++ 3 files changed, 101 insertions(+) create mode 100644 unstable/hdr-mastering-metadata/README create mode 100644 unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml diff --git a/Makefile.am b/Makefile.am index 345ae6a..c097080 100644 --- a/Makefile.am +++ b/Makefile.am @@ -23,6 +23,7 @@ unstable_protocols = \ unstable/xdg-decoration/xdg-decoration-unstable-v1.xml \ unstable/linux-explicit-synchronization/linux-explicit-synchronization-unstable-v1.xml \ unstable/primary-selection/primary-selection-unstable-v1.xml \ + unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml \ $(NULL) stable_protocols =\ diff --git a/unstable/hdr-mastering-metadata/README b/unstable/hdr-mastering-metadata/README new file mode 100644 index 000..b567860 --- /dev/null +++ b/unstable/hdr-mastering-metadata/README @@ -0,0 +1,5 @@ +HDR-MASTERING-META-DATA-PROTOCOL + +Maintainers: +Ankit Nautiyal +Harish Krupo diff --git a/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml new file mode 100644 index 000..aeddf39 --- /dev/null +++ b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml Could it be named hdr-mastering rather than hdr-mastering-metadata? Shorter C function names wouldn't hurt. Yes I guess, thanks for the suggestion, or perhaps "hdr-metadata" ? @@ -0,0 +1,95 @@ + + + + +Copyright © 2019 Intel + +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the "Software"), +to deal in the Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, sublicense, +and/or sell copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice (including the next +paragraph) shall be included in all copies or substantial portions of the +Software. + +THE SOFTWARE IS PROVIDED "AS IS",
Re: [PATCH] unstable: add HDR Mastering Meta-data Protocol
On Wed, 27 Feb 2019 10:27:16 +0530 "Nautiyal, Ankit K" wrote: > From: Ankit Nautiyal > > This protcol enables a client to send the hdr meta-data: > MAX-CLL, MAX-FALL, Max Luminance and Min Luminance as defined by > SMPTE ST.2086. > The clients get these values for an HDR video, encoded for a video > stream/file. MAX-CLL (Maximum Content Light Level) tells the brightest > pixel in the entire stream/file in nits. > MAX-FALL (Maximum Frame Average Light Level) tells the highest frame > average brightness in nits for a single frame. Max and Min Luminance > tells the max/min Luminance for the mastering display. > These values give an idea of the brightness of the video which can be > used by displays, so that they can adjust themselves for a better > viewing experience. > > The protocol depends on the Color Management Protocol which is still > under review. There are couple of propsed protocols by Neils Ole [1], > and Sebastian Wick [2], which allow a client to select a color-space > for a surface, via ICC color profile files. > The client is expected to set the color space using the icc files and > the color-management protocol. > > [1] https://patchwork.freedesktop.org/patch/134570/ > [2] https://patchwork.freedesktop.org/patch/286062/ > > Co-authored-by: Harish Krupo > Signed-off-by: Ankit Nautiyal Hi Ankit, thanks for working on this, comments inline. I do wonder if this should just be baked into a color management extension directly. More on that below. > --- > Makefile.am| 1 + > unstable/hdr-mastering-metadata/README | 5 ++ > .../hdr-mastering-metadata-unstable-v1.xml | 95 > ++ > 3 files changed, 101 insertions(+) > create mode 100644 unstable/hdr-mastering-metadata/README > create mode 100644 > unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml > > diff --git a/Makefile.am b/Makefile.am > index 345ae6a..c097080 100644 > --- a/Makefile.am > +++ b/Makefile.am > @@ -23,6 +23,7 @@ unstable_protocols = > \ > unstable/xdg-decoration/xdg-decoration-unstable-v1.xml \ > > unstable/linux-explicit-synchronization/linux-explicit-synchronization-unstable-v1.xml > \ > unstable/primary-selection/primary-selection-unstable-v1.xml > \ > + unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml > \ > $(NULL) > > stable_protocols = > \ > diff --git a/unstable/hdr-mastering-metadata/README > b/unstable/hdr-mastering-metadata/README > new file mode 100644 > index 000..b567860 > --- /dev/null > +++ b/unstable/hdr-mastering-metadata/README > @@ -0,0 +1,5 @@ > +HDR-MASTERING-META-DATA-PROTOCOL > + > +Maintainers: > +Ankit Nautiyal > +Harish Krupo > diff --git > a/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml > b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml > new file mode 100644 > index 000..aeddf39 > --- /dev/null > +++ b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml Could it be named hdr-mastering rather than hdr-mastering-metadata? Shorter C function names wouldn't hurt. > @@ -0,0 +1,95 @@ > + > + > + > + > +Copyright © 2019 Intel > + > +Permission is hereby granted, free of charge, to any person obtaining a > +copy of this software and associated documentation files (the > "Software"), > +to deal in the Software without restriction, including without limitation > +the rights to use, copy, modify, merge, publish, distribute, sublicense, > +and/or sell copies of the Software, and to permit persons to whom the > +Software is furnished to do so, subject to the following conditions: > + > +The above copyright notice and this permission notice (including the next > +paragraph) shall be included in all copies or substantial portions of the > +Software. > + > +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS > OR > +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, > +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL > +THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR > OTHER > +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING > +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER > +DEALINGS IN THE SOFTWARE. > + > + > + I think this chapter should explicitly refer to the SMPTE specification. You did it in the commit message, but I think it would be appropriate here. The commit message explains a lot of what this is. The commit message should concentrate on why this extension is needed and why it is like this, and leave the what for the protocol documentation. > +This protocol provides the ability to specify the mastering color
Re: [PATCH] unstable: add HDR Mastering Meta-data Protocol
Hi, On 2/27/2019 2:28 PM, Erwin Burema wrote: Hi, On Wed, 27 Feb 2019 at 05:47, Nautiyal, Ankit K wrote: From: Ankit Nautiyal This protcol enables a client to send the hdr meta-data: MAX-CLL, MAX-FALL, Max Luminance and Min Luminance as defined by SMPTE ST.2086. The clients get these values for an HDR video, encoded for a video stream/file. MAX-CLL (Maximum Content Light Level) tells the brightest pixel in the entire stream/file in nits. MAX-FALL (Maximum Frame Average Light Level) tells the highest frame average brightness in nits for a single frame. Max and Min Luminance tells the max/min Luminance for the mastering display. These values give an idea of the brightness of the video which can be used by displays, so that they can adjust themselves for a better viewing experience. This does sound quite good for video players (and in the future image viewers), but might have something missing for HDR content generation (Blender, Krita, Natron, etc) since you do not always know these values in advance (it is effectively before mastering), most of these work in scene linear with the convention that 0.18 is middle grey (although this is just a convention). So I think that in these cases we might need to get info from the display system on what luminance levels its supports. Hope this makes sense since at the way to busy with other obligations so not much time to look into this. Yes you are right, there would be cases where HDR content-does not have these values. The HDR luminance levels for a display are exposed to the compositor through the edid from the kernel. We can have a discussion whether a displays luminance level can be exposed by the compositor to the client and also how to do it. But currently, the present patch is focused more for HDR video players, as it will be difficult to implement and review all the scenarios/requirements at one go. Once the things set and agreed upon for this case, we can add the support to expose these luminance values too as per discussions. Does that make sense? Regards, Ankit The protocol depends on the Color Management Protocol which is still under review. There are couple of propsed protocols by Neils Ole [1], and Sebastian Wick [2], which allow a client to select a color-space for a surface, via ICC color profile files. The client is expected to set the color space using the icc files and the color-management protocol. [1] https://patchwork.freedesktop.org/patch/134570/ [2] https://patchwork.freedesktop.org/patch/286062/ Co-authored-by: Harish Krupo Signed-off-by: Ankit Nautiyal --- Makefile.am| 1 + unstable/hdr-mastering-metadata/README | 5 ++ .../hdr-mastering-metadata-unstable-v1.xml | 95 ++ 3 files changed, 101 insertions(+) create mode 100644 unstable/hdr-mastering-metadata/README create mode 100644 unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml diff --git a/Makefile.am b/Makefile.am index 345ae6a..c097080 100644 --- a/Makefile.am +++ b/Makefile.am @@ -23,6 +23,7 @@ unstable_protocols = \ unstable/xdg-decoration/xdg-decoration-unstable-v1.xml \ unstable/linux-explicit-synchronization/linux-explicit-synchronization-unstable-v1.xml \ unstable/primary-selection/primary-selection-unstable-v1.xml \ + unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml \ $(NULL) stable_protocols = \ diff --git a/unstable/hdr-mastering-metadata/README b/unstable/hdr-mastering-metadata/README new file mode 100644 index 000..b567860 --- /dev/null +++ b/unstable/hdr-mastering-metadata/README @@ -0,0 +1,5 @@ +HDR-MASTERING-META-DATA-PROTOCOL + +Maintainers: +Ankit Nautiyal +Harish Krupo diff --git a/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml new file mode 100644 index 000..aeddf39 --- /dev/null +++ b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml @@ -0,0 +1,95 @@ + + + + +Copyright © 2019 Intel + +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the "Software"), +to deal in the Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, sublicense, +and/or sell copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice (including the next +paragraph) shall be included in all copies or substantial portions of the +Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +
Re: [PATCH] unstable: add HDR Mastering Meta-data Protocol
Hi, On Wed, 27 Feb 2019 at 05:47, Nautiyal, Ankit K wrote: > > From: Ankit Nautiyal > > This protcol enables a client to send the hdr meta-data: > MAX-CLL, MAX-FALL, Max Luminance and Min Luminance as defined by > SMPTE ST.2086. > The clients get these values for an HDR video, encoded for a video > stream/file. MAX-CLL (Maximum Content Light Level) tells the brightest > pixel in the entire stream/file in nits. > MAX-FALL (Maximum Frame Average Light Level) tells the highest frame > average brightness in nits for a single frame. Max and Min Luminance > tells the max/min Luminance for the mastering display. > These values give an idea of the brightness of the video which can be > used by displays, so that they can adjust themselves for a better > viewing experience. > This does sound quite good for video players (and in the future image viewers), but might have something missing for HDR content generation (Blender, Krita, Natron, etc) since you do not always know these values in advance (it is effectively before mastering), most of these work in scene linear with the convention that 0.18 is middle grey (although this is just a convention). So I think that in these cases we might need to get info from the display system on what luminance levels its supports. Hope this makes sense since at the way to busy with other obligations so not much time to look into this. > The protocol depends on the Color Management Protocol which is still > under review. There are couple of propsed protocols by Neils Ole [1], > and Sebastian Wick [2], which allow a client to select a color-space > for a surface, via ICC color profile files. > The client is expected to set the color space using the icc files and > the color-management protocol. > > [1] https://patchwork.freedesktop.org/patch/134570/ > [2] https://patchwork.freedesktop.org/patch/286062/ > > Co-authored-by: Harish Krupo > Signed-off-by: Ankit Nautiyal > --- > Makefile.am| 1 + > unstable/hdr-mastering-metadata/README | 5 ++ > .../hdr-mastering-metadata-unstable-v1.xml | 95 > ++ > 3 files changed, 101 insertions(+) > create mode 100644 unstable/hdr-mastering-metadata/README > create mode 100644 > unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml > > diff --git a/Makefile.am b/Makefile.am > index 345ae6a..c097080 100644 > --- a/Makefile.am > +++ b/Makefile.am > @@ -23,6 +23,7 @@ unstable_protocols = > \ > unstable/xdg-decoration/xdg-decoration-unstable-v1.xml \ > > unstable/linux-explicit-synchronization/linux-explicit-synchronization-unstable-v1.xml > \ > unstable/primary-selection/primary-selection-unstable-v1.xml > \ > + > unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml \ > $(NULL) > > stable_protocols = > \ > diff --git a/unstable/hdr-mastering-metadata/README > b/unstable/hdr-mastering-metadata/README > new file mode 100644 > index 000..b567860 > --- /dev/null > +++ b/unstable/hdr-mastering-metadata/README > @@ -0,0 +1,5 @@ > +HDR-MASTERING-META-DATA-PROTOCOL > + > +Maintainers: > +Ankit Nautiyal > +Harish Krupo > diff --git > a/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml > b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml > new file mode 100644 > index 000..aeddf39 > --- /dev/null > +++ b/unstable/hdr-mastering-metadata/hdr-mastering-metadata-unstable-v1.xml > @@ -0,0 +1,95 @@ > + > + > + > + > +Copyright © 2019 Intel > + > +Permission is hereby granted, free of charge, to any person obtaining a > +copy of this software and associated documentation files (the > "Software"), > +to deal in the Software without restriction, including without limitation > +the rights to use, copy, modify, merge, publish, distribute, sublicense, > +and/or sell copies of the Software, and to permit persons to whom the > +Software is furnished to do so, subject to the following conditions: > + > +The above copyright notice and this permission notice (including the next > +paragraph) shall be included in all copies or substantial portions of the > +Software. > + > +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS > OR > +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, > +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL > +THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR > OTHER > +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING > +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER > +DEALINGS IN THE SOFTWARE. > + > + > + > +This protocol provides the ability to specify the mastering color volume > +
