Re: HDR support in Wayland/Weston

[Graeme Gill]

Sharma, Shashank wrote:

Hi,

While I'm sure you could hard code various color space assumptions into
such an implementation (and perhaps this is a pretty reasonable way
of doing a proof of concept), it's not a good long term solution,
and could end up being something of a millstone. What's missing
is any serous Color Management in Wayland. It's a bigger project
to fix that, but HDR would then be able to slot into a much more usable
framework.

> - HDR content/buffers are composed in REC2020 colorspace, with bit depth 
> 10/12/16 BPC.
> Some of the popular formats are P010,P012,P016.

While REC2020 based HDR colorspaces are very popular, they aren't the only ones 
out there.

> - Normal SRGB buffers are composed in SRGB color space following REC709 
> specifications.

As far as I'm aware (I haven't been closely monitoring this mailing
list since the last rather long and unsatisfactory discussion about color
management), Wayland works in display dependent colorspace, since there
is no facility for it to know how to convert from anything else to the
display space (i.e. no knowledge of display profiles so it doesn't
know what sRGB is). In all other computer graphic display systems, it's
up to the client to be informed about each display colorspace is, and
to do color conversion to the display space either itself, or by using
operating system libraries. The operating system provides the display
profile information to allow this. As far as I was informed, Wayland
is architected in such a way that this is not possible, since clients
have no knowledge of which display the pixels they send will end up on.

Also note that while the use of an intermediate compositing space is
essential when combining different colorspace sources together, it's
not desirable in other situations where maximum fidelity and
gamut are desired i.e. Photography. (The double conversions are a
possible accuracy loss, and it makes it very difficult to
achieve good gamut mapping from large gamut sources.)

> - For accurate blending in display engines, we need to make sure following:
>     - All the buffers are in same colorspace (Rec 709 or Rec 2020)
>     - All the buffers are liner (gamma/EOTF removed)
>     - All the buffers are tone mapped in same zone (HDR or SDR)

Is that something that Wayland should really know about though ?
i.e. shouldn't that be an application issue, where Wayland provides
the necessary mechanisms to achieve correct composition ?
(Or in fact is that what you are suggesting ?)

>     - Now, in order to re-apply the HDR curve, compositor will apply CRTC 
> level gamma, so
> that the output buffer is non-linear again.

Note that in most Color Managed systems, the CRTC per channel lookup curves are 
used for
the purposes of display calibration, although Wayland doesn't currently support
Color Management tools at all (unlike X11 or other operating systems, it 
provides no
access to the CRTC by client software).

Perhaps it's worth re-visiting some of the ideas about how to add Color 
Management
to Wayland, to see how HDR could fit into it ?

regards,
Graeme Gill
(Author of ArgyllCMS )
___
wayland-devel mailing list
wayland-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

Re: HDR support in Wayland/Weston

[Sharma, Shashank]

Hello Ole,


On 1/10/2019 9:31 PM, Niels Ole Salscheider wrote:

Hello,

on a first glance this sounds sensible. Would it work well with the last color
management protocol proposal that I made or do you see issues there?
We could add REC2020 as another predefined profile.

https://lists.freedesktop.org/archives/wayland-devel/2017-January/032769.html
I think the last proposal was mostly sane and usable for everybody, but there
was not much interest afterwards. However, there was a lot of discussion with
wishes from different sides that went into this. The relevant mailing list
threads are the following, but you have to follow the discussion over the next
months:

https://lists.freedesktop.org/archives/wayland-devel/2016-November/031728.html
https://lists.freedesktop.org/archives/wayland-devel/2014-March/013951.html
Thanks for sharing the links, let me go through the discussion history, 
I will come back with my feedback for this stack, soon.

- Shashank

Best regards,
Ole

Am Donnerstag, 10. Januar 2019, 16:02:18 CET schrieb Sharma, Shashank:

Hello All,

This mail is to propose a design for enabling HDR support in
Wayland/Weston stack, using display engine capabilities, and get more
feedback and input from community.
Here are few points (you might already know these), about HDR
framebuffers, videos and displays:
- HDR content/buffers are composed in REC2020 colorspace, with bit depth
10/12/16 BPC. Some of the popular formats are P010,P012,P016.
- HDR content come with their own Metadata to be applied to get the
right luminance at the display device.
  - The metadata can be of two type 1. static 2. dynamic . For
simplicity, this solution is focusing on static HDR only (HDR10 standard)
- HDR content also provide its supported EOTF (electro optical transfer
function) information, which is a curve (like SRGB gamma curve). One
popular EOTF is PQ(ST2084).
- HDR capable displays mention their EOTF and HDR metadata support
information in EDID CEA-861-G blocks.
- Normal SRGB buffers are composed in SRGB color space following REC709
specifications.
- For accurate blending in display engines, we need to make sure following:
  - All the buffers are in same colorspace (Rec 709 or Rec 2020)
  - All the buffers are liner (gamma/EOTF removed)
  - All the buffers are tone mapped in same zone (HDR or SDR)

Please refer to the block diagram below, which presents a simple case of
a HDR P010 movie playback, with HDR buffers as video buffers, and SDR
buffers as subtitles. The subsystem looks and works like this:
- A client decodes the buffer (using FFMpeg for example) and gets the
two buffers, one with video (HDR) and one subtitles (SDR)
- Client passes following information to the compositor:
   - The actual buffers
   - Their colorspace infromation, BT2020 for HDR buffer, REC709 for
SDR buffer (planning to add a new protocol extension for this)
   - The HDR metadata of the content (planning to add new protocol
for this)

- Compositors actions:
 - Reads the End display's HDR capabilities from display EDID. Assume
its an HDR HDMI monitor.
 - Compositor tone maps every view's framebuffer to match tone of end
display, applying a libVA filter. In this example:
  - The SDR subtitles frame will go through SDR to HDR tone
mapping (called S2H)
  - The HDR video frame will go through HDR to HDR tone mapping
(called H2H) if the HDR capabilities of monitor and content are different.
  - Now both the buffers and the monitor are in the same tone
mapped range.
  - As the end display is HDR capable, and one of the content frame
is HDR, the compositor will prepare all other planes for color space
conversion (CSC) from REC709->REC2020 using plane CSC property.
  - As the CSC and blending should be done in liner space, compositor
will also use plane level degamma to make the buffers linear.
  - These actions will make sure that, during blending:
  - All the buffers are in same colorspace (REC2020)
  - All the buffers are linear
  - All the buffers are tone mapped (HDR)
  - The plane level color properties patch, for DRM can be found
here: https://patchwork.freedesktop.org/series/30875/
  - Now, in order to re-apply the HDR curve, compositor will apply
CRTC level gamma, so that the output buffer is non-linear again.
  - To pass the output HDR information to kernel, so that it can
create and send AVI-info-frames to HDMI, compositor will set Connector
HDR metadata property.
  - Code for the same can be found here:
https://patchwork.freedesktop.org/series/25091/
  - And they will ever live happily after :).

Please provide inputs, feedbacks and suggestions for this design and
plan, so that we can improve out half cooked solution, and start sending
the patches.

   +--+ +---+

   | SDR Buffer subtitles   | HDR Buffer video
   | 

Re: HDR support in Wayland/Weston

[Niels Ole Salscheider]

Hello,

on a first glance this sounds sensible. Would it work well with the last color 
management protocol proposal that I made or do you see issues there?
We could add REC2020 as another predefined profile.

https://lists.freedesktop.org/archives/wayland-devel/2017-January/032769.html

I think the last proposal was mostly sane and usable for everybody, but there 
was not much interest afterwards. However, there was a lot of discussion with 
wishes from different sides that went into this. The relevant mailing list 
threads are the following, but you have to follow the discussion over the next 
months:

https://lists.freedesktop.org/archives/wayland-devel/2016-November/031728.html
https://lists.freedesktop.org/archives/wayland-devel/2014-March/013951.html

Best regards,
Ole

Am Donnerstag, 10. Januar 2019, 16:02:18 CET schrieb Sharma, Shashank:
> Hello All,
> 
> This mail is to propose a design for enabling HDR support in
> Wayland/Weston stack, using display engine capabilities, and get more
> feedback and input from community.
> Here are few points (you might already know these), about HDR
> framebuffers, videos and displays:
> - HDR content/buffers are composed in REC2020 colorspace, with bit depth
> 10/12/16 BPC. Some of the popular formats are P010,P012,P016.
> - HDR content come with their own Metadata to be applied to get the
> right luminance at the display device.
>  - The metadata can be of two type 1. static 2. dynamic . For
> simplicity, this solution is focusing on static HDR only (HDR10 standard)
> - HDR content also provide its supported EOTF (electro optical transfer
> function) information, which is a curve (like SRGB gamma curve). One
> popular EOTF is PQ(ST2084).
> - HDR capable displays mention their EOTF and HDR metadata support
> information in EDID CEA-861-G blocks.
> - Normal SRGB buffers are composed in SRGB color space following REC709
> specifications.
> - For accurate blending in display engines, we need to make sure following:
>  - All the buffers are in same colorspace (Rec 709 or Rec 2020)
>  - All the buffers are liner (gamma/EOTF removed)
>  - All the buffers are tone mapped in same zone (HDR or SDR)
> 
> Please refer to the block diagram below, which presents a simple case of
> a HDR P010 movie playback, with HDR buffers as video buffers, and SDR
> buffers as subtitles. The subsystem looks and works like this:
> - A client decodes the buffer (using FFMpeg for example) and gets the
> two buffers, one with video (HDR) and one subtitles (SDR)
> - Client passes following information to the compositor:
>   - The actual buffers
>   - Their colorspace infromation, BT2020 for HDR buffer, REC709 for
> SDR buffer (planning to add a new protocol extension for this)
>   - The HDR metadata of the content (planning to add new protocol
> for this)
> 
> - Compositors actions:
> - Reads the End display's HDR capabilities from display EDID. Assume
> its an HDR HDMI monitor.
> - Compositor tone maps every view's framebuffer to match tone of end
> display, applying a libVA filter. In this example:
>  - The SDR subtitles frame will go through SDR to HDR tone
> mapping (called S2H)
>  - The HDR video frame will go through HDR to HDR tone mapping
> (called H2H) if the HDR capabilities of monitor and content are different.
>  - Now both the buffers and the monitor are in the same tone
> mapped range.
>  - As the end display is HDR capable, and one of the content frame
> is HDR, the compositor will prepare all other planes for color space
> conversion (CSC) from REC709->REC2020 using plane CSC property.
>  - As the CSC and blending should be done in liner space, compositor
> will also use plane level degamma to make the buffers linear.
>  - These actions will make sure that, during blending:
>  - All the buffers are in same colorspace (REC2020)
>  - All the buffers are linear
>  - All the buffers are tone mapped (HDR)
>  - The plane level color properties patch, for DRM can be found
> here: https://patchwork.freedesktop.org/series/30875/
>  - Now, in order to re-apply the HDR curve, compositor will apply
> CRTC level gamma, so that the output buffer is non-linear again.
>  - To pass the output HDR information to kernel, so that it can
> create and send AVI-info-frames to HDMI, compositor will set Connector
> HDR metadata property.
>  - Code for the same can be found here:
> https://patchwork.freedesktop.org/series/25091/
>  - And they will ever live happily after :).
> 
> Please provide inputs, feedbacks and suggestions for this design and
> plan, so that we can improve out half cooked solution, and start sending
> the patches.
> 
>   +--+ +---+
> 
>   | SDR Buffer subtitles   | HDR Buffer video
>   | (REC  709 colorsp) | (REC 2020 colorsp |
> 
>   

Re: HDR support in Wayland/Weston

[Sharma, Shashank]

If the block diagram is not aligned due to mail client, please refer to 
the attached .txt file. Hope thats slightly better :).


Regards

Shashank

On 1/10/2019 8:32 PM, Sharma, Shashank wrote:

Hello All,

This mail is to propose a design for enabling HDR support in 
Wayland/Weston stack, using display engine capabilities, and get more 
feedback and input from community.
Here are few points (you might already know these), about HDR 
framebuffers, videos and displays:
- HDR content/buffers are composed in REC2020 colorspace, with bit 
depth 10/12/16 BPC. Some of the popular formats are P010,P012,P016.
- HDR content come with their own Metadata to be applied to get the 
right luminance at the display device.
- The metadata can be of two type 1. static 2. dynamic . For 
simplicity, this solution is focusing on static HDR only (HDR10 standard)
- HDR content also provide its supported EOTF (electro optical 
transfer function) information, which is a curve (like SRGB gamma 
curve). One popular EOTF is PQ(ST2084).
- HDR capable displays mention their EOTF and HDR metadata support 
information in EDID CEA-861-G blocks.
- Normal SRGB buffers are composed in SRGB color space following 
REC709 specifications.
- For accurate blending in display engines, we need to make sure 
following:

- All the buffers are in same colorspace (Rec 709 or Rec 2020)
- All the buffers are liner (gamma/EOTF removed)
- All the buffers are tone mapped in same zone (HDR or SDR)

Please refer to the block diagram below, which presents a simple case 
of a HDR P010 movie playback, with HDR buffers as video buffers, and 
SDR buffers as subtitles. The subsystem looks and works like this:
- A client decodes the buffer (using FFMpeg for example) and gets the 
two buffers, one with video (HDR) and one subtitles (SDR)

- Client passes following information to the compositor:
 - The actual buffers
 - Their colorspace infromation, BT2020 for HDR buffer, REC709 for 
SDR buffer (planning to add a new protocol extension for this)
 - The HDR metadata of the content (planning to add new protocol 
for this)


- Compositors actions:
   - Reads the End display's HDR capabilities from display EDID. 
Assume its an HDR HDMI monitor.
   - Compositor tone maps every view's framebuffer to match tone of 
end display, applying a libVA filter. In this example:
- The SDR subtitles frame will go through SDR to HDR tone 
mapping (called S2H)
- The HDR video frame will go through HDR to HDR tone mapping 
(called H2H) if the HDR capabilities of monitor and content are different.
- Now both the buffers and the monitor are in the same tone 
mapped range.
- As the end display is HDR capable, and one of the content frame 
is HDR, the compositor will prepare all other planes for color space 
conversion (CSC) from REC709->REC2020 using plane CSC property.
- As the CSC and blending should be done in liner space, 
compositor will also use plane level degamma to make the buffers linear.

- These actions will make sure that, during blending:
- All the buffers are in same colorspace (REC2020)
- All the buffers are linear
- All the buffers are tone mapped (HDR)
- The plane level color properties patch, for DRM can be found 
here: https://patchwork.freedesktop.org/series/30875/
- Now, in order to re-apply the HDR curve, compositor will apply 
CRTC level gamma, so that the output buffer is non-linear again.
- To pass the output HDR information to kernel, so that it can 
create and send AVI-info-frames to HDMI, compositor will set Connector 
HDR metadata property.
- Code for the same can be found here: 
https://patchwork.freedesktop.org/series/25091/

- And they will ever live happily after :).

Please provide inputs, feedbacks and suggestions for this design and 
plan, so that we can improve out half cooked solution, and start 
sending the patches.


 +--+ +---+
 | SDR Buffer subtitles   | HDR Buffer video
 | (REC  709 colorsp) | (REC 2020 colorsp |
 |  | |   |
 +---+--+ +---+---+
 ||
 ||
 ||
+--v---v+ +--+
  |   Compositor: v   | | 
LibVA|
  |   - assigns views to overlays 
+-> Tone mapping |
  |   - prepare plane/CRTC color properties   
<-+ SDR to HDR   |
  | for linear blending in display
| | HDR to SDR   |

+--+-+--+ +--+

HDR support in Wayland/Weston

[Sharma, Shashank]

Hello All,

This mail is to propose a design for enabling HDR support in 
Wayland/Weston stack, using display engine capabilities, and get more 
feedback and input from community.
Here are few points (you might already know these), about HDR 
framebuffers, videos and displays:
- HDR content/buffers are composed in REC2020 colorspace, with bit depth 
10/12/16 BPC. Some of the popular formats are P010,P012,P016.
- HDR content come with their own Metadata to be applied to get the 
right luminance at the display device.
- The metadata can be of two type 1. static 2. dynamic . For 
simplicity, this solution is focusing on static HDR only (HDR10 standard)
- HDR content also provide its supported EOTF (electro optical transfer 
function) information, which is a curve (like SRGB gamma curve). One 
popular EOTF is PQ(ST2084).
- HDR capable displays mention their EOTF and HDR metadata support 
information in EDID CEA-861-G blocks.
- Normal SRGB buffers are composed in SRGB color space following REC709 
specifications.

- For accurate blending in display engines, we need to make sure following:
- All the buffers are in same colorspace (Rec 709 or Rec 2020)
- All the buffers are liner (gamma/EOTF removed)
- All the buffers are tone mapped in same zone (HDR or SDR)

Please refer to the block diagram below, which presents a simple case of 
a HDR P010 movie playback, with HDR buffers as video buffers, and SDR 
buffers as subtitles. The subsystem looks and works like this:
- A client decodes the buffer (using FFMpeg for example) and gets the 
two buffers, one with video (HDR) and one subtitles (SDR)

- Client passes following information to the compositor:
 - The actual buffers
 - Their colorspace infromation, BT2020 for HDR buffer, REC709 for 
SDR buffer (planning to add a new protocol extension for this)
 - The HDR metadata of the content (planning to add new protocol 
for this)


- Compositors actions:
   - Reads the End display's HDR capabilities from display EDID. Assume 
its an HDR HDMI monitor.
   - Compositor tone maps every view's framebuffer to match tone of end 
display, applying a libVA filter. In this example:
- The SDR subtitles frame will go through SDR to HDR tone 
mapping (called S2H)
- The HDR video frame will go through HDR to HDR tone mapping 
(called H2H) if the HDR capabilities of monitor and content are different.
- Now both the buffers and the monitor are in the same tone 
mapped range.
- As the end display is HDR capable, and one of the content frame 
is HDR, the compositor will prepare all other planes for color space 
conversion (CSC) from REC709->REC2020 using plane CSC property.
- As the CSC and blending should be done in liner space, compositor 
will also use plane level degamma to make the buffers linear.

- These actions will make sure that, during blending:
- All the buffers are in same colorspace (REC2020)
- All the buffers are linear
- All the buffers are tone mapped (HDR)
- The plane level color properties patch, for DRM can be found 
here: https://patchwork.freedesktop.org/series/30875/
- Now, in order to re-apply the HDR curve, compositor will apply 
CRTC level gamma, so that the output buffer is non-linear again.
- To pass the output HDR information to kernel, so that it can 
create and send AVI-info-frames to HDMI, compositor will set Connector 
HDR metadata property.
- Code for the same can be found here: 
https://patchwork.freedesktop.org/series/25091/

- And they will ever live happily after :).

Please provide inputs, feedbacks and suggestions for this design and 
plan, so that we can improve out half cooked solution, and start sending 
the patches.


 +--+ +---+
 | SDR Buffer subtitles   | HDR Buffer video
 | (REC  709 colorsp) | (REC 2020 colorsp |
 |  | |   |
 +---+--+ +---+---+
 ||
 ||
 ||
+--v---v+ +--+
  |   Compositor: v   | | 
LibVA|
  |   - assigns views to overlays 
+-> Tone mapping |
  |   - prepare plane/CRTC color properties   
<-+ SDR to HDR   |
  | for linear blending in display
| | HDR to SDR   |

+--+-+--+ +--+
 | |
 | Tone mapped | Tone mapped
 | non-linear-Rec709   | non-linear 
Rec2020