Hi Caleb,
Thanks for posting this. I've been meaning to chime in on the discussion
about notches and co. for months now, so this makes a perfect opportunity
to finally do so.
On Mon Oct 9, 2023 at 7:32 PM CEST, Caleb Connolly wrote:
> Some folks have previously suggested that this information belongs in
> userspace and not in devicetree. I would like to be clear that
> devicetree is for describing hardware, and parts of a display which can
> never actually be seen are very much properties of the underlying
> hardware.
Yes, thank you! Seeing the kernel log (or Tuxes) partially hidden behind
the notch (or the top-left rounded corner) is just so annoying, so DT is
definitely the way to go here.
> Some discussion has been underway previously on how best to describe
> these features [1][2], including a reference userspace implementation
> using SVG paths [3]. Using this previous discussion as a jumping off
> point, this RFC allows for describing the following display features:
>
> * Corner radius (on a per-corner basis)
> * Circular or pill-shaped cutouts
> * Notches with arbitrary shapes
>
> It's easy to make a case for only using rectangles to describe these
> missing parts of a display, however this severely limits their utility.
> Describing display occlusions as accurately as possible allows for a lot of
> useful UX features. For example, displaying a ring around a hole-punch
> camera when it's in use, or wrapping UI elements around a notch. These
> behaviours are only possible to implement when the dimensions are known
> with near pixel-perfect accuracy.
There are two aspects at play here: simplicity and correctness. Fully
sacrificing one for the other would be a grave mistake. But that does
not mean those two are inherently opposed.
I'd argue that they actually go hand in hand. Keeping the format simple
will make gathering and submitting high-quality data easier. Conversly,
accurately describing the hardware requires *not including* redundant
or non-applicable information.
So why am I writing all of this? Well, the problem I see is that any
shape-based approach will likely suffer from both accuracy and
complexity issues. Describing curves is hard and processing them is
not something that should be included in e.g. whatever handles VTs.
[TLDR: you can skip ahead]
Morover, short of some Turing-complete arbitrary path function, it
sacrifices a bit of expressivity and thus correctness as there will
always be shapes you cannot describe accurately. This may be irrelevant
because in practice you don't need those details. But that highlights
a different problem: there will be many distinct ways to describe many
similar things.
It's also hard to tell if a given curve really has the right shape:
* similar shapes exist – How do you tell whether e.g. a rounded corner
is circular or not? If a DT gets that wrong and the UI decides to for
example draw constant-width padding around such feature it will likely
look very off and ugly.
* measurement precision is a thing – Unless you measure everything on a
scale significantly smaller than individual pixels, sections of your
curves may end up on different pixels than in reality.
* different people have different standards – And so the quality will
vary greatly between devices.
* corners will be cut – This is basically the previous point but I
liked the pun way too much to just delete it, lol
* pixels vs millimeters – Converting between these two will also
increase the overall wobbliness and worsen the user experience.
But is it feasible to measure everything in pixels accurately?
Picking a very small set of basic curve shapes might be a good option if
that can cover all the shapes we expect to find in the wild. It does not
resolve the possible accuracy problems but it would at least be simple.
[TLDR – skip to here]:
However, there's a different approach that is both extremely simple and
yet trivially correct – pixel masks!
Basically, instead of trying to describe what shape a cutout, notch or
other feature has, we just say which pixels belong to it. In short, this:
- can be easily processed even by 'dumb' programs – for example, you can
derive minimal margins to 'letterbox' the screen
- gathering the data is very straightforward – just light the relevant
pixels one-by-one and check if you see them
- pixel-perfect accuracy is the default
Of course this is not a perfect solution. Here are the drawback that I
can see:
- low resolution screens mean low resolution data
- 'smart' programs may or may not need a bit more logic than otherwise
- sub-pixel accuracy is impossible or requires extending this scheme
- non-binary (fractional) weights
- partially-occluded pixels on a separate mask (so each feature would
have two adjacent masks – for its interior and for the border)
As a futher improvement, besides looking at pixels we could think about
subpixels instead. Still, this can easily be added later and likely even
in a backwards-compatible manner.
An orthogonal issue is labeling all of those regions. I think we should
start with fully obscured areas and maybe less readable ones like the
waterfall edges. Still, different features should live on different
masks – even if we don't attach meaningfull labels (like 'notch' or
'camera cutout') to them right away.
What do you all think of that? I didn't see this approach considered in
any of the earlier discussions, yet it seems so elegant to me. Am I
missing something?
--
Cheers,
Piotr Masłowski
