Thank you very much Philip. I will write a 1-pager on this discussion for
posteriority and will proceed with the inside blink approach.

Just as the concluding question, I have a draft CL on it (
crrev.com/c/3370381). Would you suggest that I take a (possibly simplified)
copy of the DataTransfer::NodeImage
in third_party/blink/renderer/modules/accessibility, or refactor the
existing code in DataTransfer to fit the existing and the new use case. I
prefer the latter as it is a very delicate and complex process. If you
agree with that, do you have any suggestions on the refactoring or I would
just go with it and ask you to take a look after?

Best,
Ramin

On Tue, Jan 11, 2022 at 6:58 PM Philip Rogers <[email protected]> wrote:

>
>
> On Tue, Jan 11, 2022 at 7:42 AM Ramin Halavati <[email protected]>
> wrote:
>
>> Thank you very much Philip,
>>
>> About the concern that a repaint based approach (similar to NodeImage)
>> will not have the backgrounds or other higher level effects, can we say
>> that on the other hand (specially since this function is explicitly
>> querying the image for a certain node) the full-page-paint+crop approach
>> also suffers from the possibility that the node maybe be partially covered
>> by another node, or being out of the screen's visible area (needing
>> scroll), or even being covered by a dialog box?
>>
>
> That's correct: the full-page-crop approach will include content on top of
> the target content, will not include the target if it is scrolled
> offscreen, etc.
>
>
>>
>> On Mon, Jan 10, 2022 at 8:29 PM Philip Rogers <[email protected]> wrote:
>>
>>>
>>>
>>> On Mon, Jan 10, 2022 at 6:51 AM Ramin Halavati <[email protected]>
>>> wrote:
>>>
>>>> Thank you Philip.
>>>>
>>>> Since this accessibility support function is called for all renderer
>>>> types (in ChromeOS including Arc++ and Ash), there would be a better
>>>> consistency if we have the snapshotting code in the renderer. Therefore I
>>>> am trying to avoid using the browser side ui::GrabViewSnapshot (as in
>>>> DevTools), unless the alternative approach proves to be too complicated or
>>>> unnecessarily heavy.
>>>>
>>>> I tried using DataTransfer::NodeImage in this draft CL
>>>> <http://crrev.com/c/3370381>, but my understanding is that since it
>>>> triggers a repaint, we should make sure it is not in the middle of a
>>>> lifecycle update to avoid conflicts. Is there a notification of a
>>>> signal that tells when it's safe to request it?
>>>>
>>>
>>> It is important that we do not introduce lifecycle violations but I
>>> think that may already be guaranteed from when the accessibility code runs.
>>> For example, AxObject::GetRelativeBounds cannot be called from within a
>>> lifecycle update either. You can probably just synchronously call the paint
>>> function you need.
>>>
>>>
>>>> I assume it's the same for the SpoolSinglePage as it also redraws the
>>>> image. And performance wise, can we say that as this approach triggers a
>>>> repaint, it's heavier than the browser based snapshotting (using
>>>> ui::GrabViewSnapshot), as the latter is using the already painted output
>>>> and does not require a repaint?
>>>>
>>>
>>> The paint code in NodeImage is really just a walk of layout
>>> datastructures and isn't more expensive than things like layout, style, etc
>>> (i.e., it's O(|layout|)). Doing this for each object would be wildly
>>> inefficient though.
>>>
>>> Expanding on the thing about this being an ambiguous problem:
>>> data:text/html,<body style="background: rebeccapurple; filter:
>>> blur(1px)"><div style="height: 100px; width: 100px; border: 1px solid
>>> white; color:white;" draggable=true>hello world</div>
>>> The drag image for the div will only contain white and transparent
>>> pixels, and will not be blurred. If you want the purple background too,
>>> you'll want to do the full-page-paint+crop approach.
>>>
>>>
>>>>
>>>> Best,
>>>> Ramin
>>>>
>>>> On Fri, Jan 7, 2022 at 11:06 PM Philip Rogers <[email protected]> wrote:
>>>>
>>>>> This is a deceptively difficult problem in general. Our paint system
>>>>> uses the CSS definition of paint order which starts painting at stacking
>>>>> contexts and it is challenging to exclude non-stacked siblings of a
>>>>> non-stacked target element. Some other complications are whether ancestor
>>>>> effects/clips should be applied, and whether content behind the target
>>>>> should be included.
>>>>>
>>>>> DataTransfer::NodeImage is an attempt at this. Another option is to
>>>>> get the entire page painted (examples: devtools uses
>>>>> PageHandler::CaptureScreenshot and printing uses SpoolSinglePage), then
>>>>> crop it to the target element.
>>>>>
>>>>> On Fri, Jan 7, 2022 at 7:01 AM 'Ramin Halavati' via blink-dev <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to find a way to get pixels for any general node in
>>>>>> *third_party/blink/renderer/modules/accessibility. *We already have
>>>>>> AXNodeObject::ImageDataUrl
>>>>>> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/accessibility/ax_node_object.cc;drc=a3ae56ec162fcbf9ee400e9ff80f1ae572957fd3;bpv=1;bpt=1;l=2355?gsn=ImageDataUrl&gs=kythe%3A%2F%2Fchromium.googlesource.com%2Fchromium%2Fsrc%3Flang%3Dc%252B%252B%3Fpath%3Dsrc%2Fthird_party%2Fblink%2Frenderer%2Fmodules%2Faccessibility%2Fax_node_object.cc%23rau1_dUeeJHt__0C5nqtyH5ouOMnhdEN55IRAHfVRMk&gs=kythe%3A%2F%2Fchromium.googlesource.com%2Fchromium%2Fsrc%3Flang%3Dc%252B%252B%3Fpath%3Dsrc%2Fthird_party%2Fblink%2Frenderer%2Fmodules%2Faccessibility%2Fax_node_object.h%23z0USMj0KE_TnmxgTOdb4D-K1NmRUt1i1gyvtbSdYs5w>
>>>>>>  function
>>>>>> to return the bitmap of a certain node, but it is only supporting HTML
>>>>>> element types that already include a picture (namely image, canvas, 
>>>>>> video)
>>>>>> and I am trying to expand to to any general element type.
>>>>>>
>>>>>> So far I've been looking for a way to get access to a painter or
>>>>>> compositor, but I did not find any promising solution yet. Can you point 
>>>>>> me
>>>>>> in the right direction? Is it even expected at this abstraction level?
>>>>>>
>>>>>> Thanks,
>>>>>> Ramin
>>>>>>
>>>>>> P.S., Sorry for naive questions and please point me to the right
>>>>>> email thread if this is not.
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "blink-dev" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to [email protected].
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAORodQi_LSUFXWxpfAKQKKVhRzK1UvGVz%2B4unQUN2qskFSSafw%40mail.gmail.com
>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAORodQi_LSUFXWxpfAKQKKVhRzK1UvGVz%2B4unQUN2qskFSSafw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAORodQjyoWbA23t8C%2BO97oswYnfE_4UX-1uADqTXJsb%2B_dEs8Q%40mail.gmail.com.

Reply via email to