On Wed, Jan 12, 2022 at 8:06 AM Ramin Halavati <[email protected]> wrote:
> Thank you very much Philip. I will write a 1-pager on this discussion for > posteriority and will proceed with the inside blink approach. > > Just as the concluding question, I have a draft CL on it ( > crrev.com/c/3370381). Would you suggest that I take a (possibly > simplified) copy of the DataTransfer::NodeImage > in third_party/blink/renderer/modules/accessibility, or refactor the > existing code in DataTransfer to fit the existing and the new use case. I > prefer the latter as it is a very delicate and complex process. If you > agree with that, do you have any suggestions on the refactoring or I would > just go with it and ask you to take a look after? > If the accessibility and drag-image code are unlikely to diverge too much, I think it makes sense to factor out the code that creates a PaintArtifact for a Node out of DraggedNodeImageBuilder and into a helper. This helper could include a comment about how the painting starts from the containing stacking context. > > Best, > Ramin > > On Tue, Jan 11, 2022 at 6:58 PM Philip Rogers <[email protected]> wrote: > >> >> >> On Tue, Jan 11, 2022 at 7:42 AM Ramin Halavati <[email protected]> >> wrote: >> >>> Thank you very much Philip, >>> >>> About the concern that a repaint based approach (similar to NodeImage) >>> will not have the backgrounds or other higher level effects, can we say >>> that on the other hand (specially since this function is explicitly >>> querying the image for a certain node) the full-page-paint+crop approach >>> also suffers from the possibility that the node maybe be partially covered >>> by another node, or being out of the screen's visible area (needing >>> scroll), or even being covered by a dialog box? >>> >> >> That's correct: the full-page-crop approach will include content on top >> of the target content, will not include the target if it is scrolled >> offscreen, etc. >> >> >>> >>> On Mon, Jan 10, 2022 at 8:29 PM Philip Rogers <[email protected]> wrote: >>> >>>> >>>> >>>> On Mon, Jan 10, 2022 at 6:51 AM Ramin Halavati <[email protected]> >>>> wrote: >>>> >>>>> Thank you Philip. >>>>> >>>>> Since this accessibility support function is called for all renderer >>>>> types (in ChromeOS including Arc++ and Ash), there would be a better >>>>> consistency if we have the snapshotting code in the renderer. Therefore I >>>>> am trying to avoid using the browser side ui::GrabViewSnapshot (as in >>>>> DevTools), unless the alternative approach proves to be too complicated or >>>>> unnecessarily heavy. >>>>> >>>>> I tried using DataTransfer::NodeImage in this draft CL >>>>> <http://crrev.com/c/3370381>, but my understanding is that since it >>>>> triggers a repaint, we should make sure it is not in the middle of a >>>>> lifecycle update to avoid conflicts. Is there a notification of a >>>>> signal that tells when it's safe to request it? >>>>> >>>> >>>> It is important that we do not introduce lifecycle violations but I >>>> think that may already be guaranteed from when the accessibility code runs. >>>> For example, AxObject::GetRelativeBounds cannot be called from within a >>>> lifecycle update either. You can probably just synchronously call the paint >>>> function you need. >>>> >>>> >>>>> I assume it's the same for the SpoolSinglePage as it also redraws the >>>>> image. And performance wise, can we say that as this approach triggers a >>>>> repaint, it's heavier than the browser based snapshotting (using >>>>> ui::GrabViewSnapshot), as the latter is using the already painted output >>>>> and does not require a repaint? >>>>> >>>> >>>> The paint code in NodeImage is really just a walk of layout >>>> datastructures and isn't more expensive than things like layout, style, etc >>>> (i.e., it's O(|layout|)). Doing this for each object would be wildly >>>> inefficient though. >>>> >>>> Expanding on the thing about this being an ambiguous problem: >>>> data:text/html,<body style="background: rebeccapurple; filter: >>>> blur(1px)"><div style="height: 100px; width: 100px; border: 1px solid >>>> white; color:white;" draggable=true>hello world</div> >>>> The drag image for the div will only contain white and transparent >>>> pixels, and will not be blurred. If you want the purple background too, >>>> you'll want to do the full-page-paint+crop approach. >>>> >>>> >>>>> >>>>> Best, >>>>> Ramin >>>>> >>>>> On Fri, Jan 7, 2022 at 11:06 PM Philip Rogers <[email protected]> >>>>> wrote: >>>>> >>>>>> This is a deceptively difficult problem in general. Our paint system >>>>>> uses the CSS definition of paint order which starts painting at stacking >>>>>> contexts and it is challenging to exclude non-stacked siblings of a >>>>>> non-stacked target element. Some other complications are whether ancestor >>>>>> effects/clips should be applied, and whether content behind the target >>>>>> should be included. >>>>>> >>>>>> DataTransfer::NodeImage is an attempt at this. Another option is to >>>>>> get the entire page painted (examples: devtools uses >>>>>> PageHandler::CaptureScreenshot and printing uses SpoolSinglePage), then >>>>>> crop it to the target element. >>>>>> >>>>>> On Fri, Jan 7, 2022 at 7:01 AM 'Ramin Halavati' via blink-dev < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am trying to find a way to get pixels for any general node in >>>>>>> *third_party/blink/renderer/modules/accessibility. *We already have >>>>>>> AXNodeObject::ImageDataUrl >>>>>>> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/accessibility/ax_node_object.cc;drc=a3ae56ec162fcbf9ee400e9ff80f1ae572957fd3;bpv=1;bpt=1;l=2355?gsn=ImageDataUrl&gs=kythe%3A%2F%2Fchromium.googlesource.com%2Fchromium%2Fsrc%3Flang%3Dc%252B%252B%3Fpath%3Dsrc%2Fthird_party%2Fblink%2Frenderer%2Fmodules%2Faccessibility%2Fax_node_object.cc%23rau1_dUeeJHt__0C5nqtyH5ouOMnhdEN55IRAHfVRMk&gs=kythe%3A%2F%2Fchromium.googlesource.com%2Fchromium%2Fsrc%3Flang%3Dc%252B%252B%3Fpath%3Dsrc%2Fthird_party%2Fblink%2Frenderer%2Fmodules%2Faccessibility%2Fax_node_object.h%23z0USMj0KE_TnmxgTOdb4D-K1NmRUt1i1gyvtbSdYs5w> >>>>>>> function >>>>>>> to return the bitmap of a certain node, but it is only supporting HTML >>>>>>> element types that already include a picture (namely image, canvas, >>>>>>> video) >>>>>>> and I am trying to expand to to any general element type. >>>>>>> >>>>>>> So far I've been looking for a way to get access to a painter or >>>>>>> compositor, but I did not find any promising solution yet. Can you >>>>>>> point me >>>>>>> in the right direction? Is it even expected at this abstraction level? >>>>>>> >>>>>>> Thanks, >>>>>>> Ramin >>>>>>> >>>>>>> P.S., Sorry for naive questions and please point me to the right >>>>>>> email thread if this is not. >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "blink-dev" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAORodQi_LSUFXWxpfAKQKKVhRzK1UvGVz%2B4unQUN2qskFSSafw%40mail.gmail.com >>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAORodQi_LSUFXWxpfAKQKKVhRzK1UvGVz%2B4unQUN2qskFSSafw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAJgFLLvctS5hZxY5SHVisrpgkfX0zL5C_Y%2BH%2Bbkhg%3D8TontLAQ%40mail.gmail.com.
