Title: [290211] trunk/Source/WebKit
Revision
290211
Author
wenson_hs...@apple.com
Date
2022-02-19 11:24:02 -0800 (Sat, 19 Feb 2022)

Log Message

[macOS] Hovering over "Copy Cropped Image" context menu item should reveal the cropped image
https://bugs.webkit.org/show_bug.cgi?id=236845
rdar://89152746

Reviewed by Dean Jackson.

Add support for revealing the "cropped" region of an image element (while dimming out the rest of the image)
when the user highlights the new "Copy Cropped Image" context menu item on macOS. See below for more details.

* Platform/cocoa/TextRecognitionUtilities.h:

Adjust `requestImageAnalysisMarkup` to take a completion handler with both the resulting CGImageRef, as well as
a CGRect that represents the frame of the cropped image with respect to the source image, normalized to the unit
square. Additionally, tweak all call sites of `requestImageAnalysisMarkup` to pass in completion handlers that
accept both arguments.

* Shared/ContextMenuContextData.cpp:
(WebKit::ContextMenuContextData::ContextMenuContextData):
(WebKit::ContextMenuContextData::encode const):
(WebKit::ContextMenuContextData::decode):
* Shared/ContextMenuContextData.h:
(WebKit::ContextMenuContextData::hitTestedElementContext const):

Add an optional ElementContext representing the inner hit-tested element for the context menu. We use this below
in `WebPageProxy::willHighlightContextMenuItem` to know where we need to install the cropped image overlay.

* UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::didDismissContextMenu):

Reset the cached cropped image result as well as the cropped image overlay state when the context menu is
dismissed.

(WebKit::WebPageProxy::resetStateAfterProcessExited):

Also clear the above state if the process crashes.

* UIProcess/WebPageProxy.h:
* UIProcess/ios/WKContentViewInteraction.mm:
(-[WKContentView doAfterComputingImageAnalysisResultsForMarkup:]):
(-[WKContentView actionSheetAssistant:copyCroppedImage:sourceMIMEType:]):
* UIProcess/mac/WebContextMenuProxyMac.mm:
(-[WKMenuDelegate menu:willHighlightItem:]):

Implement this NSMenu delegate method and call out to WebPageProxy whenever the highlighted context menu item
changes. See below for more details.

(WebKit::WebContextMenuProxyMac::applyMarkupToControlledImage):
* UIProcess/mac/WebPageProxyMac.mm:
(WebKit::WebPageProxy::willHighlightContextMenuItem):

This method coordinates showing or hiding the cropped image overlay when the user highlights (but does not yet
select) the "Copy Cropped Image" context menu item. When the user first highlights this item, we trigger an
analysis request in VisionKit; when we later obtain the results, we send the image to the web process as TIFF
data, and inject it into the image using `ImageOverlay::CroppedImage::install`. Un-highlighting and re-
highlighting the menu item then respectively hides and shows the cropped image overlay by setting visibility.

(WebKit::WebPageProxy::handleContextMenuCopyCroppedImage):

Adjust this to use the cached `m_croppedImageResult` if it has already been computed.

* WebProcess/WebPage/WebContextMenu.cpp:
(WebKit::WebContextMenu::show):
* WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::didDismissContextMenu):

Destroy the `m_croppedImageOverlay` after the context menu is dismissed (which uninstalls the cropped image
overlay in the process).

(WebKit::WebPage::installCroppedImageOverlay):
(WebKit::WebPage::setCroppedImageOverlayVisibility):

Add methods to install and toggle the visibility of the cropped image overlay.

* WebProcess/WebPage/WebPage.h:
* WebProcess/WebPage/WebPage.messages.in:

Modified Paths

Diff

Modified: trunk/Source/WebKit/ChangeLog (290210 => 290211)


--- trunk/Source/WebKit/ChangeLog	2022-02-19 17:51:44 UTC (rev 290210)
+++ trunk/Source/WebKit/ChangeLog	2022-02-19 19:24:02 UTC (rev 290211)
@@ -1,3 +1,81 @@
+2022-02-19  Wenson Hsieh  <wenson_hs...@apple.com>
+
+        [macOS] Hovering over "Copy Cropped Image" context menu item should reveal the cropped image
+        https://bugs.webkit.org/show_bug.cgi?id=236845
+        rdar://89152746
+
+        Reviewed by Dean Jackson.
+
+        Add support for revealing the "cropped" region of an image element (while dimming out the rest of the image)
+        when the user highlights the new "Copy Cropped Image" context menu item on macOS. See below for more details.
+
+        * Platform/cocoa/TextRecognitionUtilities.h:
+
+        Adjust `requestImageAnalysisMarkup` to take a completion handler with both the resulting CGImageRef, as well as
+        a CGRect that represents the frame of the cropped image with respect to the source image, normalized to the unit
+        square. Additionally, tweak all call sites of `requestImageAnalysisMarkup` to pass in completion handlers that
+        accept both arguments.
+
+        * Shared/ContextMenuContextData.cpp:
+        (WebKit::ContextMenuContextData::ContextMenuContextData):
+        (WebKit::ContextMenuContextData::encode const):
+        (WebKit::ContextMenuContextData::decode):
+        * Shared/ContextMenuContextData.h:
+        (WebKit::ContextMenuContextData::hitTestedElementContext const):
+
+        Add an optional ElementContext representing the inner hit-tested element for the context menu. We use this below
+        in `WebPageProxy::willHighlightContextMenuItem` to know where we need to install the cropped image overlay.
+
+        * UIProcess/WebPageProxy.cpp:
+        (WebKit::WebPageProxy::didDismissContextMenu):
+
+        Reset the cached cropped image result as well as the cropped image overlay state when the context menu is
+        dismissed.
+
+        (WebKit::WebPageProxy::resetStateAfterProcessExited):
+
+        Also clear the above state if the process crashes.
+
+        * UIProcess/WebPageProxy.h:
+        * UIProcess/ios/WKContentViewInteraction.mm:
+        (-[WKContentView doAfterComputingImageAnalysisResultsForMarkup:]):
+        (-[WKContentView actionSheetAssistant:copyCroppedImage:sourceMIMEType:]):
+        * UIProcess/mac/WebContextMenuProxyMac.mm:
+        (-[WKMenuDelegate menu:willHighlightItem:]):
+
+        Implement this NSMenu delegate method and call out to WebPageProxy whenever the highlighted context menu item
+        changes. See below for more details.
+
+        (WebKit::WebContextMenuProxyMac::applyMarkupToControlledImage):
+        * UIProcess/mac/WebPageProxyMac.mm:
+        (WebKit::WebPageProxy::willHighlightContextMenuItem):
+
+        This method coordinates showing or hiding the cropped image overlay when the user highlights (but does not yet
+        select) the "Copy Cropped Image" context menu item. When the user first highlights this item, we trigger an
+        analysis request in VisionKit; when we later obtain the results, we send the image to the web process as TIFF
+        data, and inject it into the image using `ImageOverlay::CroppedImage::install`. Un-highlighting and re-
+        highlighting the menu item then respectively hides and shows the cropped image overlay by setting visibility.
+
+        (WebKit::WebPageProxy::handleContextMenuCopyCroppedImage):
+
+        Adjust this to use the cached `m_croppedImageResult` if it has already been computed.
+
+        * WebProcess/WebPage/WebContextMenu.cpp:
+        (WebKit::WebContextMenu::show):
+        * WebProcess/WebPage/WebPage.cpp:
+        (WebKit::WebPage::didDismissContextMenu):
+
+        Destroy the `m_croppedImageOverlay` after the context menu is dismissed (which uninstalls the cropped image
+        overlay in the process).
+
+        (WebKit::WebPage::installCroppedImageOverlay):
+        (WebKit::WebPage::setCroppedImageOverlayVisibility):
+
+        Add methods to install and toggle the visibility of the cropped image overlay.
+
+        * WebProcess/WebPage/WebPage.h:
+        * WebProcess/WebPage/WebPage.messages.in:
+
 2022-02-19  Kimmo Kinnunen  <kkinnu...@apple.com>
 
         REGRESSION(r290175): ASSERTION FAILED: platformSample.type == PlatformSample::CMSampleBufferType while running webgl tests

Modified: trunk/Source/WebKit/Platform/cocoa/TextRecognitionUtilities.h (290210 => 290211)


--- trunk/Source/WebKit/Platform/cocoa/TextRecognitionUtilities.h	2022-02-19 17:51:44 UTC (rev 290210)
+++ trunk/Source/WebKit/Platform/cocoa/TextRecognitionUtilities.h	2022-02-19 19:24:02 UTC (rev 290211)
@@ -59,7 +59,7 @@
 
 #if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
 void requestImageAnalysisWithIdentifier(CocoaImageAnalyzer *, const String& identifier, CGImageRef, CompletionHandler<void(WebCore::TextRecognitionResult&&)>&&);
-void requestImageAnalysisMarkup(CGImageRef, CompletionHandler<void(CGImageRef)>&&);
+void requestImageAnalysisMarkup(CGImageRef, CompletionHandler<void(CGImageRef, CGRect)>&&);
 #endif
 
 }

Modified: trunk/Source/WebKit/Shared/ContextMenuContextData.cpp (290210 => 290211)


--- trunk/Source/WebKit/Shared/ContextMenuContextData.cpp	2022-02-19 17:51:44 UTC (rev 290210)
+++ trunk/Source/WebKit/Shared/ContextMenuContextData.cpp	2022-02-19 19:24:02 UTC (rev 290211)
@@ -43,7 +43,7 @@
 {
 }
 
-ContextMenuContextData::ContextMenuContextData(const WebCore::IntPoint& menuLocation, const Vector<WebKit::WebContextMenuItemData>& menuItems, const ContextMenuContext& context)
+ContextMenuContextData::ContextMenuContextData(const IntPoint& menuLocation, std::optional<ElementContext>&& hitTestedElementContext, const Vector<WebKit::WebContextMenuItemData>& menuItems, const ContextMenuContext& context)
 #if ENABLE(SERVICE_CONTROLS)
     : m_type(context.controlledImage() ? Type::ServicesMenu : context.type())
 #else
@@ -50,6 +50,7 @@
     : m_type(context.type())
 #endif
     , m_menuLocation(menuLocation)
+    , m_hitTestedElementContext(WTFMove(hitTestedElementContext))
     , m_menuItems(menuItems)
     , m_webHitTestResultData({ context.hitTestResult(), true })
     , m_selectedText(context.selectedText())
@@ -94,6 +95,7 @@
 {
     encoder << m_type;
     encoder << m_menuLocation;
+    encoder << m_hitTestedElementContext;
     encoder << m_menuItems;
     encoder << m_webHitTestResultData;
     encoder << m_selectedText;
@@ -121,6 +123,9 @@
     if (!decoder.decode(result.m_menuLocation))
         return false;
 
+    if (!decoder.decode(result.m_hitTestedElementContext))
+        return false;
+
     if (!decoder.decode(result.m_menuItems))
         return false;
 

Modified: trunk/Source/WebKit/Shared/ContextMenuContextData.h (290210 => 290211)


--- trunk/Source/WebKit/Shared/ContextMenuContextData.h	2022-02-19 17:51:44 UTC (rev 290210)
+++ trunk/Source/WebKit/Shared/ContextMenuContextData.h	2022-02-19 19:24:02 UTC (rev 290211)
@@ -46,12 +46,13 @@
     using Type = WebCore::ContextMenuContext::Type;
 
     ContextMenuContextData();
-    ContextMenuContextData(const WebCore::IntPoint& menuLocation, const Vector<WebKit::WebContextMenuItemData>& menuItems, const WebCore::ContextMenuContext&);
+    ContextMenuContextData(const WebCore::IntPoint& menuLocation, std::optional<WebCore::ElementContext>&& hitTestedElementContext, const Vector<WebKit::WebContextMenuItemData>& menuItems, const WebCore::ContextMenuContext&);
 
     Type type() const { return m_type; }
     const WebCore::IntPoint& menuLocation() const { return m_menuLocation; }
     const Vector<WebKit::WebContextMenuItemData>& menuItems() const { return m_menuItems; }
 
+    std::optional<WebCore::ElementContext> hitTestedElementContext() const { return m_hitTestedElementContext; }
     const std::optional<WebHitTestResultData>& webHitTestResultData() const { return m_webHitTestResultData; }
     const String& selectedText() const { return m_selectedText; }
 
@@ -86,6 +87,7 @@
     Type m_type;
 
     WebCore::IntPoint m_menuLocation;
+    std::optional<WebCore::ElementContext> m_hitTestedElementContext;
     Vector<WebKit::WebContextMenuItemData> m_menuItems;
 
     std::optional<WebHitTestResultData> m_webHitTestResultData;

Modified: trunk/Source/WebKit/UIProcess/WebPageProxy.cpp (290210 => 290211)


--- trunk/Source/WebKit/UIProcess/WebPageProxy.cpp	2022-02-19 17:51:44 UTC (rev 290210)
+++ trunk/Source/WebKit/UIProcess/WebPageProxy.cpp	2022-02-19 19:24:02 UTC (rev 290211)
@@ -7005,6 +7005,11 @@
     send(Messages::WebPage::DidDismissContextMenu());
 
     pageClient().didDismissContextMenu();
+
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+    m_croppedImageResult = { };
+    m_croppedImageOverlayState = CroppedImageOverlayState::Inactive;
+#endif
 }
 
 void WebPageProxy::contextMenuItemSelected(const WebContextMenuItemData& item)
@@ -8140,6 +8145,11 @@
     m_currentFullscreenVideoSessionIdentifier = std::nullopt;
 #endif
 
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+    m_croppedImageResult = { };
+    m_croppedImageOverlayState = CroppedImageOverlayState::Inactive;
+#endif
+
     // FIXME: <rdar://problem/38676604> In case of process swaps, the old process should gracefully suspend instead of terminating.
     m_process->processTerminated();
 }

Modified: trunk/Source/WebKit/UIProcess/WebPageProxy.h (290210 => 290211)


--- trunk/Source/WebKit/UIProcess/WebPageProxy.h	2022-02-19 17:51:44 UTC (rev 290210)
+++ trunk/Source/WebKit/UIProcess/WebPageProxy.h	2022-02-19 19:24:02 UTC (rev 290211)
@@ -1371,6 +1371,7 @@
     void didDismissContextMenu();
     void contextMenuItemSelected(const WebContextMenuItemData&);
     void handleContextMenuKeyEvent();
+    void willHighlightContextMenuItem(WebCore::ContextMenuAction);
 #endif
 
     // Called by the WebOpenPanelResultListenerProxy.
@@ -3211,6 +3212,17 @@
     std::optional<PlaybackSessionContextIdentifier> m_currentFullscreenVideoSessionIdentifier;
     RunLoop::Timer<WebPageProxy> m_fullscreenVideoExtractionTimer;
 #endif
+
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+    enum class CroppedImageOverlayState : uint8_t {
+        Inactive,
+        Analyzing,
+        Hidden,
+        Showing,
+    };
+    WebCore::PlatformImagePtr m_croppedImageResult;
+    CroppedImageOverlayState m_croppedImageOverlayState { CroppedImageOverlayState::Inactive };
+#endif // ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
 };
 
 #ifdef __OBJC__

Modified: trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm (290210 => 290211)


--- trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm	2022-02-19 17:51:44 UTC (rev 290210)
+++ trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm	2022-02-19 19:24:02 UTC (rev 290211)
@@ -4765,7 +4765,7 @@
             return;
         }
 
-        WebKit::requestImageAnalysisMarkup(cgImage.get(), [sourceMIMEType, context, completion = WTFMove(completion), weakSelf](CGImageRef result) mutable {
+        WebKit::requestImageAnalysisMarkup(cgImage.get(), [sourceMIMEType, context, completion = WTFMove(completion), weakSelf](CGImageRef result, CGRect) mutable {
             auto strongSelf = weakSelf.get();
             if (!strongSelf) {
                 completion();
@@ -10903,7 +10903,7 @@
 - (void)actionSheetAssistant:(WKActionSheetAssistant *)assistant copyCroppedImage:(UIImage *)image sourceMIMEType:(NSString *)sourceMIMEType
 {
     auto changeCount = UIPasteboard.generalPasteboard.changeCount;
-    WebKit::requestImageAnalysisMarkup(image.CGImage, [changeCount, weakSelf = WeakObjCPtr<WKContentView>(self), originalImage = RetainPtr { image }, sourceMIMEType = RetainPtr { sourceMIMEType }](CGImageRef result) mutable {
+    WebKit::requestImageAnalysisMarkup(image.CGImage, [changeCount, weakSelf = WeakObjCPtr<WKContentView>(self), originalImage = RetainPtr { image }, sourceMIMEType = RetainPtr { sourceMIMEType }](CGImageRef result, CGRect) mutable {
         auto strongSelf = weakSelf.get();
         if (!strongSelf)
             return;

Modified: trunk/Source/WebKit/UIProcess/mac/WebContextMenuProxyMac.mm (290210 => 290211)


--- trunk/Source/WebKit/UIProcess/mac/WebContextMenuProxyMac.mm	2022-02-19 17:51:44 UTC (rev 290210)
+++ trunk/Source/WebKit/UIProcess/mac/WebContextMenuProxyMac.mm	2022-02-19 19:24:02 UTC (rev 290211)
@@ -186,6 +186,11 @@
     _menuProxy->page()->didDismissContextMenu();
 }
 
+- (void)menu:(NSMenu *)menu willHighlightItem:(NSMenuItem *)item
+{
+    _menuProxy->page()->willHighlightContextMenuItem(static_cast<WebCore::ContextMenuAction>(item.tag));
+}
+
 @end
 
 namespace WebKit {
@@ -343,7 +348,7 @@
     if (!image)
         return;
 
-    requestImageAnalysisMarkup(image.get(), [weakPage = WeakPtr { page() }, preferredMIMEType = m_context.controlledImageMIMEType(), elementContext = WTFMove(*elementContext)](CGImageRef result) {
+    requestImageAnalysisMarkup(image.get(), [weakPage = WeakPtr { page() }, preferredMIMEType = m_context.controlledImageMIMEType(), elementContext = WTFMove(*elementContext)](CGImageRef result, CGRect) {
         RefPtr protectedPage = weakPage.get();
         if (!protectedPage || !result)
             return;

Modified: trunk/Source/WebKit/UIProcess/mac/WebPageProxyMac.mm (290210 => 290211)


--- trunk/Source/WebKit/UIProcess/mac/WebPageProxyMac.mm	2022-02-19 17:51:44 UTC (rev 290210)
+++ trunk/Source/WebKit/UIProcess/mac/WebPageProxyMac.mm	2022-02-19 19:24:02 UTC (rev 290211)
@@ -775,26 +775,103 @@
 
 #endif // ENABLE(IMAGE_ANALYSIS)
 
+void WebPageProxy::willHighlightContextMenuItem(ContextMenuAction action)
+{
 #if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+    if (action != ContextMenuItemTagCopyCroppedImage) {
+        if (m_croppedImageOverlayState == CroppedImageOverlayState::Showing) {
+            m_croppedImageOverlayState = CroppedImageOverlayState::Hidden;
+            send(Messages::WebPage::SetCroppedImageOverlayVisibility(false));
+        }
+        return;
+    }
 
+    if (m_croppedImageOverlayState == CroppedImageOverlayState::Hidden) {
+        m_croppedImageOverlayState = CroppedImageOverlayState::Showing;
+        send(Messages::WebPage::SetCroppedImageOverlayVisibility(true));
+        return;
+    }
+
+    if (m_croppedImageOverlayState != CroppedImageOverlayState::Inactive)
+        return;
+
+    auto elementContext = m_activeContextMenuContextData.hitTestedElementContext();
+    if (!elementContext)
+        return;
+
+    auto& hitTestData = m_activeContextMenuContextData.webHitTestResultData().value();
+    auto imageBitmap = hitTestData.imageBitmap;
+    if (!imageBitmap)
+        return;
+
+    auto image = imageBitmap->makeCGImageCopy();
+    if (!image)
+        return;
+
+    m_croppedImageOverlayState = CroppedImageOverlayState::Analyzing;
+
+    requestImageAnalysisMarkup(image.get(), [weakPage = WeakPtr { *this }, elementContext = WTFMove(*elementContext)](CGImageRef resultImage, CGRect normalizedCropRect) {
+        if (!resultImage || CGRectIsEmpty(normalizedCropRect))
+            return;
+
+        RefPtr protectedPage = weakPage.get();
+        if (!protectedPage)
+            return;
+
+        protectedPage->m_croppedImageResult = resultImage;
+
+        if (protectedPage->m_croppedImageOverlayState != CroppedImageOverlayState::Analyzing)
+            return;
+
+        auto tiffData = transcode(resultImage, (__bridge CFStringRef)UTTypeTIFF.identifier);
+        if (!tiffData)
+            return;
+
+        auto sharedMemory = SharedMemory::allocate([tiffData length]);
+        if (!sharedMemory)
+            return;
+
+        [tiffData getBytes:sharedMemory->data() length:[tiffData length]];
+
+        SharedMemory::Handle handle;
+        sharedMemory->createHandle(handle, SharedMemory::Protection::ReadOnly);
+        protectedPage->send(Messages::WebPage::InstallCroppedImageOverlay(elementContext, { WTFMove(handle), sharedMemory->size() }, "image/tiff"_s, normalizedCropRect));
+        protectedPage->m_croppedImageOverlayState = CroppedImageOverlayState::Showing;
+    });
+#else
+    UNUSED_PARAM(action);
+#endif
+}
+
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+
 void WebPageProxy::handleContextMenuCopyCroppedImage(ShareableBitmap& imageBitmap, const String& preferredMIMEType)
 {
-    auto originalImage = imageBitmap.makeCGImage();
-    if (!originalImage)
-        return;
-
     auto changeCount = NSPasteboard.generalPasteboard.changeCount;
-    requestImageAnalysisMarkup(originalImage.get(), [changeCount, originalImage, preferredMIMEType](CGImageRef resultImage) {
+    auto performCopy = [changeCount, preferredMIMEType](CGImageRef resultImage) {
         auto pasteboard = NSPasteboard.generalPasteboard;
-        if (changeCount != pasteboard.changeCount)
+        if (changeCount != pasteboard.changeCount || !resultImage)
             return;
 
-        auto [data, type] = WebKit::transcodeWithPreferredMIMEType(resultImage ?: originalImage.get(), preferredMIMEType.createCFString().get(), (__bridge CFStringRef)UTTypeTIFF.identifier);
+        auto [data, type] = transcodeWithPreferredMIMEType(resultImage, preferredMIMEType.createCFString().get(), (__bridge CFStringRef)UTTypeTIFF.identifier);
         if (!data)
             return;
 
         [pasteboard declareTypes:@[(__bridge NSString *)type.get()] owner:nil];
         [pasteboard setData:data.get() forType:(__bridge NSString *)type.get()];
+    };
+
+    if (m_croppedImageResult) {
+        performCopy(m_croppedImageResult.get());
+        return;
+    }
+
+    auto originalImage = imageBitmap.makeCGImageCopy();
+    if (!originalImage)
+        return;
+
+    requestImageAnalysisMarkup(originalImage.get(), [performCopy = WTFMove(performCopy)](auto image, auto) {
+        performCopy(image);
     });
 }
 

Modified: trunk/Source/WebKit/WebProcess/WebPage/WebContextMenu.cpp (290210 => 290211)


--- trunk/Source/WebKit/WebProcess/WebPage/WebContextMenu.cpp	2022-02-19 17:51:44 UTC (rev 290210)
+++ trunk/Source/WebKit/WebProcess/WebPage/WebContextMenu.cpp	2022-02-19 19:24:02 UTC (rev 290211)
@@ -63,9 +63,13 @@
     RefPtr<API::Object> userData;
     menuItemsWithUserData(menuItems, userData);
 
+    std::optional<ElementContext> hitTestedElementContext;
+    if (RefPtr hitTestedElement = controller.hitTestResult().innerNonSharedElement())
+        hitTestedElementContext = m_page->contextForElement(*hitTestedElement);
+
     auto menuLocation = view->contentsToRootView(controller.hitTestResult().roundedPointInInnerNodeFrame());
 
-    ContextMenuContextData contextMenuContextData(menuLocation, menuItems, controller.context());
+    ContextMenuContextData contextMenuContextData(menuLocation, WTFMove(hitTestedElementContext), menuItems, controller.context());
 
     // Mark the WebPage has having a shown context menu then notify the UIProcess.
     m_page->startWaitingForContextMenuToShow();

Modified: trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp (290210 => 290211)


--- trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp	2022-02-19 17:51:44 UTC (rev 290210)
+++ trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp	2022-02-19 19:24:02 UTC (rev 290211)
@@ -2905,6 +2905,10 @@
 void WebPage::didDismissContextMenu()
 {
     corePage()->contextMenuController().didDismissContextMenu();
+
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+    m_croppedImageOverlay = nullptr;
+#endif
 }
 
 #endif // ENABLE(CONTEXT_MENUS)
@@ -7791,6 +7795,29 @@
     completion(handle, mimeType);
 }
 
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+
+void WebPage::installCroppedImageOverlay(const ElementContext& context, const SharedMemory::IPCHandle& imageData, const String& mimeType, FloatRect normalizedCropRect)
+{
+    auto sharedMemory = SharedMemory::map(imageData.handle, SharedMemory::Protection::ReadOnly);
+    if (!sharedMemory)
+        return;
+
+    RefPtr element = dynamicDowncast<HTMLElement>(elementForContext(context).get());
+    if (!element)
+        return;
+
+    m_croppedImageOverlay = ImageOverlay::CroppedImage::install(*element, sharedMemory->createSharedBuffer(imageData.dataSize), mimeType, normalizedCropRect);
+}
+
+void WebPage::setCroppedImageOverlayVisibility(bool visible)
+{
+    if (m_croppedImageOverlay)
+        m_croppedImageOverlay->setVisibility(visible);
+}
+
+#endif // ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+
 #if ENABLE(MEDIA_CONTROLS_CONTEXT_MENUS) && USE(UICONTEXTMENU)
 void WebPage::showMediaControlsContextMenu(FloatRect&& targetFrame, Vector<MediaControlsContextMenuItem>&& items, CompletionHandler<void(MediaControlsContextMenuItem::ID)>&& completionHandler)
 {

Modified: trunk/Source/WebKit/WebProcess/WebPage/WebPage.h (290210 => 290211)


--- trunk/Source/WebKit/WebProcess/WebPage/WebPage.h	2022-02-19 17:51:44 UTC (rev 290210)
+++ trunk/Source/WebKit/WebProcess/WebPage/WebPage.h	2022-02-19 19:24:02 UTC (rev 290211)
@@ -268,7 +268,11 @@
 #if HAVE(TRANSLATION_UI_SERVICES) && ENABLE(CONTEXT_MENUS)
 struct TranslationContextMenuInfo;
 #endif
+
+namespace ImageOverlay {
+class CroppedImage;
 }
+}
 
 namespace WebKit {
 
@@ -1452,6 +1456,10 @@
 #endif
 
     void requestImageBitmap(const WebCore::ElementContext&, CompletionHandler<void(const ShareableBitmap::Handle&, const String& sourceMIMEType)>&&);
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+    void installCroppedImageOverlay(const WebCore::ElementContext&, const SharedMemory::IPCHandle& imageData, const String& mimeType, WebCore::FloatRect normalizedCropRect);
+    void setCroppedImageOverlayVisibility(bool);
+#endif
 
 #if HAVE(TRANSLATION_UI_SERVICES) && ENABLE(CONTEXT_MENUS)
     void handleContextMenuTranslation(const WebCore::TranslationContextMenuInfo&);
@@ -2441,6 +2449,10 @@
 #if ENABLE(APP_HIGHLIGHTS)
     WebCore::HighlightVisibility m_appHighlightsVisible { WebCore::HighlightVisibility::Hidden };
 #endif
+
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+    std::unique_ptr<WebCore::ImageOverlay::CroppedImage> m_croppedImageOverlay;
+#endif
 };
 
 #if !PLATFORM(IOS_FAMILY)

Modified: trunk/Source/WebKit/WebProcess/WebPage/WebPage.messages.in (290210 => 290211)


--- trunk/Source/WebKit/WebProcess/WebPage/WebPage.messages.in	2022-02-19 17:51:44 UTC (rev 290210)
+++ trunk/Source/WebKit/WebProcess/WebPage/WebPage.messages.in	2022-02-19 19:24:02 UTC (rev 290211)
@@ -134,6 +134,11 @@
 
     RequestImageBitmap(struct WebCore::ElementContext elementContext) -> (WebKit::ShareableBitmap::Handle image, String sourceMIMEType) Async
 
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+    InstallCroppedImageOverlay(struct WebCore::ElementContext elementContext, WebKit::SharedMemory::IPCHandle imageData, String mimeType, WebCore::FloatRect normalizedCropRect)
+    SetCroppedImageOverlayVisibility(bool visible)
+#endif
+
     SetControlledByAutomation(bool controlled)
 
     ConnectInspector(String targetId, Inspector::FrontendChannel::ConnectionType connectionType)
_______________________________________________
webkit-changes mailing list
webkit-changes@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-changes

Reply via email to