[webkit-changes] [275270] trunk/Source/WebKit

Tue, 30 Mar 2021 22:53:22 -0700

Title: [275270] trunk/Source/WebKit
Revision
275270
Author
wenson_hs...@apple.com
Date
2021-03-30 22:52:41 -0700 (Tue, 30 Mar 2021)

Log Message

[iOS] Fall back to context menu presentation after long pressing image overlay text
https://bugs.webkit.org/show_bug.cgi?id=223967
<rdar://problem/76028620>

Reviewed by Tim Horton.

Add support for this behavior; see <rdar://problem/76028620> for more details.

* Shared/ios/InteractionInformationAtPosition.h:
* Shared/ios/InteractionInformationAtPosition.mm:
(WebKit::InteractionInformationAtPosition::encode const):
(WebKit::InteractionInformationAtPosition::decode):

Add a couple of new flags (`isSelected` and `isImageOverlayText`) to indicate whether the interaction is over
selected text, and also whether the interaction is over text in an image overlay, respectively.

* Shared/ios/InteractionInformationRequest.cpp:
(WebKit::InteractionInformationRequest::encode const):
(WebKit::InteractionInformationRequest::decode):
(WebKit::InteractionInformationRequest::isValidForRequest const):

Add a new position information request option to ignore user agent shadow root content.

* Shared/ios/InteractionInformationRequest.h:
* UIProcess/ios/WKContentViewInteraction.h:
* UIProcess/ios/WKContentViewInteraction.mm:

Add a flag that's set when `_imageExtractionTimeoutGestureRecognizer` is recognized, and is about to present a
context menu, and is unset upon starting the context menu presentation.

(-[WKContentView setUpInteraction]):
(-[WKContentView cleanUpInteraction]):
(-[WKContentView gestureRecognizer:shouldRequireFailureOfGestureRecognizer:]):
(-[WKContentView gestureRecognizer:shouldBeRequiredToFailByGestureRecognizer:]):

Make the timeout gesture (`_imageExtractionTimeoutGestureRecognizer`) require the failure of the context menu
initiation gesture, such that this image extraction timeout gesture won't trigger if the context menu has
already been triggered.

(-[WKContentView _invalidateCurrentPositionInformation]):
(-[WKContentView _didCommitLoadForMainFrame]):

Pull logic for resetting cached position information into a helper, and call it inside
`-_didCommitLoadForMainFrame`.

(-[WKContentView _contextMenuInteraction:configurationForMenuAtLocation:completion:]):

Reset the new flag (`_contextMenuWasTriggeredByImageExtractionTimeout`) if necessary, and use its existing value
to determine whether we should ignore UA shadow root content when performing a hit-test for the context menu
configuration.

* WebProcess/WebPage/ios/WebPageIOS.mm:
(WebKit::imageRendererAndImage):
(WebKit::imagePositionInformation):
(WebKit::elementPositionInformation):

Additionally supply the image URL and image data if `includeImageData` is set, even if the hit-tested node is
just a text node underneath an image element's overlay.

(WebKit::selectionPositionInformation):
(WebKit::WebPage::positionInformation):

Add support for the new position information flags.

Modified Paths

Diff

Modified: trunk/Source/WebKit/ChangeLog (275269 => 275270)


--- trunk/Source/WebKit/ChangeLog	2021-03-31 05:48:55 UTC (rev 275269)
+++ trunk/Source/WebKit/ChangeLog	2021-03-31 05:52:41 UTC (rev 275270)
@@ -1,3 +1,69 @@
+2021-03-30  Wenson Hsieh  <wenson_hs...@apple.com>
+
+        [iOS] Fall back to context menu presentation after long pressing image overlay text
+        https://bugs.webkit.org/show_bug.cgi?id=223967
+        <rdar://problem/76028620>
+
+        Reviewed by Tim Horton.
+
+        Add support for this behavior; see <rdar://problem/76028620> for more details.
+
+        * Shared/ios/InteractionInformationAtPosition.h:
+        * Shared/ios/InteractionInformationAtPosition.mm:
+        (WebKit::InteractionInformationAtPosition::encode const):
+        (WebKit::InteractionInformationAtPosition::decode):
+
+        Add a couple of new flags (`isSelected` and `isImageOverlayText`) to indicate whether the interaction is over
+        selected text, and also whether the interaction is over text in an image overlay, respectively.
+
+        * Shared/ios/InteractionInformationRequest.cpp:
+        (WebKit::InteractionInformationRequest::encode const):
+        (WebKit::InteractionInformationRequest::decode):
+        (WebKit::InteractionInformationRequest::isValidForRequest const):
+
+        Add a new position information request option to ignore user agent shadow root content.
+
+        * Shared/ios/InteractionInformationRequest.h:
+        * UIProcess/ios/WKContentViewInteraction.h:
+        * UIProcess/ios/WKContentViewInteraction.mm:
+
+        Add a flag that's set when `_imageExtractionTimeoutGestureRecognizer` is recognized, and is about to present a
+        context menu, and is unset upon starting the context menu presentation.
+
+        (-[WKContentView setUpInteraction]):
+        (-[WKContentView cleanUpInteraction]):
+        (-[WKContentView gestureRecognizer:shouldRequireFailureOfGestureRecognizer:]):
+        (-[WKContentView gestureRecognizer:shouldBeRequiredToFailByGestureRecognizer:]):
+
+        Make the timeout gesture (`_imageExtractionTimeoutGestureRecognizer`) require the failure of the context menu
+        initiation gesture, such that this image extraction timeout gesture won't trigger if the context menu has
+        already been triggered.
+
+        (-[WKContentView _invalidateCurrentPositionInformation]):
+        (-[WKContentView _didCommitLoadForMainFrame]):
+
+        Pull logic for resetting cached position information into a helper, and call it inside
+        `-_didCommitLoadForMainFrame`.
+
+        (-[WKContentView _contextMenuInteraction:configurationForMenuAtLocation:completion:]):
+
+        Reset the new flag (`_contextMenuWasTriggeredByImageExtractionTimeout`) if necessary, and use its existing value
+        to determine whether we should ignore UA shadow root content when performing a hit-test for the context menu
+        configuration.
+
+        * WebProcess/WebPage/ios/WebPageIOS.mm:
+        (WebKit::imageRendererAndImage):
+        (WebKit::imagePositionInformation):
+        (WebKit::elementPositionInformation):
+
+        Additionally supply the image URL and image data if `includeImageData` is set, even if the hit-tested node is
+        just a text node underneath an image element's overlay.
+
+        (WebKit::selectionPositionInformation):
+        (WebKit::WebPage::positionInformation):
+
+        Add support for the new position information flags.
+
 2021-03-30  Chris Dumez  <cdu...@apple.com>
 
         ASSERT(m_sendPort) in IPC::Connection::open() when running some iOS unit tests

Modified: trunk/Source/WebKit/Shared/ios/InteractionInformationAtPosition.h (275269 => 275270)


--- trunk/Source/WebKit/Shared/ios/InteractionInformationAtPosition.h	2021-03-31 05:48:55 UTC (rev 275269)
+++ trunk/Source/WebKit/Shared/ios/InteractionInformationAtPosition.h	2021-03-31 05:52:41 UTC (rev 275270)
@@ -54,6 +54,7 @@
     bool canBeValid { true };
     Optional<bool> nodeAtPositionHasDoubleClickHandler;
     bool isSelectable { false };
+    bool isSelected { false };
     bool prefersDraggingOverTextSelection { false };
     bool isNearMarkedText { false };
     bool touchCalloutEnabled { true };
@@ -70,6 +71,7 @@
     bool preventTextInteraction { false };
 #endif
     bool shouldNotUseIBeamInEditableContent { false };
+    bool isImageOverlayText { false };
     WebCore::FloatPoint adjustedPointForNodeRespondingToClickEvents;
     URL url;
     URL imageURL;

Modified: trunk/Source/WebKit/Shared/ios/InteractionInformationAtPosition.mm (275269 => 275270)


--- trunk/Source/WebKit/Shared/ios/InteractionInformationAtPosition.mm	2021-03-31 05:48:55 UTC (rev 275269)
+++ trunk/Source/WebKit/Shared/ios/InteractionInformationAtPosition.mm	2021-03-31 05:52:41 UTC (rev 275270)
@@ -45,6 +45,7 @@
     encoder << canBeValid;
     encoder << nodeAtPositionHasDoubleClickHandler;
     encoder << isSelectable;
+    encoder << isSelected;
     encoder << prefersDraggingOverTextSelection;
     encoder << isNearMarkedText;
     encoder << touchCalloutEnabled;
@@ -85,6 +86,7 @@
     encoder << preventTextInteraction;
 #endif
     encoder << shouldNotUseIBeamInEditableContent;
+    encoder << isImageOverlayText;
     encoder << elementContext;
 }
 
@@ -102,6 +104,9 @@
     if (!decoder.decode(result.isSelectable))
         return false;
 
+    if (!decoder.decode(result.isSelected))
+        return false;
+
     if (!decoder.decode(result.prefersDraggingOverTextSelection))
         return false;
 
@@ -204,6 +209,9 @@
     if (!decoder.decode(result.shouldNotUseIBeamInEditableContent))
         return false;
 
+    if (!decoder.decode(result.isImageOverlayText))
+        return false;
+
     if (!decoder.decode(result.elementContext))
         return false;

Modified: trunk/Source/WebKit/Shared/ios/InteractionInformationRequest.cpp (275269 => 275270)


--- trunk/Source/WebKit/Shared/ios/InteractionInformationRequest.cpp	2021-03-31 05:48:55 UTC (rev 275269)
+++ trunk/Source/WebKit/Shared/ios/InteractionInformationRequest.cpp	2021-03-31 05:52:41 UTC (rev 275270)
@@ -42,6 +42,7 @@
     encoder << includeHasDoubleClickHandler;
     encoder << includeImageData;
     encoder << linkIndicatorShouldHaveLegacyMargins;
+    encoder << disallowUserAgentShadowContent;
 }
 
 bool InteractionInformationRequest::decode(IPC::Decoder& decoder, InteractionInformationRequest& result)
@@ -67,6 +68,9 @@
     if (!decoder.decode(result.linkIndicatorShouldHaveLegacyMargins))
         return false;
 
+    if (!decoder.decode(result.disallowUserAgentShadowContent))
+        return false;
+
     return true;
 }
 
@@ -91,6 +95,9 @@
     if (other.linkIndicatorShouldHaveLegacyMargins != linkIndicatorShouldHaveLegacyMargins)
         return false;
 
+    if (other.disallowUserAgentShadowContent != disallowUserAgentShadowContent)
+        return false;
+
     return (other.point - point).diagonalLengthSquared() <= radius * radius;
 }

Modified: trunk/Source/WebKit/Shared/ios/InteractionInformationRequest.h (275269 => 275270)


--- trunk/Source/WebKit/Shared/ios/InteractionInformationRequest.h	2021-03-31 05:48:55 UTC (rev 275269)
+++ trunk/Source/WebKit/Shared/ios/InteractionInformationRequest.h	2021-03-31 05:52:41 UTC (rev 275270)
@@ -46,6 +46,7 @@
     bool includeImageData { false };
 
     bool linkIndicatorShouldHaveLegacyMargins { false };
+    bool disallowUserAgentShadowContent { false };
 
     InteractionInformationRequest() { }
     explicit InteractionInformationRequest(WebCore::IntPoint point)

Modified: trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.h (275269 => 275270)


--- trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.h	2021-03-31 05:48:55 UTC (rev 275269)
+++ trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.h	2021-03-31 05:52:41 UTC (rev 275270)
@@ -475,7 +475,9 @@
     Vector<BlockPtr<void(WebKit::ProceedWithImageExtraction)>> _actionsToPerformAfterPendingImageExtraction;
 #if USE(UICONTEXTMENU)
     RetainPtr<UIMenu> _imageExtractionContextMenu;
+    BOOL _contextMenuWasTriggeredByImageExtractionTimeout;
 #endif // USE(UICONTEXTMENU)
+    BOOL _isProceedingWithImageExtraction;
 #endif // ENABLE(IMAGE_EXTRACTION)
 
 #if USE(APPLE_INTERNAL_SDK)

Modified: trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm (275269 => 275270)


--- trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm	2021-03-31 05:48:55 UTC (rev 275269)
+++ trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm	2021-03-31 05:52:41 UTC (rev 275270)
@@ -908,6 +908,10 @@
     _isChangingFocus = NO;
     _isBlurringFocusedElement = NO;
 
+#if USE(UICONTEXTMENU) && ENABLE(IMAGE_EXTRACTION)
+    _contextMenuWasTriggeredByImageExtractionTimeout = NO;
+#endif
+
 #if ENABLE(DATALIST_ELEMENT)
     _dataListTextSuggestionsInputView = nil;
     _dataListTextSuggestions = nil;
@@ -964,6 +968,10 @@
 
     _treatAsContentEditableUntilNextEditorStateUpdate = NO;
 
+#if USE(UICONTEXTMENU) && ENABLE(IMAGE_EXTRACTION)
+    _contextMenuWasTriggeredByImageExtractionTimeout = NO;
+#endif
+
     if (_interactionViewsContainerView) {
         [self.layer removeObserver:self forKeyPath:@"transform" context:WKContentViewKVOTransformContext];
         [_interactionViewsContainerView removeFromSuperview];
@@ -2307,6 +2315,11 @@
     if ([otherGestureRecognizer isKindOfClass:WKDeferringGestureRecognizer.class])
         return [(WKDeferringGestureRecognizer *)otherGestureRecognizer shouldDeferGestureRecognizer:gestureRecognizer];
 
+#if USE(UICONTEXTMENU) && ENABLE(IMAGE_EXTRACTION)
+    if (gestureRecognizer == _imageExtractionTimeoutGestureRecognizer && otherGestureRecognizer == [_contextMenuInteraction gestureRecognizerForFailureRelationships])
+        return YES;
+#endif
+
     return NO;
 }
 
@@ -2315,6 +2328,11 @@
     if ([gestureRecognizer isKindOfClass:WKDeferringGestureRecognizer.class])
         return [(WKDeferringGestureRecognizer *)gestureRecognizer shouldDeferGestureRecognizer:otherGestureRecognizer];
 
+#if USE(UICONTEXTMENU) && ENABLE(IMAGE_EXTRACTION)
+    if (gestureRecognizer == [_contextMenuInteraction gestureRecognizerForFailureRelationships] && otherGestureRecognizer == _imageExtractionTimeoutGestureRecognizer)
+        return YES;
+#endif
+
     return NO;
 }
 
@@ -3048,6 +3066,12 @@
     _page->clearSelection();
 }
 
+- (void)_invalidateCurrentPositionInformation
+{
+    _hasValidPositionInformation = NO;
+    _positionInformation = { };
+}
+
 - (void)_positionInformationDidChange:(const WebKit::InteractionInformationAtPosition&)info
 {
     if (_lastOutstandingPositionInformationRequest && info.request.isValidForRequest(*_lastOutstandingPositionInformationRequest))
@@ -4493,8 +4517,7 @@
     _textInteractionDidChangeFocusedElement = NO;
     _activeTextInteractionCount = 0;
     _treatAsContentEditableUntilNextEditorStateUpdate = NO;
-    _hasValidPositionInformation = NO;
-    _positionInformation = { };
+    [self _invalidateCurrentPositionInformation];
 }
 
 #if !USE(UIKIT_KEYBOARD_ADDITIONS)
@@ -9954,6 +9977,12 @@
 
 - (void)_contextMenuInteraction:(UIContextMenuInteraction *)interaction configurationForMenuAtLocation:(CGPoint)location completion:(void(^)(UIContextMenuConfiguration *))completion
 {
+#if ENABLE(IMAGE_EXTRACTION)
+    BOOL triggeredByImageExtractionTimeout = std::exchange(_contextMenuWasTriggeredByImageExtractionTimeout, NO);
+#else
+    BOOL triggeredByImageExtractionTimeout = NO;
+#endif
+
     if (!_webView)
         return completion(nil);
 
@@ -9960,7 +9989,7 @@
     if (!self.webView.configuration._longPressActionsEnabled)
         return completion(nil);
 
-    auto getConfigurationAndContinue = [weakSelf = WeakObjCPtr<WKContentView>(self), interaction = retainPtr(interaction), completion = makeBlockPtr(completion)] (WebKit::ProceedWithImageExtraction proceedWithImageExtraction) {
+    auto getConfigurationAndContinue = [weakSelf = WeakObjCPtr<WKContentView>(self), interaction = retainPtr(interaction), completion = makeBlockPtr(completion), triggeredByImageExtractionTimeout] (WebKit::ProceedWithImageExtraction proceedWithImageExtraction) {
         auto strongSelf = weakSelf.get();
         if (!strongSelf || proceedWithImageExtraction == WebKit::ProceedWithImageExtraction::Yes) {
             completion(nil);
@@ -9976,6 +10005,7 @@
         WebKit::InteractionInformationRequest request { WebCore::roundedIntPoint([interaction locationInView:strongSelf.get()]) };
         request.includeSnapshot = true;
         request.includeLinkIndicator = true;
+        request.disallowUserAgentShadowContent = triggeredByImageExtractionTimeout;
         request.linkIndicatorShouldHaveLegacyMargins = ![strongSelf _shouldUseContextMenus];
 
         [strongSelf doAfterPositionInformationUpdate:[weakSelf = WeakObjCPtr<WKContentView>(strongSelf.get()), completion] (WebKit::InteractionInformationAtPosition) {

Modified: trunk/Source/WebKit/WebProcess/WebPage/ios/WebPageIOS.mm (275269 => 275270)


--- trunk/Source/WebKit/WebProcess/WebPage/ios/WebPageIOS.mm	2021-03-31 05:48:55 UTC (rev 275269)
+++ trunk/Source/WebKit/WebProcess/WebPage/ios/WebPageIOS.mm	2021-03-31 05:52:41 UTC (rev 275270)
@@ -2665,19 +2665,32 @@
 
 #endif
 
-static void imagePositionInformation(WebPage& page, Element& element, const InteractionInformationRequest& request, InteractionInformationAtPosition& info)
+static Optional<std::pair<RenderImage&, Image&>> imageRendererAndImage(Element& element)
 {
-    auto& renderImage = downcast<RenderImage>(*(element.renderer()));
+    if (!is<RenderImage>(element.renderer()))
+        return WTF::nullopt;
+
+    auto& renderImage = downcast<RenderImage>(*element.renderer());
     if (!renderImage.cachedImage() || renderImage.cachedImage()->errorOccurred())
-        return;
+        return WTF::nullopt;
 
     auto* image = renderImage.cachedImage()->imageForRenderer(&renderImage);
     if (!image || image->width() <= 1 || image->height() <= 1)
+        return WTF::nullopt;
+
+    return {{ renderImage, *image }};
+}
+
+static void imagePositionInformation(WebPage& page, Element& element, const InteractionInformationRequest& request, InteractionInformationAtPosition& info)
+{
+    auto rendererAndImage = imageRendererAndImage(element);
+    if (!rendererAndImage)
         return;
 
+    auto& [renderImage, image] = *rendererAndImage;
     info.isImage = true;
     info.imageURL = element.document().completeURL(renderImage.cachedImage()->url().string());
-    info.isAnimatedImage = image->isAnimated();
+    info.isAnimatedImage = image.isAnimated();
 
     if (request.includeSnapshot || request.includeImageData)
         info.image = createShareableBitmap(renderImage, screenSize() * page.corePage()->deviceScaleFactor());
@@ -2733,12 +2746,17 @@
 
     if (auto* renderer = element.renderer()) {
         bool shouldCollectImagePositionInformation = renderer->isRenderImage();
-#if ENABLE(IMAGE_EXTRACTION)
-        if (innerNonSharedNode && HTMLElement::isImageOverlayText(*innerNonSharedNode))
+        if (shouldCollectImagePositionInformation && innerNonSharedNode && HTMLElement::isImageOverlayText(*innerNonSharedNode)) {
             shouldCollectImagePositionInformation = false;
-#else
-        UNUSED_PARAM(innerNonSharedNode);
-#endif
+            info.isImageOverlayText = true;
+            if (request.includeImageData) {
+                if (auto rendererAndImage = imageRendererAndImage(element)) {
+                    auto& [renderImage, image] = *rendererAndImage;
+                    info.imageURL = element.document().completeURL(renderImage.cachedImage()->url().string());
+                    info.image = createShareableBitmap(renderImage, screenSize() * page.corePage()->deviceScaleFactor());
+                }
+            }
+        }
         if (shouldCollectImagePositionInformation)
             imagePositionInformation(page, element, request, info);
         boundsPositionInformation(*renderer, info);
@@ -2759,6 +2777,8 @@
 
     RenderObject* renderer = hitNode->renderer();
     boundsPositionInformation(*renderer, info);
+
+    info.isSelected = result.isSelected();
     
     if (is<Element>(*hitNode)) {
         Element& element = downcast<Element>(*hitNode);
@@ -2921,8 +2941,16 @@
         info.nodeAtPositionHasDoubleClickHandler = m_page->mainFrame().nodeRespondingToDoubleClickEvent(request.point, adjustedPoint);
 
     auto& eventHandler = m_page->mainFrame().eventHandler();
-    constexpr OptionSet<HitTestRequest::RequestType> hitType { HitTestRequest::ReadOnly, HitTestRequest::AllowFrameScrollbars, HitTestRequest::AllowVisibleChildFrameContentOnly };
-    auto hitTestResult = eventHandler.hitTestResultAtPoint(request.point, hitType);
+    auto hitTestRequestTypes = OptionSet<HitTestRequest::RequestType> {
+        HitTestRequest::ReadOnly,
+        HitTestRequest::AllowFrameScrollbars,
+        HitTestRequest::AllowVisibleChildFrameContentOnly,
+    };
+
+    if (request.disallowUserAgentShadowContent)
+        hitTestRequestTypes.add(HitTestRequest::DisallowUserAgentShadowContent);
+
+    auto hitTestResult = eventHandler.hitTestResultAtPoint(request.point, hitTestRequestTypes);
     if (auto* hitFrame = hitTestResult.innerNodeFrame()) {
         info.cursor = hitFrame->eventHandler().selectCursor(hitTestResult, false);
         if (request.includeCaretContext)

_______________________________________________
webkit-changes mailing list
webkit-changes@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-changes

Reply via email to