- Revision
- 289623
- Author
- wenson_hs...@apple.com
- Date
- 2022-02-11 09:01:52 -0800 (Fri, 11 Feb 2022)
Log Message
[iOS] Add support for a "markup image" item in the callout bar when selecting a single image
https://bugs.webkit.org/show_bug.cgi?id=236415
rdar://88714333
Reviewed by Aditya Keerthi.
Source/WebCore:
Export a couple of functions; see WebKit/ChangeLog for more details.
* platform/graphics/Image.h:
* platform/graphics/cg/UTIRegistry.h:
Source/WebKit:
This patch introduces support for a new "Markup Image" callout menu item, which is only shown when the user has
selected exactly one image. If the image is suitable for "Markup Image" (that is, the relevant VisionKit API
returns a non-null image), then we show a new item in the callout bar which, when activated, replaces the
current selection with the image returned by the aforementioned VisionKit API.
See below for more details.
* Platform/cocoa/TextRecognitionUtilities.h:
* Platform/cocoa/TextRecognitionUtilities.mm:
(WebKit::isImageAnalysisMarkupSystemFeatureEnabled):
Add a new helper method to return whether or not the corresponding system feature flag is enabled.
* Shared/EditorState.cpp:
(WebKit::EditorState::PostLayoutData::encode const):
(WebKit::EditorState::PostLayoutData::decode):
* Shared/EditorState.h:
Add an optional `ElementContext` that's populated only if the user's selection spans a single HTMLImageElement.
* UIProcess/API/ios/WKWebViewIOS.mm:
(-[WKWebView didMoveToWindow]):
* UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::requestImageBitmap):
Add a new async IPC message that allows the UI process to grab a ShareableBitmap handle (along with the MIME
type of the source image) given an ElementContext that identifies an image element.
* UIProcess/WebPageProxy.h:
* UIProcess/ios/WKContentViewInteraction.h:
Cache the last result from requesting "Image Markup" data from VisionKit in `_imageAnalysisMarkupResults`, and
use this information when invoking the new selector, `-performImageAnalysisMarkup:`.
* UIProcess/ios/WKContentViewInteraction.mm:
(WebKit::transcode):
Add a helper method that takes a CGImageRef, transcodes it to the given uniform type identifier, and finally
returns a blob of image data. Used by the two new call sites for Image Markup below, as well as an existing call
site for Visual Look Up in `-provideDataForItem:`
(-[WKContentView targetForAction:withSender:]):
Add a check for the new Image Markup action selector.
(-[WKContentView requestRectsToEvadeForSelectionCommandsWithCompletionHandler:]):
Use this existing UIKit delegate hook to defer callout bar presentation only in the case where a single image
element is selected, such that we only show the callout bar in this scenario once VisionKit has determined
whether or not the "Image Markup" action is suitable for the data of the selected image.
The initial purpose of this delegate hook was to allow WebKit to defer callout bar presentation for a short,
hard-coded delay to wait for the web page to layout out or add clickable items that might underlap the callout
bar; as such, we still perform this logic with a hard-coded delay of 250 ms (including whatever time was taken
during image analysis).
(-[WKContentView updateImageAnalysisMarkupMenuItems:]):
Append the new menu item only if the system feature flag is enabled, and we've selected an editable image. Note
that this intentionally does not consult `_imageAnalysisMarkupResults` to ensure that the state of additional
menu items in the shared menu controller is consistent between Markup Image and Quick Note, and updated in the
same lifecycle as editor state updates.
(-[WKContentView canPerformImageAnalysisMarkup]):
This is consulted when we're actually about to show the callout bar action for "Image Markup" (importantly,
after callout bar presentation deferral), and depends on the state of `_imageAnalysisMarkupResults`.
(-[WKContentView performImageAnalysisMarkup:]):
Add logic to handle the new action by calling into WebPageProxy to (basically) paste the modified image data
after transcoding it to match the image element's original source type. After #236406, this codepath is now
available on all Cocoa platforms (as opposed to macOS-only).
(-[WKContentView doAfterComputingImageAnalysisResultsForMarkup:]):
Use the new `requestImageBitmap` IPC message to grab a bitmap for the currently selected image, call out to
VisionKit to perform image analysis, and then cache the resulting image. After all of this is done, we proceed
with showing the callout bar.
In a followup, we should enforce some (reasonable) upper bound on the amount of time by which the callout bar
can be delayed.
(-[WKContentView _selectionChanged]):
(-[WKContentView setUpAdditionalMenuControllerActions]):
Refactor existing logic for supplying additional callout bar menu items, such that we update items as needed for
both Quick Note as well as Image Markup. Note that we need to preserve any existing menu items in the shared
UIMenuController, since the WebKit client (e.g. Mail) may have already supplied custom menu items. This
currently doees not affect Quick Note since the only internal client that enables Quick Note is Safari, which
does not attempt to add any of its own items; however, "Markup Image" needs to be available in Mail on iOS as
well, which does use custom menu items.
(findMenuItemWithAction):
(-[WKContentView updateAppHighlightMenuItems:]):
Make this adjust the given mutable array of UIMenuItems, which are then combined with any markup menu items
when setting custom items on the shared menu controller.
(-[WKContentView provideDataForItem:]):
Adjust this to use the new `transcode` helper function, declared above.
(-[WKContentView _setUpImageAnalysis]):
(-[WKContentView _tearDownImageAnalysis]):
(-[WKContentView setUpAppHighlightMenusIfNeeded]): Deleted.
* WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::requestImageBitmap):
* WebProcess/WebPage/WebPage.h:
* WebProcess/WebPage/WebPage.messages.in:
* WebProcess/WebPage/ios/WebPageIOS.mm:
(WebKit::WebPage::getPlatformEditorState const):
Compute and populate the new `selectedEditableImage` element context; to achieve this, we use `TextIterator` to
scan for an image element, and bail immediately if we either find anything that is not an image, or find more
than one image.
Modified Paths
Diff
Modified: trunk/Source/WebCore/ChangeLog (289622 => 289623)
--- trunk/Source/WebCore/ChangeLog 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebCore/ChangeLog 2022-02-11 17:01:52 UTC (rev 289623)
@@ -1,3 +1,16 @@
+2022-02-11 Wenson Hsieh <wenson_hs...@apple.com>
+
+ [iOS] Add support for a "markup image" item in the callout bar when selecting a single image
+ https://bugs.webkit.org/show_bug.cgi?id=236415
+ rdar://88714333
+
+ Reviewed by Aditya Keerthi.
+
+ Export a couple of functions; see WebKit/ChangeLog for more details.
+
+ * platform/graphics/Image.h:
+ * platform/graphics/cg/UTIRegistry.h:
+
2022-02-11 Kimmo Kinnunen <kkinnu...@apple.com>
Introduce a RemoteMediaSampleProxy to represent captured video frames used in Media Streams and present in GPUP
Modified: trunk/Source/WebCore/platform/graphics/Image.h (289622 => 289623)
--- trunk/Source/WebCore/platform/graphics/Image.h 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebCore/platform/graphics/Image.h 2022-02-11 17:01:52 UTC (rev 289623)
@@ -151,7 +151,7 @@
ImageObserver* imageObserver() const { return m_imageObserver; }
void setImageObserver(ImageObserver* observer) { m_imageObserver = observer; }
URL sourceURL() const;
- String mimeType() const;
+ WEBCORE_EXPORT String mimeType() const;
long long expectedContentLength() const;
enum TileRule { StretchTile, RoundTile, SpaceTile, RepeatTile };
Modified: trunk/Source/WebCore/platform/graphics/cg/UTIRegistry.h (289622 => 289623)
--- trunk/Source/WebCore/platform/graphics/cg/UTIRegistry.h 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebCore/platform/graphics/cg/UTIRegistry.h 2022-02-11 17:01:52 UTC (rev 289623)
@@ -34,7 +34,7 @@
HashSet<String>& additionalSupportedImageTypes();
WEBCORE_EXPORT void setAdditionalSupportedImageTypes(const Vector<String>&);
WEBCORE_EXPORT void setAdditionalSupportedImageTypesForTesting(const String&);
-bool isSupportedImageType(const String&);
+WEBCORE_EXPORT bool isSupportedImageType(const String&);
bool isGIFImageType(StringView);
}
Modified: trunk/Source/WebKit/ChangeLog (289622 => 289623)
--- trunk/Source/WebKit/ChangeLog 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebKit/ChangeLog 2022-02-11 17:01:52 UTC (rev 289623)
@@ -1,3 +1,128 @@
+2022-02-11 Wenson Hsieh <wenson_hs...@apple.com>
+
+ [iOS] Add support for a "markup image" item in the callout bar when selecting a single image
+ https://bugs.webkit.org/show_bug.cgi?id=236415
+ rdar://88714333
+
+ Reviewed by Aditya Keerthi.
+
+ This patch introduces support for a new "Markup Image" callout menu item, which is only shown when the user has
+ selected exactly one image. If the image is suitable for "Markup Image" (that is, the relevant VisionKit API
+ returns a non-null image), then we show a new item in the callout bar which, when activated, replaces the
+ current selection with the image returned by the aforementioned VisionKit API.
+
+ See below for more details.
+
+ * Platform/cocoa/TextRecognitionUtilities.h:
+ * Platform/cocoa/TextRecognitionUtilities.mm:
+ (WebKit::isImageAnalysisMarkupSystemFeatureEnabled):
+
+ Add a new helper method to return whether or not the corresponding system feature flag is enabled.
+
+ * Shared/EditorState.cpp:
+ (WebKit::EditorState::PostLayoutData::encode const):
+ (WebKit::EditorState::PostLayoutData::decode):
+ * Shared/EditorState.h:
+
+ Add an optional `ElementContext` that's populated only if the user's selection spans a single HTMLImageElement.
+
+ * UIProcess/API/ios/WKWebViewIOS.mm:
+ (-[WKWebView didMoveToWindow]):
+ * UIProcess/WebPageProxy.cpp:
+ (WebKit::WebPageProxy::requestImageBitmap):
+
+ Add a new async IPC message that allows the UI process to grab a ShareableBitmap handle (along with the MIME
+ type of the source image) given an ElementContext that identifies an image element.
+
+ * UIProcess/WebPageProxy.h:
+ * UIProcess/ios/WKContentViewInteraction.h:
+
+ Cache the last result from requesting "Image Markup" data from VisionKit in `_imageAnalysisMarkupResults`, and
+ use this information when invoking the new selector, `-performImageAnalysisMarkup:`.
+
+ * UIProcess/ios/WKContentViewInteraction.mm:
+ (WebKit::transcode):
+
+ Add a helper method that takes a CGImageRef, transcodes it to the given uniform type identifier, and finally
+ returns a blob of image data. Used by the two new call sites for Image Markup below, as well as an existing call
+ site for Visual Look Up in `-provideDataForItem:`
+
+ (-[WKContentView targetForAction:withSender:]):
+
+ Add a check for the new Image Markup action selector.
+
+ (-[WKContentView requestRectsToEvadeForSelectionCommandsWithCompletionHandler:]):
+
+ Use this existing UIKit delegate hook to defer callout bar presentation only in the case where a single image
+ element is selected, such that we only show the callout bar in this scenario once VisionKit has determined
+ whether or not the "Image Markup" action is suitable for the data of the selected image.
+
+ The initial purpose of this delegate hook was to allow WebKit to defer callout bar presentation for a short,
+ hard-coded delay to wait for the web page to layout out or add clickable items that might underlap the callout
+ bar; as such, we still perform this logic with a hard-coded delay of 250 ms (including whatever time was taken
+ during image analysis).
+
+ (-[WKContentView updateImageAnalysisMarkupMenuItems:]):
+
+ Append the new menu item only if the system feature flag is enabled, and we've selected an editable image. Note
+ that this intentionally does not consult `_imageAnalysisMarkupResults` to ensure that the state of additional
+ menu items in the shared menu controller is consistent between Markup Image and Quick Note, and updated in the
+ same lifecycle as editor state updates.
+
+ (-[WKContentView canPerformImageAnalysisMarkup]):
+
+ This is consulted when we're actually about to show the callout bar action for "Image Markup" (importantly,
+ after callout bar presentation deferral), and depends on the state of `_imageAnalysisMarkupResults`.
+
+ (-[WKContentView performImageAnalysisMarkup:]):
+
+ Add logic to handle the new action by calling into WebPageProxy to (basically) paste the modified image data
+ after transcoding it to match the image element's original source type. After #236406, this codepath is now
+ available on all Cocoa platforms (as opposed to macOS-only).
+
+ (-[WKContentView doAfterComputingImageAnalysisResultsForMarkup:]):
+
+ Use the new `requestImageBitmap` IPC message to grab a bitmap for the currently selected image, call out to
+ VisionKit to perform image analysis, and then cache the resulting image. After all of this is done, we proceed
+ with showing the callout bar.
+
+ In a followup, we should enforce some (reasonable) upper bound on the amount of time by which the callout bar
+ can be delayed.
+
+ (-[WKContentView _selectionChanged]):
+ (-[WKContentView setUpAdditionalMenuControllerActions]):
+
+ Refactor existing logic for supplying additional callout bar menu items, such that we update items as needed for
+ both Quick Note as well as Image Markup. Note that we need to preserve any existing menu items in the shared
+ UIMenuController, since the WebKit client (e.g. Mail) may have already supplied custom menu items. This
+ currently doees not affect Quick Note since the only internal client that enables Quick Note is Safari, which
+ does not attempt to add any of its own items; however, "Markup Image" needs to be available in Mail on iOS as
+ well, which does use custom menu items.
+
+ (findMenuItemWithAction):
+ (-[WKContentView updateAppHighlightMenuItems:]):
+
+ Make this adjust the given mutable array of UIMenuItems, which are then combined with any markup menu items
+ when setting custom items on the shared menu controller.
+
+ (-[WKContentView provideDataForItem:]):
+
+ Adjust this to use the new `transcode` helper function, declared above.
+
+ (-[WKContentView _setUpImageAnalysis]):
+ (-[WKContentView _tearDownImageAnalysis]):
+ (-[WKContentView setUpAppHighlightMenusIfNeeded]): Deleted.
+ * WebProcess/WebPage/WebPage.cpp:
+ (WebKit::WebPage::requestImageBitmap):
+ * WebProcess/WebPage/WebPage.h:
+ * WebProcess/WebPage/WebPage.messages.in:
+ * WebProcess/WebPage/ios/WebPageIOS.mm:
+ (WebKit::WebPage::getPlatformEditorState const):
+
+ Compute and populate the new `selectedEditableImage` element context; to achieve this, we use `TextIterator` to
+ scan for an image element, and bail immediately if we either find anything that is not an image, or find more
+ than one image.
+
2022-02-11 Kimmo Kinnunen <kkinnu...@apple.com>
Introduce a RemoteMediaSampleProxy to represent captured video frames used in Media Streams and present in GPUP
Modified: trunk/Source/WebKit/Platform/cocoa/TextRecognitionUtilities.h (289622 => 289623)
--- trunk/Source/WebKit/Platform/cocoa/TextRecognitionUtilities.h 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebKit/Platform/cocoa/TextRecognitionUtilities.h 2022-02-11 17:01:52 UTC (rev 289623)
@@ -50,6 +50,7 @@
bool isLiveTextAvailableAndEnabled();
bool textRecognitionEnhancementsSystemFeatureEnabled();
bool imageAnalysisQueueSystemFeatureEnabled();
+bool isImageAnalysisMarkupSystemFeatureEnabled();
WebCore::TextRecognitionResult makeTextRecognitionResult(CocoaImageAnalysis *);
Modified: trunk/Source/WebKit/Platform/cocoa/TextRecognitionUtilities.mm (289622 => 289623)
--- trunk/Source/WebKit/Platform/cocoa/TextRecognitionUtilities.mm 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebKit/Platform/cocoa/TextRecognitionUtilities.mm 2022-02-11 17:01:52 UTC (rev 289623)
@@ -151,8 +151,17 @@
#endif
}
+bool isImageAnalysisMarkupSystemFeatureEnabled()
+{
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+ return true;
+#else
+ return false;
#endif
+}
+#endif
+
bool isLiveTextAvailableAndEnabled()
{
return PAL::isVisionKitCoreFrameworkAvailable();
Modified: trunk/Source/WebKit/Shared/EditorState.cpp (289622 => 289623)
--- trunk/Source/WebKit/Shared/EditorState.cpp 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebKit/Shared/EditorState.cpp 2022-02-11 17:01:52 UTC (rev 289623)
@@ -145,6 +145,7 @@
encoder << caretColor;
encoder << selectionStartIsAtParagraphBoundary;
encoder << selectionEndIsAtParagraphBoundary;
+ encoder << selectedEditableImage;
#endif
#if PLATFORM(MAC)
encoder << selectionBoundingRect;
@@ -228,6 +229,8 @@
return false;
if (!decoder.decode(result.selectionEndIsAtParagraphBoundary))
return false;
+ if (!decoder.decode(result.selectedEditableImage))
+ return false;
#endif
#if PLATFORM(MAC)
if (!decoder.decode(result.selectionBoundingRect))
Modified: trunk/Source/WebKit/Shared/EditorState.h (289622 => 289623)
--- trunk/Source/WebKit/Shared/EditorState.h 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebKit/Shared/EditorState.h 2022-02-11 17:01:52 UTC (rev 289623)
@@ -28,6 +28,7 @@
#include "ArgumentCoders.h"
#include "IdentifierTypes.h"
#include <WebCore/Color.h>
+#include <WebCore/ElementContext.h>
#include <WebCore/FontAttributes.h>
#include <WebCore/IntRect.h>
#include <WebCore/WritingDirection.h>
@@ -124,6 +125,7 @@
bool atStartOfSentence { false };
bool selectionStartIsAtParagraphBoundary { false };
bool selectionEndIsAtParagraphBoundary { false };
+ std::optional<WebCore::ElementContext> selectedEditableImage;
#endif
#if PLATFORM(MAC)
WebCore::IntRect selectionBoundingRect;
Modified: trunk/Source/WebKit/UIProcess/API/ios/WKWebViewIOS.mm (289622 => 289623)
--- trunk/Source/WebKit/UIProcess/API/ios/WKWebViewIOS.mm 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebKit/UIProcess/API/ios/WKWebViewIOS.mm 2022-02-11 17:01:52 UTC (rev 289623)
@@ -1528,9 +1528,7 @@
_page->activityStateDidChange(WebCore::ActivityState::allFlags());
_page->webViewDidMoveToWindow();
-#if ENABLE(APP_HIGHLIGHTS)
- [_contentView setUpAppHighlightMenusIfNeeded];
-#endif
+ [_contentView setUpAdditionalMenuControllerActions];
}
- (void)_setOpaqueInternal:(BOOL)opaque
Modified: trunk/Source/WebKit/UIProcess/WebPageProxy.cpp (289622 => 289623)
--- trunk/Source/WebKit/UIProcess/WebPageProxy.cpp 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebKit/UIProcess/WebPageProxy.cpp 2022-02-11 17:01:52 UTC (rev 289623)
@@ -8834,6 +8834,16 @@
#endif // ENABLE(IMAGE_ANALYSIS)
+void WebPageProxy::requestImageBitmap(const ElementContext& elementContext, CompletionHandler<void(const ShareableBitmap::Handle&, const String&)>&& completion)
+{
+ if (!hasRunningProcess()) {
+ completion({ }, { });
+ return;
+ }
+
+ sendWithAsyncReply(Messages::WebPage::RequestImageBitmap(elementContext), WTFMove(completion));
+}
+
#if ENABLE(ENCRYPTED_MEDIA)
MediaKeySystemPermissionRequestManagerProxy& WebPageProxy::mediaKeySystemPermissionRequestManager()
{
Modified: trunk/Source/WebKit/UIProcess/WebPageProxy.h (289622 => 289623)
--- trunk/Source/WebKit/UIProcess/WebPageProxy.h 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebKit/UIProcess/WebPageProxy.h 2022-02-11 17:01:52 UTC (rev 289623)
@@ -2044,6 +2044,8 @@
WKQuickLookPreviewController *quickLookPreviewController() const { return m_quickLookPreviewController.get(); }
#endif
+ void requestImageBitmap(const WebCore::ElementContext&, CompletionHandler<void(const ShareableBitmap::Handle&, const String& sourceMIMEType)>&&);
+
#if PLATFORM(MAC)
bool isQuarantinedAndNotUserApproved(const String&);
#endif
Modified: trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.h (289622 => 289623)
--- trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.h 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.h 2022-02-11 17:01:52 UTC (rev 289623)
@@ -245,6 +245,12 @@
CGRect textLastRect;
};
+struct ImageAnalysisMarkupData {
+ WebCore::ElementContext element;
+ RetainPtr<CGImageRef> image;
+ String preferredMIMEType;
+};
+
enum class ProceedWithTextSelectionInImage : bool {
No,
Yes
@@ -531,6 +537,9 @@
#endif // USE(QUICK_LOOK)
#endif // ENABLE(IMAGE_ANALYSIS)
uint32_t _fullscreenVideoExtractionRequestIdentifier;
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+ std::optional<WebKit::ImageAnalysisMarkupData> _imageAnalysisMarkupData;
+#endif
}
@end
@@ -773,9 +782,7 @@
- (WebCore::DataOwnerType)_dataOwnerForPasteboard:(WebKit::PasteboardAccessIntent)intent;
#endif
-#if ENABLE(APP_HIGHLIGHTS)
-- (void)setUpAppHighlightMenusIfNeeded;
-#endif
+- (void)setUpAdditionalMenuControllerActions;
#if ENABLE(IMAGE_ANALYSIS)
- (void)_endImageAnalysisGestureDeferral:(WebKit::ShouldPreventGestures)shouldPreventGestures;
Modified: trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm (289622 => 289623)
--- trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm 2022-02-11 17:01:52 UTC (rev 289623)
@@ -127,6 +127,7 @@
#import <WebCore/TextIndicatorWindow.h>
#import <WebCore/TextRecognitionResult.h>
#import <WebCore/TouchAction.h>
+#import <WebCore/UTIRegistry.h>
#import <WebCore/UTIUtilities.h>
#import <WebCore/VisibleSelection.h>
#import <WebCore/WebCoreCALayerExtras.h>
@@ -372,6 +373,20 @@
bool m_shouldPreventTextSelection { false };
};
+static RetainPtr<NSData> transcode(CGImageRef image, CFStringRef typeIdentifier)
+{
+ if (!image)
+ return nil;
+
+ auto data = "" alloc] init]);
+ auto destination = adoptCF(CGImageDestinationCreateWithData((__bridge CFMutableDataRef)data.get(), typeIdentifier, 1, nil));
+ CGImageDestinationAddImage(destination.get(), image, nil);
+ if (!CGImageDestinationFinalize(destination.get()))
+ return nil;
+
+ return data;
+}
+
#endif // ENABLE(IMAGE_ANALYSIS)
} // namespace WebKit
@@ -4072,6 +4087,10 @@
if (action == @selector(createHighlightForNewQuickNoteWithRange:))
return self.shouldAllowAppHighlightCreation && !_page->appHighlightsVisibility() ? self : nil;
#endif
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+ if (action == @selector(performImageAnalysisMarkup:))
+ return self.canPerformImageAnalysisMarkup ? self : nil;
+#endif
return [_webView targetForAction:action withSender:sender];
}
@@ -4597,36 +4616,168 @@
return;
}
- if ([self _shouldSuppressSelectionCommands] || self.webView._editable) {
+ if ([self _shouldSuppressSelectionCommands]) {
completionHandler(@[ ]);
return;
}
- if (_focusedElementInformation.elementType != WebKit::InputType::ContentEditable && _focusedElementInformation.elementType != WebKit::InputType::TextArea) {
- completionHandler(@[ ]);
- return;
- }
+ auto requestRectsToEvadeIfNeeded = [startTime = ApproximateTime::now(), weakSelf = WeakObjCPtr<WKContentView>(self), completion = makeBlockPtr(completionHandler)] {
+ auto strongSelf = weakSelf.get();
+ if (!strongSelf) {
+ completion(@[ ]);
+ return;
+ }
- // Give the page some time to present custom editing UI before attempting to detect and evade it.
- auto delayBeforeShowingCalloutBar = 0.25_s;
- WorkQueue::main().dispatchAfter(delayBeforeShowingCalloutBar, [completion = makeBlockPtr(completionHandler), weakSelf = WeakObjCPtr<WKContentView>(self)] () mutable {
- if (!weakSelf) {
+ if ([strongSelf webView]._editable) {
completion(@[ ]);
return;
}
- auto strongSelf = weakSelf.get();
- if (!strongSelf->_page) {
+ auto focusedElementType = strongSelf->_focusedElementInformation.elementType;
+ if (focusedElementType != WebKit::InputType::ContentEditable && focusedElementType != WebKit::InputType::TextArea) {
completion(@[ ]);
return;
}
- strongSelf->_page->requestEvasionRectsAboveSelection([completion = WTFMove(completion)] (auto& rects) {
- completion(createNSArray(rects).get());
+ // Give the page some time to present custom editing UI before attempting to detect and evade it.
+ auto delayBeforeShowingCalloutBar = std::max(0_s, 0.25_s - (ApproximateTime::now() - startTime));
+ WorkQueue::main().dispatchAfter(delayBeforeShowingCalloutBar, [completion, weakSelf] () mutable {
+ auto strongSelf = weakSelf.get();
+ if (!strongSelf) {
+ completion(@[ ]);
+ return;
+ }
+
+ if (!strongSelf->_page) {
+ completion(@[ ]);
+ return;
+ }
+
+ strongSelf->_page->requestEvasionRectsAboveSelection([completion = WTFMove(completion)] (auto& rects) {
+ completion(createNSArray(rects).get());
+ });
});
+ };
+
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+ [self doAfterComputingImageAnalysisResultsForMarkup:WTFMove(requestRectsToEvadeIfNeeded)];
+#else
+ requestRectsToEvadeIfNeeded();
+#endif
+}
+
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+
+- (void)updateImageAnalysisMarkupMenuItems:(NSMutableArray<UIMenuItem *> *)updatedItems
+{
+ auto currentItem = findMenuItemWithAction(updatedItems, @selector(performImageAnalysisMarkup:));
+ auto& editorState = _page->editorState();
+ if (!WebKit::isImageAnalysisMarkupSystemFeatureEnabled() || !self.window || editorState.isMissingPostLayoutData || !editorState.postLayoutData().selectedEditableImage) {
+ if (currentItem)
+ [updatedItems removeObject:currentItem];
+ } else if (!currentItem)
+ [updatedItems addObject:adoptNS([[UIMenuItem alloc] initWithTitle:WEB_UI_STRING("Markup Image", "Image analysis markup menu item") action:@selector(performImageAnalysisMarkup:)]).get()];
+}
+
+- (BOOL)canPerformImageAnalysisMarkup
+{
+ if (!WebKit::isImageAnalysisMarkupSystemFeatureEnabled())
+ return NO;
+
+ if (!_imageAnalysisMarkupData)
+ return NO;
+
+ auto [elementContext, image, preferredMIMEType] = *_imageAnalysisMarkupData;
+ return !_page->editorState().isMissingPostLayoutData && elementContext == _page->editorState().postLayoutData().selectedEditableImage;
+}
+
+- (void)performImageAnalysisMarkup:(id)sender
+{
+ if (!self.canPerformImageAnalysisMarkup)
+ return;
+
+ auto [elementContext, image, preferredMIMEType] = *_imageAnalysisMarkupData;
+ auto targetType = RetainPtr { UTTypeTIFF.identifier };
+ if (!preferredMIMEType.isEmpty()) {
+ NSString *preferredTypeIdentifier = [UTType typeWithMIMEType:preferredMIMEType conformingToType:UTTypeImage].identifier;
+ if (WebCore::isSupportedImageType(preferredTypeIdentifier))
+ targetType = preferredTypeIdentifier;
+ }
+
+ if (auto data = "" (__bridge CFStringRef)targetType.get()); [data length])
+ _page->replaceSelectionWithPasteboardData({ String { targetType.get() } }, { static_cast<const uint8_t*>([data bytes]), [data length] });
+}
+
+- (void)doAfterComputingImageAnalysisResultsForMarkup:(CompletionHandler<void()>&&)completion
+{
+ if (_page->editorState().isMissingPostLayoutData) {
+ completion();
+ return;
+ }
+
+ auto elementToAnalyze = _page->editorState().postLayoutData().selectedEditableImage;
+ if (_imageAnalysisMarkupData && _imageAnalysisMarkupData->element == elementToAnalyze) {
+ completion();
+ return;
+ }
+
+ _imageAnalysisMarkupData = std::nullopt;
+
+ if (!elementToAnalyze) {
+ completion();
+ return;
+ }
+
+ _page->requestImageBitmap(*elementToAnalyze, [context = *elementToAnalyze, completion = WTFMove(completion), weakSelf = WeakObjCPtr<WKContentView>(self)](auto& imageData, auto& sourceMIMEType) mutable {
+ auto strongSelf = weakSelf.get();
+ if (!strongSelf) {
+ completion();
+ return;
+ }
+
+ auto imageBitmap = WebKit::ShareableBitmap::create(imageData);
+ if (!imageBitmap) {
+ completion();
+ return;
+ }
+
+ auto cgImage = imageBitmap->makeCGImage();
+ if (!cgImage) {
+ completion();
+ return;
+ }
+
+ // FIXME: Check to see if we can avoid this extra transcoding step once VisionKit starts using `mediaanalysisd` for this.
+ auto tiffData = WebKit::transcode(cgImage.get(), (__bridge CFStringRef)UTTypeTIFF.identifier);
+ if (![tiffData length]) {
+ completion();
+ return;
+ }
+
+ RetainPtr transcodedImage = [UIImage imageWithData:tiffData.get()];
+ if (!transcodedImage) {
+ completion();
+ return;
+ }
+
+ WebKit::requestImageAnalysisMarkup([transcodedImage CGImage], [sourceMIMEType, context, completion = WTFMove(completion), weakSelf](CGImageRef result) mutable {
+ auto strongSelf = weakSelf.get();
+ if (!strongSelf) {
+ completion();
+ return;
+ }
+
+ if (result)
+ strongSelf->_imageAnalysisMarkupData = { { context, { result }, sourceMIMEType } };
+ else
+ strongSelf->_imageAnalysisMarkupData = std::nullopt;
+ completion();
+ });
});
}
+#endif // ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+
- (void)selectPositionAtPoint:(CGPoint)point completionHandler:(void (^)(void))completionHandler
{
_autocorrectionContextNeedsUpdate = YES;
@@ -7313,10 +7464,8 @@
- (void)_selectionChanged
{
_autocorrectionContextNeedsUpdate = YES;
-#if ENABLE(APP_HIGHLIGHTS)
- [self setUpAppHighlightMenusIfNeeded];
-#endif
+ [self setUpAdditionalMenuControllerActions];
[self _updateSelectionAssistantSuppressionState];
_cachedSelectedTextRange = nil;
@@ -9602,21 +9751,51 @@
}
#endif
+- (void)setUpAdditionalMenuControllerActions
+{
+ auto updatedItems = adoptNS(UIMenuController.sharedMenuController.menuItems.mutableCopy ?: [NSMutableArray<UIMenuItem *> new]);
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+ [self updateImageAnalysisMarkupMenuItems:updatedItems.get()];
+#endif
#if ENABLE(APP_HIGHLIGHTS)
+ [self updateAppHighlightMenuItems:updatedItems.get()];
+#endif
+ UIMenuController.sharedMenuController.menuItems = updatedItems.get();
+}
-- (void)setUpAppHighlightMenusIfNeeded
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) || ENABLE(APP_HIGHLIGHTS)
+
+static UIMenuItem *findMenuItemWithAction(NSArray<UIMenuItem *> *items, SEL action)
{
- if (!_page->preferences().appHighlightsEnabled() || !self.window || !_page->editorState().selectionIsRange)
+ for (UIMenuItem *item in items) {
+ if (item.action == action)
+ return item;
+ }
+ return nil;
+}
+
+#endif // ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS) || ENABLE(APP_HIGHLIGHTS)
+
+#if ENABLE(APP_HIGHLIGHTS)
+
+- (void)updateAppHighlightMenuItems:(NSMutableArray<UIMenuItem *> *)updatedItems
+{
+ auto currentQuickNoteItem = findMenuItemWithAction(updatedItems, @selector(createHighlightForCurrentQuickNoteWithRange:));
+ auto newQuickNoteItem = findMenuItemWithAction(updatedItems, @selector(createHighlightForNewQuickNoteWithRange:));
+
+ if (!_page->preferences().appHighlightsEnabled() || !self.window || !_page->editorState().selectionIsRange) {
+ if (currentQuickNoteItem)
+ [updatedItems removeObject:currentQuickNoteItem];
+ if (newQuickNoteItem)
+ [updatedItems removeObject:newQuickNoteItem];
return;
-
- for (UIMenuItem *menuItem in [[UIMenuController sharedMenuController] menuItems]) {
- if ([menuItem action] == @selector(createHighlightForCurrentQuickNoteWithRange:) || [menuItem action] == @selector(createHighlightForNewQuickNoteWithRange:))
- return;
}
-
- auto addHighlightCurrentQuickNoteItem = adoptNS([[UIMenuItem alloc] initWithTitle:WebCore::contextMenuItemTagAddHighlightToCurrentQuickNote() action:@selector(createHighlightForCurrentQuickNoteWithRange:)]);
- auto addHighlightNewQuickNoteItem = adoptNS([[UIMenuItem alloc] initWithTitle:WebCore::contextMenuItemTagAddHighlightToNewQuickNote() action:@selector(createHighlightForNewQuickNoteWithRange:)]);
- [[UIMenuController sharedMenuController] setMenuItems:@[ addHighlightCurrentQuickNoteItem.get(), addHighlightNewQuickNoteItem.get() ]];
+
+ if (!currentQuickNoteItem)
+ [updatedItems addObject:adoptNS([[UIMenuItem alloc] initWithTitle:WebCore::contextMenuItemTagAddHighlightToCurrentQuickNote() action:@selector(createHighlightForCurrentQuickNoteWithRange:)]).get()];
+
+ if (!newQuickNoteItem)
+ [updatedItems addObject:adoptNS([[UIMenuItem alloc] initWithTitle:WebCore::contextMenuItemTagAddHighlightToNewQuickNote() action:@selector(createHighlightForNewQuickNoteWithRange:)]).get()];
}
- (void)createHighlightForCurrentQuickNoteWithRange:(id)sender
@@ -10315,14 +10494,7 @@
- (NSData *)provideDataForItem:(QLItem *)item
{
ASSERT(_visualSearchPreviewImage);
-
- auto data = "" 0));
- auto destination = adoptCF(CGImageDestinationCreateWithData(data.get(), (__bridge CFStringRef)UTTypeTIFF.identifier, 1, NULL));
- CGImageDestinationAddImage(destination.get(), [_visualSearchPreviewImage CGImage], nil);
- if (!CGImageDestinationFinalize(destination.get()))
- return nil;
-
- return data.bridgingAutorelease();
+ return WebKit::transcode([_visualSearchPreviewImage CGImage], (__bridge CFStringRef)UTTypeTIFF.identifier).autorelease();
}
#pragma mark - WKActionSheetAssistantDelegate
@@ -10385,6 +10557,9 @@
#if USE(UICONTEXTMENU) && ENABLE(IMAGE_ANALYSIS_FOR_MACHINE_READABLE_CODES)
_contextMenuForMachineReadableCode.clear();
#endif // USE(UICONTEXTMENU) && ENABLE(IMAGE_ANALYSIS_FOR_MACHINE_READABLE_CODES)
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+ _imageAnalysisMarkupData = std::nullopt;
+#endif
}
- (void)_tearDownImageAnalysis
@@ -10410,6 +10585,9 @@
_contextMenuForMachineReadableCode.clear();
#endif // USE(UICONTEXTMENU) && ENABLE(IMAGE_ANALYSIS_FOR_MACHINE_READABLE_CODES)
[self _invokeAllActionsToPerformAfterPendingImageAnalysis:WebKit::ProceedWithTextSelectionInImage::No];
+#if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
+ _imageAnalysisMarkupData = std::nullopt;
+#endif
}
- (void)_cancelImageAnalysis
Modified: trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp (289622 => 289623)
--- trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp 2022-02-11 17:01:52 UTC (rev 289623)
@@ -7743,6 +7743,42 @@
#endif // ENABLE(IMAGE_ANALYSIS)
+void WebPage::requestImageBitmap(const ElementContext& context, CompletionHandler<void(const ShareableBitmap::Handle&, const String& sourceMIMEType)>&& completion)
+{
+ RefPtr element = elementForContext(context);
+ if (!element) {
+ completion({ }, { });
+ return;
+ }
+
+ auto* renderer = dynamicDowncast<RenderImage>(element->renderer());
+ if (!renderer) {
+ completion({ }, { });
+ return;
+ }
+
+ auto bitmap = createShareableBitmap(*renderer);
+ if (!bitmap) {
+ completion({ }, { });
+ return;
+ }
+
+ ShareableBitmap::Handle handle;
+ bitmap->createHandle(handle);
+ if (handle.isNull()) {
+ completion({ }, { });
+ return;
+ }
+
+ String mimeType;
+ if (auto* cachedImage = renderer->cachedImage()) {
+ if (auto* image = cachedImage->image())
+ mimeType = image->mimeType();
+ }
+ ASSERT(!mimeType.isEmpty());
+ completion(handle, mimeType);
+}
+
#if ENABLE(MEDIA_CONTROLS_CONTEXT_MENUS) && USE(UICONTEXTMENU)
void WebPage::showMediaControlsContextMenu(FloatRect&& targetFrame, Vector<MediaControlsContextMenuItem>&& items, CompletionHandler<void(MediaControlsContextMenuItem::ID)>&& completionHandler)
{
Modified: trunk/Source/WebKit/WebProcess/WebPage/WebPage.h (289622 => 289623)
--- trunk/Source/WebKit/WebProcess/WebPage/WebPage.h 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebKit/WebProcess/WebPage/WebPage.h 2022-02-11 17:01:52 UTC (rev 289623)
@@ -1440,6 +1440,8 @@
void startImageAnalysis(const String& identifier);
#endif
+ void requestImageBitmap(const WebCore::ElementContext&, CompletionHandler<void(const ShareableBitmap::Handle&, const String& sourceMIMEType)>&&);
+
#if HAVE(TRANSLATION_UI_SERVICES) && ENABLE(CONTEXT_MENUS)
void handleContextMenuTranslation(const WebCore::TranslationContextMenuInfo&);
#endif
Modified: trunk/Source/WebKit/WebProcess/WebPage/WebPage.messages.in (289622 => 289623)
--- trunk/Source/WebKit/WebProcess/WebPage/WebPage.messages.in 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebKit/WebProcess/WebPage/WebPage.messages.in 2022-02-11 17:01:52 UTC (rev 289623)
@@ -128,6 +128,8 @@
ClearServiceWorkerEntitlementOverride() -> () Async
#endif
+ RequestImageBitmap(struct WebCore::ElementContext elementContext) -> (WebKit::ShareableBitmap::Handle image, String sourceMIMEType) Async
+
SetControlledByAutomation(bool controlled)
ConnectInspector(String targetId, Inspector::FrontendChannel::ConnectionType connectionType)
Modified: trunk/Source/WebKit/WebProcess/WebPage/ios/WebPageIOS.mm (289622 => 289623)
--- trunk/Source/WebKit/WebProcess/WebPage/ios/WebPageIOS.mm 2022-02-11 16:08:45 UTC (rev 289622)
+++ trunk/Source/WebKit/WebProcess/WebPage/ios/WebPageIOS.mm 2022-02-11 17:01:52 UTC (rev 289623)
@@ -320,6 +320,25 @@
postLayoutData.selectedTextLength = selectedText.length();
const int maxSelectedTextLength = 200;
postLayoutData.wordAtSelection = selectedText.left(maxSelectedTextLength);
+ auto findSelectedEditableImageElement = [&] {
+ RefPtr<HTMLImageElement> foundImage;
+ if (!result.isContentEditable)
+ return foundImage;
+
+ for (TextIterator iterator { *selectedRange, { } }; !iterator.atEnd(); iterator.advance()) {
+ if (foundImage) {
+ foundImage = nullptr;
+ break;
+ }
+ foundImage = dynamicDowncast<HTMLImageElement>(iterator.node());
+ if (!foundImage)
+ break;
+ }
+ return foundImage;
+ };
+
+ if (auto imageElement = findSelectedEditableImageElement())
+ postLayoutData.selectedEditableImage = contextForElement(*imageElement);
}
// FIXME: We should disallow replace when the string contains only CJ characters.
postLayoutData.isReplaceAllowed = result.isContentEditable && !result.isInPasswordField && !selectedText.isAllSpecialCharacters<isHTMLSpace>();
