- Revision
- 287296
- Author
- wenson_hs...@apple.com
- Date
- 2021-12-20 19:09:46 -0800 (Mon, 20 Dec 2021)
Log Message
Add ModalContainerControlClassifier and use it to implement classifyModalContainerControls()
https://bugs.webkit.org/show_bug.cgi?id=234322
Reviewed by Devin Rousso.
Introduce and implement ModalContainerControlClassifier. This singleton uses the NaturalLanguage and CoreML
frameworks on Cocoa to classify strings as one of four modal container control types. See below for more
details.
* SourcesCocoa.txt:
* UIProcess/Cocoa/ModalContainerControlClassifier.h: Added.
* UIProcess/Cocoa/ModalContainerControlClassifier.mm: Added.
(-[WKModalContainerClassifierBatch initWithRawInputs:]):
(-[WKModalContainerClassifierBatch count]):
(-[WKModalContainerClassifierBatch featuresAtIndex:]):
(-[WKModalContainerClassifierInput initWithTokenizer:rawInput:]):
(-[WKModalContainerClassifierInput featureNames]):
(-[WKModalContainerClassifierInput featureValueForName:]):
Add Objective-C objects that implement the MLBatchProvider and MLFeatureProvider protocols, respectively.
WKModalContainerClassifierBatch is essentially a wrapper around a list of WKModalContainerClassifierInput; each
WKModalContainerClassifierInput is initialized with a raw string, and uses NLTokenizer to filter out non-word
characters and tokenize the raw input into a single space-separated, lower case string (referred to as the
"canonical" input format).
(WebKit::ModalContainerControlClassifier::ModalContainerControlClassifier):
(WebKit::ModalContainerControlClassifier::sharedClassifier):
Return the singleton instance (this must be accessed on the main thread).
(WebKit::computePredictions):
Static helper method that takes a list of strings and an MLModel, and classifies each string using the model,
and the Objective-C helper classes above.
(WebKit::ModalContainerControlClassifier::classify):
This method exposes the primary functionality of the classifier, which is to take a list of raw strings
representing text in clickable controls, and asynchronously return a list of class labels representing the
predicted control type for each of the strings. Note that this method needs to be invoked on the main thread
(and will also invoke the completion handler on the main thread), but the process of loading the MLModel and
using it to predict input strings is done in a work queue ("com.apple.WebKit.ModalContainerControlClassifier").
(WebKit::ModalContainerControlClassifier::loadModelIfNeeded):
Load the MLModel from a predetermined bundle resource name; returns immediately if the model has
already been created. While this happens synchronously, this is always invoked on a background queue and never
blocks the main thread.
* UIProcess/Cocoa/WebPageProxyCocoa.mm:
(WebKit::WebPageProxy::classifyModalContainerControls):
* UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::classifyModalContainerControls):
* WebKit.xcodeproj/project.pbxproj:
Modified Paths
Added Paths
Diff
Modified: trunk/Source/WebKit/ChangeLog (287295 => 287296)
--- trunk/Source/WebKit/ChangeLog 2021-12-21 02:30:57 UTC (rev 287295)
+++ trunk/Source/WebKit/ChangeLog 2021-12-21 03:09:46 UTC (rev 287296)
@@ -1,3 +1,60 @@
+2021-12-20 Wenson Hsieh <wenson_hs...@apple.com>
+
+ Add ModalContainerControlClassifier and use it to implement classifyModalContainerControls()
+ https://bugs.webkit.org/show_bug.cgi?id=234322
+
+ Reviewed by Devin Rousso.
+
+ Introduce and implement ModalContainerControlClassifier. This singleton uses the NaturalLanguage and CoreML
+ frameworks on Cocoa to classify strings as one of four modal container control types. See below for more
+ details.
+
+ * SourcesCocoa.txt:
+ * UIProcess/Cocoa/ModalContainerControlClassifier.h: Added.
+ * UIProcess/Cocoa/ModalContainerControlClassifier.mm: Added.
+ (-[WKModalContainerClassifierBatch initWithRawInputs:]):
+ (-[WKModalContainerClassifierBatch count]):
+ (-[WKModalContainerClassifierBatch featuresAtIndex:]):
+ (-[WKModalContainerClassifierInput initWithTokenizer:rawInput:]):
+ (-[WKModalContainerClassifierInput featureNames]):
+ (-[WKModalContainerClassifierInput featureValueForName:]):
+
+ Add Objective-C objects that implement the MLBatchProvider and MLFeatureProvider protocols, respectively.
+ WKModalContainerClassifierBatch is essentially a wrapper around a list of WKModalContainerClassifierInput; each
+ WKModalContainerClassifierInput is initialized with a raw string, and uses NLTokenizer to filter out non-word
+ characters and tokenize the raw input into a single space-separated, lower case string (referred to as the
+ "canonical" input format).
+
+ (WebKit::ModalContainerControlClassifier::ModalContainerControlClassifier):
+ (WebKit::ModalContainerControlClassifier::sharedClassifier):
+
+ Return the singleton instance (this must be accessed on the main thread).
+
+ (WebKit::computePredictions):
+
+ Static helper method that takes a list of strings and an MLModel, and classifies each string using the model,
+ and the Objective-C helper classes above.
+
+ (WebKit::ModalContainerControlClassifier::classify):
+
+ This method exposes the primary functionality of the classifier, which is to take a list of raw strings
+ representing text in clickable controls, and asynchronously return a list of class labels representing the
+ predicted control type for each of the strings. Note that this method needs to be invoked on the main thread
+ (and will also invoke the completion handler on the main thread), but the process of loading the MLModel and
+ using it to predict input strings is done in a work queue ("com.apple.WebKit.ModalContainerControlClassifier").
+
+ (WebKit::ModalContainerControlClassifier::loadModelIfNeeded):
+
+ Load the MLModel from a predetermined bundle resource name; returns immediately if the model has
+ already been created. While this happens synchronously, this is always invoked on a background queue and never
+ blocks the main thread.
+
+ * UIProcess/Cocoa/WebPageProxyCocoa.mm:
+ (WebKit::WebPageProxy::classifyModalContainerControls):
+ * UIProcess/WebPageProxy.cpp:
+ (WebKit::WebPageProxy::classifyModalContainerControls):
+ * WebKit.xcodeproj/project.pbxproj:
+
2021-12-20 Alexey Shvayka <ashva...@apple.com>
[WebIDL] convertVariadicArguments() should return a FixedVector
Modified: trunk/Source/WebKit/SourcesCocoa.txt (287295 => 287296)
--- trunk/Source/WebKit/SourcesCocoa.txt 2021-12-21 02:30:57 UTC (rev 287295)
+++ trunk/Source/WebKit/SourcesCocoa.txt 2021-12-21 03:09:46 UTC (rev 287296)
@@ -403,6 +403,7 @@
UIProcess/Cocoa/LegacyCustomProtocolManagerClient.mm
UIProcess/Cocoa/MediaUtilities.mm
UIProcess/Cocoa/MediaPermissionUtilities.mm
+UIProcess/Cocoa/ModalContainerControlClassifier.mm
UIProcess/Cocoa/ModelElementControllerCocoa.mm
UIProcess/Cocoa/NavigationState.mm
UIProcess/Cocoa/PageClientImplCocoa.mm
Added: trunk/Source/WebKit/UIProcess/Cocoa/ModalContainerControlClassifier.h (0 => 287296)
--- trunk/Source/WebKit/UIProcess/Cocoa/ModalContainerControlClassifier.h (rev 0)
+++ trunk/Source/WebKit/UIProcess/Cocoa/ModalContainerControlClassifier.h 2021-12-21 03:09:46 UTC (rev 287296)
@@ -0,0 +1,55 @@
+/*
+ * Copyright (C) 2021 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#import <wtf/Forward.h>
+
+@class MLModel;
+
+namespace WebCore {
+enum class ModalContainerControlType : uint8_t;
+}
+
+namespace WebKit {
+
+class ModalContainerControlClassifier {
+ WTF_MAKE_FAST_ALLOCATED;
+public:
+ static ModalContainerControlClassifier& sharedClassifier();
+
+ void classify(Vector<String>&& text, CompletionHandler<void(Vector<WebCore::ModalContainerControlType>&&)>&&);
+
+private:
+ friend std::unique_ptr<ModalContainerControlClassifier> std::make_unique<ModalContainerControlClassifier>();
+
+ ModalContainerControlClassifier();
+ void loadModelIfNeeded();
+
+ Ref<WorkQueue> m_queue;
+ RetainPtr<MLModel> m_model;
+};
+
+} // namespace WebKit
Added: trunk/Source/WebKit/UIProcess/Cocoa/ModalContainerControlClassifier.mm (0 => 287296)
--- trunk/Source/WebKit/UIProcess/Cocoa/ModalContainerControlClassifier.mm (rev 0)
+++ trunk/Source/WebKit/UIProcess/Cocoa/ModalContainerControlClassifier.mm 2021-12-21 03:09:46 UTC (rev 287296)
@@ -0,0 +1,194 @@
+/*
+ * Copyright (C) 2021 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#import "config.h"
+#import "ModalContainerControlClassifier.h"
+
+#import <WebCore/ModalContainerControlType.h>
+#import <pal/cocoa/CoreMLSoftLink.h>
+#import <pal/cocoa/NaturalLanguageSoftLink.h>
+
+static NSString *const classifierInputFeatureKey = @"text";
+static NSString *const classifierOutputFeatureKey = @"label";
+
+@interface WKModalContainerClassifierBatch : NSObject<MLBatchProvider>
+- (instancetype)initWithRawInputs:(Vector<String>&&)inputStrings;
+@end
+
+@interface WKModalContainerClassifierInput : NSObject<MLFeatureProvider>
+- (instancetype)initWithTokenizer:(NLTokenizer *)tokenizer rawInput:(NSString *)rawInput;
+@end
+
+@implementation WKModalContainerClassifierBatch {
+ Vector<RetainPtr<WKModalContainerClassifierInput>> _inputs;
+}
+
+- (instancetype)initWithRawInputs:(Vector<String>&&)inputStrings
+{
+ if (!(self = [super init]))
+ return nil;
+
+ auto tokenizer = adoptNS([PAL::allocNLTokenizerInstance() initWithUnit:NLTokenUnitWord]);
+ _inputs = inputStrings.map([&](auto& rawInput) {
+ return adoptNS([[WKModalContainerClassifierInput alloc] initWithTokenizer:tokenizer.get() rawInput:rawInput]);
+ });
+ return self;
+}
+
+- (NSInteger)count
+{
+ return _inputs.size();
+}
+
+- (id <MLFeatureProvider>)featuresAtIndex:(NSInteger)index
+{
+ if (index >= static_cast<NSInteger>(_inputs.size())) {
+ ASSERT_NOT_REACHED();
+ return nil;
+ }
+
+ return _inputs[index].get();
+}
+
+@end
+
+@implementation WKModalContainerClassifierInput {
+ RetainPtr<NSString> _canonicalInput;
+}
+
+- (instancetype)initWithTokenizer:(NLTokenizer *)tokenizer rawInput:(NSString *)rawInput
+{
+ if (!(self = [super init]))
+ return nil;
+
+ [tokenizer setString:rawInput];
+
+ auto tokens = adoptNS([NSMutableArray<NSString *> new]);
+ [tokenizer enumerateTokensInRange:NSMakeRange(0, rawInput.length) usingBlock:[&](NSRange range, NLTokenizerAttributes attributes, BOOL *stop) {
+ if (attributes)
+ return;
+
+ NSString *lowercaseToken = [rawInput substringWithRange:range].lowercaseString;
+ if (!lowercaseToken.length)
+ return;
+
+ [tokens addObject:lowercaseToken];
+ }];
+
+ _canonicalInput = [tokens componentsJoinedByString:@" "];
+ return self;
+}
+
+- (NSSet<NSString *> *)featureNames
+{
+ return [NSSet<NSString *> setWithObject:classifierInputFeatureKey];
+}
+
+- (MLFeatureValue *)featureValueForName:(NSString *)featureName
+{
+ return [featureName isEqualToString:classifierInputFeatureKey] ? [PAL::getMLFeatureValueClass() featureValueWithString:_canonicalInput.get()] : nil;
+}
+
+@end
+
+namespace WebKit {
+using namespace WebCore;
+
+ModalContainerControlClassifier::ModalContainerControlClassifier()
+ : m_queue(WorkQueue::create("com.apple.WebKit.ModalContainerControlClassifier"))
+{
+ ASSERT(RunLoop::isMain());
+}
+
+ModalContainerControlClassifier& ModalContainerControlClassifier::sharedClassifier()
+{
+ static NeverDestroyed<std::unique_ptr<ModalContainerControlClassifier>> classifier;
+ if (!classifier.get())
+ classifier.get() = makeUnique<ModalContainerControlClassifier>();
+ return *classifier.get();
+}
+
+static Vector<WebCore::ModalContainerControlType> computePredictions(MLModel *model, Vector<String>&& texts)
+{
+ ASSERT(!RunLoop::isMain());
+ if (!model)
+ return { };
+
+ auto batch = adoptNS([[WKModalContainerClassifierBatch alloc] initWithRawInputs:WTFMove(texts)]);
+ NSError *predictionError = nil;
+ auto resultProvider = [model predictionsFromBatch:batch.get() error:&predictionError];
+ if (predictionError || resultProvider.count < [batch count]) {
+ // FIXME: We may want to log the error here.
+ return { };
+ }
+
+ Vector<ModalContainerControlType> results;
+ results.reserveInitialCapacity(resultProvider.count);
+
+ for (NSInteger index = 0; index < resultProvider.count; ++index) {
+ auto result = [resultProvider featuresAtIndex:index];
+ auto stringResult = [result featureValueForName:classifierOutputFeatureKey].stringValue;
+ if ([stringResult isEqualToString:@"neutral"])
+ results.uncheckedAppend(ModalContainerControlType::Neutral);
+ else if ([stringResult isEqualToString:@"positive"])
+ results.uncheckedAppend(ModalContainerControlType::Positive);
+ else if ([stringResult isEqualToString:@"negative"])
+ results.uncheckedAppend(ModalContainerControlType::Negative);
+ else
+ results.uncheckedAppend(ModalContainerControlType::Other);
+ }
+
+ return results;
+}
+
+void ModalContainerControlClassifier::classify(Vector<String>&& texts, CompletionHandler<void(Vector<ModalContainerControlType>&&)>&& completion)
+{
+ ASSERT(RunLoop::isMain());
+ m_queue->dispatch([this, texts = texts.isolatedCopy(), completion = WTFMove(completion)]() mutable {
+ loadModelIfNeeded();
+ RunLoop::main().dispatch([completion = WTFMove(completion), predictions = computePredictions(m_model.get(), WTFMove(texts))]() mutable {
+ completion(WTFMove(predictions));
+ });
+ });
+}
+
+void ModalContainerControlClassifier::loadModelIfNeeded()
+{
+ ASSERT(!RunLoop::isMain());
+ if (m_model)
+ return;
+
+ auto bundle = [NSBundle bundleWithIdentifier:@"com.apple.WebKit"];
+ auto compiledModelURL = [bundle URLForResource:@"ModalContainerControls" withExtension:@"mlmodelc"];
+ if (!compiledModelURL)
+ return;
+
+ auto configuration = adoptNS([PAL::allocMLModelConfigurationInstance() init]);
+ [configuration setComputeUnits:MLComputeUnitsCPUOnly];
+ NSError *loadingError = nil;
+ m_model = [PAL::getMLModelClass() modelWithContentsOfURL:compiledModelURL configuration:configuration.get() error:&loadingError];
+}
+
+} // namespace WebKit
Modified: trunk/Source/WebKit/UIProcess/Cocoa/WebPageProxyCocoa.mm (287295 => 287296)
--- trunk/Source/WebKit/UIProcess/Cocoa/WebPageProxyCocoa.mm 2021-12-21 02:30:57 UTC (rev 287295)
+++ trunk/Source/WebKit/UIProcess/Cocoa/WebPageProxyCocoa.mm 2021-12-21 03:09:46 UTC (rev 287296)
@@ -35,6 +35,7 @@
#import "DataDetectionResult.h"
#import "InsertTextOptions.h"
#import "LoadParameters.h"
+#import "ModalContainerControlClassifier.h"
#import "PageClient.h"
#import "QuarantineSPI.h"
#import "QuickLookThumbnailLoader.h"
@@ -797,6 +798,11 @@
}
#endif
+void WebPageProxy::classifyModalContainerControls(Vector<String>&& texts, CompletionHandler<void(Vector<ModalContainerControlType>&&)>&& completion)
+{
+ ModalContainerControlClassifier::sharedClassifier().classify(WTFMove(texts), WTFMove(completion));
+}
+
} // namespace WebKit
#undef MESSAGE_CHECK_COMPLETION
Modified: trunk/Source/WebKit/UIProcess/WebPageProxy.cpp (287295 => 287296)
--- trunk/Source/WebKit/UIProcess/WebPageProxy.cpp 2021-12-21 02:30:57 UTC (rev 287295)
+++ trunk/Source/WebKit/UIProcess/WebPageProxy.cpp 2021-12-21 03:09:46 UTC (rev 287296)
@@ -11002,6 +11002,11 @@
{
return { };
}
+
+void WebPageProxy::classifyModalContainerControls(Vector<String>&&, CompletionHandler<void(Vector<ModalContainerControlType>&&)>&& completion)
+{
+ completion({ });
+}
#endif
#if ENABLE(MEDIA_SESSION_COORDINATOR)
@@ -11052,13 +11057,6 @@
m_uiClient->requestCookieConsent(WTFMove(completion));
}
-void WebPageProxy::classifyModalContainerControls(Vector<String>&& texts, CompletionHandler<void(Vector<ModalContainerControlType>&&)>&& completion)
-{
- // FIXME: Not implemented yet.
- UNUSED_PARAM(texts);
- completion({ });
-}
-
} // namespace WebKit
#undef WEBPAGEPROXY_RELEASE_LOG
Modified: trunk/Source/WebKit/WebKit.xcodeproj/project.pbxproj (287295 => 287296)
--- trunk/Source/WebKit/WebKit.xcodeproj/project.pbxproj 2021-12-21 02:30:57 UTC (rev 287295)
+++ trunk/Source/WebKit/WebKit.xcodeproj/project.pbxproj 2021-12-21 03:09:46 UTC (rev 287296)
@@ -6425,6 +6425,8 @@
F48BB8DF26F96392001C1C40 /* RemoteDisplayListRecorder.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = RemoteDisplayListRecorder.h; sourceTree = "<group>"; };
F48BB8E026F96392001C1C40 /* RemoteDisplayListRecorder.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = RemoteDisplayListRecorder.cpp; sourceTree = "<group>"; };
F48D2A8421583A0200C6752B /* AppKitSPI.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = AppKitSPI.h; sourceTree = "<group>"; };
+ F493288527690FCF003E20F8 /* ModalContainerControlClassifier.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = ModalContainerControlClassifier.h; sourceTree = "<group>"; };
+ F493288627690FCF003E20F8 /* ModalContainerControlClassifier.mm */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.objcpp; path = ModalContainerControlClassifier.mm; sourceTree = "<group>"; };
F496A42F1F58A272004C1757 /* DragDropInteractionState.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; name = DragDropInteractionState.h; path = ios/DragDropInteractionState.h; sourceTree = "<group>"; };
F496A4301F58A272004C1757 /* DragDropInteractionState.mm */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.objcpp; name = DragDropInteractionState.mm; path = ios/DragDropInteractionState.mm; sourceTree = "<group>"; };
F4975CF12624B80A003C626E /* WKQuickLookPreviewController.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = WKQuickLookPreviewController.h; sourceTree = "<group>"; };
@@ -7408,6 +7410,8 @@
9342588F2555DCA50059EEDD /* MediaPermissionUtilities.mm */,
411286EF21C8A90C003A8550 /* MediaUtilities.h */,
411286F021C8A90D003A8550 /* MediaUtilities.mm */,
+ F493288527690FCF003E20F8 /* ModalContainerControlClassifier.h */,
+ F493288627690FCF003E20F8 /* ModalContainerControlClassifier.mm */,
7137BA7D25F153E900914EE3 /* ModelElementControllerCocoa.mm */,
1ABC3DF41899E437004F0626 /* NavigationState.h */,
1ABC3DF31899E437004F0626 /* NavigationState.mm */,
