- Revision
- 226085
- Author
- wenson_hs...@apple.com
- Date
- 2017-12-18 16:19:17 -0800 (Mon, 18 Dec 2017)
Log Message
[Attachment Support] Insert images as inline attachments when pasting and dropping
https://bugs.webkit.org/show_bug.cgi?id=180853
<rdar://problem/35756268>
Reviewed by Tim Horton.
Source/WebCore:
Adds support for transforming dropped and pasted web content fragments prior to insertion, such that inline
elements (so far, only images) are replaced with attachment elements that have an inline representation. See
below comments for more detail.
Tests: WKAttachmentTests.InsertPastedImageAsAttachment
WKAttachmentTests.InsertPastedAttributedStringContainingImage
WKAttachmentTestsMac.InsertPastedFileURLsAsAttachments
WKAttachmentTestsIOS.InsertDroppedImageAsAttachment
WKAttachmentTestsIOS.InsertDroppedAttributedStringContainingAttachment
* editing/WebContentReader.h:
* editing/cocoa/EditorCocoa.mm:
(WebCore::Editor::replaceSelectionWithAttributedString):
Add a new helper to replace elements in a DOM fragment with inline attachment elements instead, using the given
Blobs. So far, we only replace image elements with these attachments, by mapping the source of each image to a
Blob, and constructing a replacement attachment backed by a File constructed from the image's corresponding
Blob. However, this mechanism can be generalized in the future to handle transformations from arbitrary elements
to attachment elements capable of representing the same elements using inline display mode.
This function is a noop if the attachment elements are disabled via runtime-enabled features.
* editing/cocoa/WebContentReaderCocoa.mm:
(WebCore::replaceRichContentWithAttachmentsIfNecessary):
(WebCore::createFragmentAndAddResources):
(WebCore::sanitizeMarkupWithArchive):
Add out-params to both of these helper functions that capture the map of blob URLs to Blobs being used to
replace subresource URLs in the pasted/dropped DOM fragment.
(WebCore::WebContentReader::readWebArchive):
(WebCore::WebContentMarkupReader::readWebArchive):
(WebCore::createFragmentFromAttributedString):
(WebCore::WebContentReader::readRTFD):
(WebCore::WebContentMarkupReader::readRTFD):
(WebCore::WebContentReader::readRTF):
(WebCore::WebContentMarkupReader::readRTF):
(WebCore::WebContentReader::readImage):
In these places where we swap out subresource URLs for blob URLs, collect a map of blob URL ="" Blob, and use it
to replace images in the DOM with attachments (if needed, and possible).
* editing/mac/WebContentReaderMac.mm:
(WebCore::WebContentReader::readFilenames):
Augment existing logic to generate attachment elements when pasting or dropping file URLs, so that the generated
attachment elements also have titles, subtitles, and content type information.
* html/HTMLAttachmentElement.cpp:
(WebCore::HTMLAttachmentElement::setFile):
Add an optional second param, UpdateDisplayAttributes. If UpdateDisplayAttributes::Yes is passed in, then we set
the elements's attributes that are displayed in the attachment representation (title, subtitle, and type) using
the given File.
(WebCore::HTMLAttachmentElement::updateFileWithData):
(WebCore::HTMLAttachmentElement::populateShadowRootIfNecessary):
Make a small tweak to correctly handle the case where an attachment with a content type that is a UTI is being
displayed inline. Content type can either be a UTI or a MIME type, but the code to construct the shadow tree of
an attachment element only handles MIME types.
* html/HTMLAttachmentElement.h:
Tools:
Adds new WKAttachment API tests and test support. See comments below for more detail.
* TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm:
Tweak the editing test page markup, such that document.body.innerHTML doesn't additionally contain the contents
of the script tag.
(webViewForTestingAttachments):
(testImageFileURL):
(testImageData):
(testPDFFileURL):
(testPDFData):
(platformCopyRichTextWithAttachment):
(platformCopyPNG):
(platformImageWithData):
Add some platform helper functions, which some of the platform-agnostic tests below use to resolve differences
between iOS and macOS when writing to the pasteboard and creating an image from data.
(TestWebKitAPI::TEST):
Add new API tests to exercise drag and drop/copy and paste of rich content (namely, images and files) on iOS and
macOS. iOS attachment tests use DataInteractionSimulator to simulate UIKit drag and drop coordination. On macOS,
handling of dropped content is much closer to handling of pasted content (they use the same codepaths to read
from the platform pasteboard), so exercising paste codepaths on Mac is sufficient.
* TestWebKitAPI/ios/DataInteractionSimulator.h:
* TestWebKitAPI/ios/DataInteractionSimulator.mm:
(-[DataInteractionSimulator _resetSimulatedState]):
(-[DataInteractionSimulator insertedAttachments]):
(-[DataInteractionSimulator removedAttachments]):
Teach the iOS drag and drop simulator to keep track of attachment elements that are inserted during a drop. We
also keep track of removed attachments here too, though no default drop handling scenario should trigger
attachment removal, so we simply use this to check that no _WKAttachments were removed during a drop.
(-[DataInteractionSimulator _webView:didInsertAttachment:]):
(-[DataInteractionSimulator _webView:didRemoveAttachment:]):
Modified Paths
Diff
Modified: trunk/Source/WebCore/ChangeLog (226084 => 226085)
--- trunk/Source/WebCore/ChangeLog 2017-12-19 00:15:33 UTC (rev 226084)
+++ trunk/Source/WebCore/ChangeLog 2017-12-19 00:19:17 UTC (rev 226085)
@@ -1,3 +1,75 @@
+2017-12-18 Wenson Hsieh <wenson_hs...@apple.com>
+
+ [Attachment Support] Insert images as inline attachments when pasting and dropping
+ https://bugs.webkit.org/show_bug.cgi?id=180853
+ <rdar://problem/35756268>
+
+ Reviewed by Tim Horton.
+
+ Adds support for transforming dropped and pasted web content fragments prior to insertion, such that inline
+ elements (so far, only images) are replaced with attachment elements that have an inline representation. See
+ below comments for more detail.
+
+ Tests: WKAttachmentTests.InsertPastedImageAsAttachment
+ WKAttachmentTests.InsertPastedAttributedStringContainingImage
+ WKAttachmentTestsMac.InsertPastedFileURLsAsAttachments
+ WKAttachmentTestsIOS.InsertDroppedImageAsAttachment
+ WKAttachmentTestsIOS.InsertDroppedAttributedStringContainingAttachment
+
+ * editing/WebContentReader.h:
+ * editing/cocoa/EditorCocoa.mm:
+ (WebCore::Editor::replaceSelectionWithAttributedString):
+
+ Add a new helper to replace elements in a DOM fragment with inline attachment elements instead, using the given
+ Blobs. So far, we only replace image elements with these attachments, by mapping the source of each image to a
+ Blob, and constructing a replacement attachment backed by a File constructed from the image's corresponding
+ Blob. However, this mechanism can be generalized in the future to handle transformations from arbitrary elements
+ to attachment elements capable of representing the same elements using inline display mode.
+
+ This function is a noop if the attachment elements are disabled via runtime-enabled features.
+
+ * editing/cocoa/WebContentReaderCocoa.mm:
+ (WebCore::replaceRichContentWithAttachmentsIfNecessary):
+ (WebCore::createFragmentAndAddResources):
+ (WebCore::sanitizeMarkupWithArchive):
+
+ Add out-params to both of these helper functions that capture the map of blob URLs to Blobs being used to
+ replace subresource URLs in the pasted/dropped DOM fragment.
+
+ (WebCore::WebContentReader::readWebArchive):
+ (WebCore::WebContentMarkupReader::readWebArchive):
+ (WebCore::createFragmentFromAttributedString):
+ (WebCore::WebContentReader::readRTFD):
+ (WebCore::WebContentMarkupReader::readRTFD):
+ (WebCore::WebContentReader::readRTF):
+ (WebCore::WebContentMarkupReader::readRTF):
+ (WebCore::WebContentReader::readImage):
+
+ In these places where we swap out subresource URLs for blob URLs, collect a map of blob URL ="" Blob, and use it
+ to replace images in the DOM with attachments (if needed, and possible).
+
+ * editing/mac/WebContentReaderMac.mm:
+ (WebCore::WebContentReader::readFilenames):
+
+ Augment existing logic to generate attachment elements when pasting or dropping file URLs, so that the generated
+ attachment elements also have titles, subtitles, and content type information.
+
+ * html/HTMLAttachmentElement.cpp:
+ (WebCore::HTMLAttachmentElement::setFile):
+
+ Add an optional second param, UpdateDisplayAttributes. If UpdateDisplayAttributes::Yes is passed in, then we set
+ the elements's attributes that are displayed in the attachment representation (title, subtitle, and type) using
+ the given File.
+
+ (WebCore::HTMLAttachmentElement::updateFileWithData):
+ (WebCore::HTMLAttachmentElement::populateShadowRootIfNecessary):
+
+ Make a small tweak to correctly handle the case where an attachment with a content type that is a UTI is being
+ displayed inline. Content type can either be a UTI or a MIME type, but the code to construct the shadow tree of
+ an attachment element only handles MIME types.
+
+ * html/HTMLAttachmentElement.h:
+
2017-12-18 Youenn Fablet <you...@apple.com>
SameOrigin and CORS fetch should fail on opaque responses served from ServiceWorker
Modified: trunk/Source/WebCore/editing/WebContentReader.h (226084 => 226085)
--- trunk/Source/WebCore/editing/WebContentReader.h 2017-12-19 00:15:33 UTC (rev 226084)
+++ trunk/Source/WebCore/editing/WebContentReader.h 2017-12-19 00:19:17 UTC (rev 226085)
@@ -32,6 +32,7 @@
namespace WebCore {
class ArchiveResource;
+class Blob;
class FrameWebContentReader : public PasteboardWebContentReader {
public:
@@ -105,7 +106,8 @@
Vector<Ref<ArchiveResource>> resources;
};
-RefPtr<DocumentFragment> createFragmentAndAddResources(Frame&, NSAttributedString*);
+void replaceRichContentWithAttachmentsIfNecessary(DocumentFragment&, HashMap<AtomicString, RefPtr<Blob>>&& urlToBlobMap);
+RefPtr<DocumentFragment> createFragmentAndAddResources(Frame&, NSAttributedString*, HashMap<AtomicString, RefPtr<Blob>>& urlToBlobMap);
#endif
}
Modified: trunk/Source/WebCore/editing/cocoa/EditorCocoa.mm (226084 => 226085)
--- trunk/Source/WebCore/editing/cocoa/EditorCocoa.mm 2017-12-19 00:15:33 UTC (rev 226084)
+++ trunk/Source/WebCore/editing/cocoa/EditorCocoa.mm 2017-12-19 00:19:17 UTC (rev 226085)
@@ -222,9 +222,12 @@
return;
if (m_frame.selection().selection().isContentRichlyEditable()) {
- RefPtr<DocumentFragment> fragment = createFragmentAndAddResources(m_frame, attributedString);
- if (fragment && shouldInsertFragment(*fragment, selectedRange().get(), EditorInsertAction::Pasted))
- pasteAsFragment(fragment.releaseNonNull(), false, false, mailBlockquoteHandling);
+ HashMap<AtomicString, RefPtr<Blob>> urlToBlobMap;
+ if (auto fragment = createFragmentAndAddResources(m_frame, attributedString, urlToBlobMap)) {
+ replaceRichContentWithAttachmentsIfNecessary(*fragment, WTFMove(urlToBlobMap));
+ if (shouldInsertFragment(*fragment, selectedRange().get(), EditorInsertAction::Pasted))
+ pasteAsFragment(fragment.releaseNonNull(), false, false, mailBlockquoteHandling);
+ }
} else {
String text = attributedString.string;
if (shouldInsertText(text, selectedRange().get(), EditorInsertAction::Pasted))
Modified: trunk/Source/WebCore/editing/cocoa/WebContentReaderCocoa.mm (226084 => 226085)
--- trunk/Source/WebCore/editing/cocoa/WebContentReaderCocoa.mm 2017-12-19 00:15:33 UTC (rev 226084)
+++ trunk/Source/WebCore/editing/cocoa/WebContentReaderCocoa.mm 2017-12-19 00:19:17 UTC (rev 226085)
@@ -34,9 +34,11 @@
#import "Document.h"
#import "DocumentFragment.h"
#import "DocumentLoader.h"
+#import "File.h"
#import "Frame.h"
#import "FrameLoader.h"
#import "FrameLoaderClient.h"
+#import "HTMLAttachmentElement.h"
#import "HTMLBodyElement.h"
#import "HTMLIFrameElement.h"
#import "HTMLImageElement.h"
@@ -44,6 +46,7 @@
#import "MainFrame.h"
#import "Page.h"
#import "PublicURLManager.h"
+#import "RuntimeEnabledFeatures.h"
#import "Settings.h"
#import "SocketProvider.h"
#import "WebArchiveResourceFromNSAttributedString.h"
@@ -52,6 +55,7 @@
#import "markup.h"
#import <pal/spi/cocoa/NSAttributedStringSPI.h>
#import <wtf/SoftLinking.h>
+#import <wtf/UUID.h>
#if (PLATFORM(IOS) && __IPHONE_OS_VERSION_MIN_REQUIRED >= 110000) || (PLATFORM(MAC) && __MAC_OS_X_VERSION_MIN_REQUIRED >= 101300)
@interface NSAttributedString ()
@@ -69,6 +73,62 @@
namespace WebCore {
+#if ENABLE(ATTACHMENT_ELEMENT)
+
+void replaceRichContentWithAttachmentsIfNecessary(DocumentFragment& fragment, HashMap<AtomicString, RefPtr<Blob>>&& urlToBlobMap)
+{
+ if (!RuntimeEnabledFeatures::sharedFeatures().attachmentElementEnabled() || urlToBlobMap.isEmpty())
+ return;
+
+ Vector<Ref<Element>> elementsToRemove;
+ Vector<std::pair<Ref<File>, Ref<Element>>> filesForElementsToReplace;
+ for (auto& image : descendantsOfType<HTMLImageElement>(fragment)) {
+ auto url = ""
+
+ if (url.isEmpty()) {
+ elementsToRemove.append(image);
+ continue;
+ }
+
+ auto blob = urlToBlobMap.get(url);
+ if (!blob) {
+ elementsToRemove.append(image);
+ continue;
+ }
+
+ auto title = image.attributeWithoutSynchronization(HTMLNames::titleAttr);
+ if (title.isEmpty())
+ title = AtomicString("media");
+
+ filesForElementsToReplace.append({ File::create(*blob, title), image });
+ }
+
+ for (auto& fileAndElement : filesForElementsToReplace) {
+ auto& file = fileAndElement.first;
+ auto& elementToReplace = fileAndElement.second;
+ auto parent = makeRefPtr(elementToReplace->parentNode());
+ if (!parent)
+ continue;
+
+ auto attachment = HTMLAttachmentElement::create(HTMLNames::attachmentTag, fragment.document());
+ attachment->setUniqueIdentifier(createCanonicalUUIDString());
+ attachment->setFile(WTFMove(file), HTMLAttachmentElement::UpdateDisplayAttributes::Yes);
+ attachment->updateDisplayMode(AttachmentDisplayMode::InPlace);
+ parent->replaceChild(attachment, elementToReplace);
+ }
+
+ for (auto& elementToRemove : elementsToRemove)
+ elementToRemove->remove();
+}
+
+#else
+
+void replaceRichContentWithAttachmentsIfNecessary(DocumentFragment&, HashMap<AtomicString, RefPtr<Blob>>&&)
+{
+}
+
+#endif
+
#if (PLATFORM(IOS) && __IPHONE_OS_VERSION_MIN_REQUIRED >= 110000) || (PLATFORM(MAC) && __MAC_OS_X_VERSION_MIN_REQUIRED >= 101300)
static NSDictionary *attributesForAttributedStringConversion()
@@ -159,7 +219,7 @@
bool m_didDisableImage { false };
};
-RefPtr<DocumentFragment> createFragmentAndAddResources(Frame& frame, NSAttributedString *string)
+RefPtr<DocumentFragment> createFragmentAndAddResources(Frame& frame, NSAttributedString *string, HashMap<AtomicString, RefPtr<Blob>>& urlToBlobMap)
{
if (!frame.page() || !frame.document())
return nullptr;
@@ -178,6 +238,7 @@
auto blob = Blob::create(subresource->data(), subresource->mimeType());
String blobURL = DOMURL::createObjectURL(document, blob);
blobURLMap.set(subresource->url().string(), blobURL);
+ urlToBlobMap.set(blobURL, WTFMove(blob));
}
replaceSubresourceURLs(*fragmentAndResources.fragment, WTFMove(blobURLMap));
@@ -207,7 +268,7 @@
return MarkupAndArchive { String::fromUTF8(mainResource->data().data(), mainResource->data().size()), mainResource.releaseNonNull(), archive.releaseNonNull() };
}
-static String sanitizeMarkupWithArchive(Document& destinationDocument, MarkupAndArchive& markupAndArchive, const std::function<bool(const String)>& canShowMIMETypeAsHTML)
+static String sanitizeMarkupWithArchive(Document& destinationDocument, MarkupAndArchive& markupAndArchive, const std::function<bool(const String)>& canShowMIMETypeAsHTML, HashMap<AtomicString, RefPtr<Blob>>& urlToBlobMap)
{
auto page = createPageForSanitizingWebContent();
Document* stagingDocument = page->mainFrame().document();
@@ -219,6 +280,7 @@
auto blob = Blob::create(subresource->data(), subresource->mimeType());
String blobURL = DOMURL::createObjectURL(destinationDocument, blob);
blobURLMap.set(subresource->url().string(), blobURL);
+ urlToBlobMap.set(blobURL, WTFMove(blob));
}
auto contentOrigin = SecurityOrigin::create(markupAndArchive.mainResource->url());
@@ -234,7 +296,7 @@
auto subframeURL = subframeMainResource->url();
MarkupAndArchive subframeContent = { String::fromUTF8(subframeMainResource->data().data(), subframeMainResource->data().size()),
subframeMainResource.releaseNonNull(), subframeArchive.copyRef() };
- auto subframeMarkup = sanitizeMarkupWithArchive(destinationDocument, subframeContent, canShowMIMETypeAsHTML);
+ auto subframeMarkup = sanitizeMarkupWithArchive(destinationDocument, subframeContent, canShowMIMETypeAsHTML, urlToBlobMap);
CString utf8 = subframeMarkup.utf8();
Vector<uint8_t> blobBuffer;
@@ -244,6 +306,7 @@
String subframeBlobURL = DOMURL::createObjectURL(destinationDocument, blob);
blobURLMap.set(subframeURL.string(), subframeBlobURL);
+ urlToBlobMap.set(subframeBlobURL, WTFMove(blob));
}
replaceSubresourceURLs(fragment.get(), WTFMove(blobURLMap));
@@ -274,11 +337,16 @@
return true;
}
+ HashMap<AtomicString, RefPtr<Blob>> urlToBlobMap;
String sanitizedMarkup = sanitizeMarkupWithArchive(*frame.document(), *result, [&] (const String& type) {
return frame.loader().client().canShowMIMETypeAsHTML(type);
- });
+ }, urlToBlobMap);
fragment = createFragmentFromMarkup(*frame.document(), sanitizedMarkup, blankURL(), DisallowScriptingAndPluginContent);
+ if (!fragment)
+ return false;
+
+ replaceRichContentWithAttachmentsIfNecessary(*fragment, WTFMove(urlToBlobMap));
return true;
}
@@ -298,9 +366,10 @@
return true;
}
+ HashMap<AtomicString, RefPtr<Blob>> urlToBlobMap;
markup = sanitizeMarkupWithArchive(*frame.document(), *result, [&] (const String& type) {
return frame.loader().client().canShowMIMETypeAsHTML(type);
- });
+ }, urlToBlobMap);
return true;
}
@@ -348,12 +417,24 @@
return !markup.isEmpty();
}
+static RefPtr<DocumentFragment> createFragmentFromAttributedString(Frame& frame, NSAttributedString *string)
+{
+ HashMap<AtomicString, RefPtr<Blob>> urlToBlobMap;
+ auto fragment = createFragmentAndAddResources(frame, string, urlToBlobMap);
+ if (!fragment)
+ return nullptr;
+
+ replaceRichContentWithAttachmentsIfNecessary(*fragment, WTFMove(urlToBlobMap));
+ return fragment;
+}
+
bool WebContentReader::readRTFD(SharedBuffer& buffer)
{
if (frame.settings().preferMIMETypeForImages() || !frame.document())
return false;
- auto fragment = createFragmentAndAddResources(frame, adoptNS([[NSAttributedString alloc] initWithRTFD:buffer.createNSData().get() documentAttributes:nullptr]).get());
+ auto string = adoptNS([[NSAttributedString alloc] initWithRTFD:buffer.createNSData().get() documentAttributes:nullptr]);
+ auto fragment = createFragmentFromAttributedString(frame, string.get());
if (!fragment)
return false;
addFragment(fragment.releaseNonNull());
@@ -365,7 +446,11 @@
{
if (!frame.document())
return false;
- auto fragment = createFragmentAndAddResources(frame, adoptNS([[NSAttributedString alloc] initWithRTFD:buffer.createNSData().get() documentAttributes:nullptr]).get());
+ auto string = adoptNS([[NSAttributedString alloc] initWithRTFD:buffer.createNSData().get() documentAttributes:nullptr]);
+ auto fragment = createFragmentFromAttributedString(frame, string.get());
+ if (!fragment)
+ return false;
+
markup = createMarkup(*fragment);
return true;
}
@@ -375,7 +460,8 @@
if (frame.settings().preferMIMETypeForImages())
return false;
- auto fragment = createFragmentAndAddResources(frame, adoptNS([[NSAttributedString alloc] initWithRTF:buffer.createNSData().get() documentAttributes:nullptr]).get());
+ auto string = adoptNS([[NSAttributedString alloc] initWithRTF:buffer.createNSData().get() documentAttributes:nullptr]);
+ auto fragment = createFragmentFromAttributedString(frame, string.get());
if (!fragment)
return false;
addFragment(fragment.releaseNonNull());
@@ -387,7 +473,8 @@
{
if (!frame.document())
return false;
- auto fragment = createFragmentAndAddResources(frame, adoptNS([[NSAttributedString alloc] initWithRTF:buffer.createNSData().get() documentAttributes:nullptr]).get());
+ auto string = adoptNS([[NSAttributedString alloc] initWithRTF:buffer.createNSData().get() documentAttributes:nullptr]);
+ auto fragment = createFragmentFromAttributedString(frame, string.get());
if (!fragment)
return false;
markup = createMarkup(*fragment);
@@ -412,6 +499,11 @@
auto& document = *frame.document();
String blobURL = DOMURL::createObjectURL(document, blob);
addFragment(createFragmentForImageAndURL(document, blobURL));
+
+ if (!fragment)
+ return false;
+
+ replaceRichContentWithAttachmentsIfNecessary(*fragment, {{ blobURL, WTFMove(blob) }});
return true;
}
Modified: trunk/Source/WebCore/editing/mac/WebContentReaderMac.mm (226084 => 226085)
--- trunk/Source/WebCore/editing/mac/WebContentReaderMac.mm 2017-12-19 00:15:33 UTC (rev 226084)
+++ trunk/Source/WebCore/editing/mac/WebContentReaderMac.mm 2017-12-19 00:19:17 UTC (rev 226085)
@@ -47,6 +47,7 @@
#import "Text.h"
#import "WebCoreNSURLExtras.h"
#import "markup.h"
+#import <wtf/UUID.h>
namespace WebCore {
@@ -65,7 +66,8 @@
#if ENABLE(ATTACHMENT_ELEMENT)
if (RuntimeEnabledFeatures::sharedFeatures().attachmentElementEnabled()) {
auto attachment = HTMLAttachmentElement::create(HTMLNames::attachmentTag, document);
- attachment->setFile(File::create([[NSURL fileURLWithPath:text] path]));
+ attachment->setUniqueIdentifier(createCanonicalUUIDString());
+ attachment->setFile(File::create([NSURL fileURLWithPath:text].path), HTMLAttachmentElement::UpdateDisplayAttributes::Yes);
fragment->appendChild(attachment);
continue;
}
Modified: trunk/Source/WebCore/html/HTMLAttachmentElement.cpp (226084 => 226085)
--- trunk/Source/WebCore/html/HTMLAttachmentElement.cpp 2017-12-19 00:15:33 UTC (rev 226084)
+++ trunk/Source/WebCore/html/HTMLAttachmentElement.cpp 2017-12-19 00:19:17 UTC (rev 226085)
@@ -45,6 +45,10 @@
#include "SharedBuffer.h"
#include <pal/FileSizeFormatter.h>
+#if PLATFORM(COCOA)
+#include "UTIUtilities.h"
+#endif
+
namespace WebCore {
using namespace HTMLNames;
@@ -112,12 +116,24 @@
return { { }, attributeWithoutSynchronization(HTMLNames::webkitattachmentbloburlAttr).string() };
}
-void HTMLAttachmentElement::setFile(RefPtr<File>&& file)
+void HTMLAttachmentElement::setFile(RefPtr<File>&& file, UpdateDisplayAttributes updateAttributes)
{
m_file = WTFMove(file);
setAttributeWithoutSynchronization(HTMLNames::webkitattachmentbloburlAttr, m_file ? m_file->url() : emptyString());
+ if (updateAttributes == UpdateDisplayAttributes::Yes) {
+ if (m_file) {
+ setAttributeWithoutSynchronization(HTMLNames::titleAttr, m_file->name());
+ setAttributeWithoutSynchronization(HTMLNames::subtitleAttr, fileSizeDescription(m_file->size()));
+ setAttributeWithoutSynchronization(HTMLNames::typeAttr, m_file->type());
+ } else {
+ removeAttribute(HTMLNames::titleAttr);
+ removeAttribute(HTMLNames::subtitleAttr);
+ removeAttribute(HTMLNames::typeAttr);
+ }
+ }
+
if (auto* renderAttachment = attachmentRenderer())
renderAttachment->invalidate();
@@ -223,11 +239,7 @@
auto filename = newFilename ? *newFilename : attachmentTitle();
auto contentType = newContentType ? *newContentType : File::contentTypeForFile(filename);
auto file = File::create(Blob::create(WTFMove(data), contentType), filename);
-
- setAttributeWithoutSynchronization(titleAttr, filename);
- setAttributeWithoutSynchronization(subtitleAttr, fileSizeDescription(file->size()));
- setAttributeWithoutSynchronization(typeAttr, contentType);
- setFile(WTFMove(file));
+ setFile(WTFMove(file), UpdateDisplayAttributes::Yes);
}
Ref<HTMLImageElement> HTMLAttachmentElement::ensureInnerImage()
@@ -266,8 +278,17 @@
void HTMLAttachmentElement::populateShadowRootIfNecessary()
{
+ if (!m_file)
+ return;
+
auto mimeType = attachmentType();
- if (!m_file || mimeType.isEmpty())
+
+#if PLATFORM(COCOA)
+ if (isDeclaredUTI(mimeType))
+ mimeType = MIMETypeFromUTI(mimeType);
+#endif
+
+ if (mimeType.isEmpty())
return;
if (MIMETypeRegistry::isSupportedImageMIMEType(mimeType) || MIMETypeRegistry::isPDFMIMEType(mimeType)) {
Modified: trunk/Source/WebCore/html/HTMLAttachmentElement.h (226084 => 226085)
--- trunk/Source/WebCore/html/HTMLAttachmentElement.h 2017-12-19 00:15:33 UTC (rev 226084)
+++ trunk/Source/WebCore/html/HTMLAttachmentElement.h 2017-12-19 00:19:17 UTC (rev 226085)
@@ -45,8 +45,10 @@
WEBCORE_EXPORT URL blobURL() const;
WEBCORE_EXPORT File* file() const;
- void setFile(RefPtr<File>&&);
+ enum class UpdateDisplayAttributes { No, Yes };
+ void setFile(RefPtr<File>&&, UpdateDisplayAttributes = UpdateDisplayAttributes::No);
+
WEBCORE_EXPORT String uniqueIdentifier() const;
void setUniqueIdentifier(const String&);
Modified: trunk/Tools/ChangeLog (226084 => 226085)
--- trunk/Tools/ChangeLog 2017-12-19 00:15:33 UTC (rev 226084)
+++ trunk/Tools/ChangeLog 2017-12-19 00:19:17 UTC (rev 226085)
@@ -1,3 +1,50 @@
+2017-12-18 Wenson Hsieh <wenson_hs...@apple.com>
+
+ [Attachment Support] Insert images as inline attachments when pasting and dropping
+ https://bugs.webkit.org/show_bug.cgi?id=180853
+ <rdar://problem/35756268>
+
+ Reviewed by Tim Horton.
+
+ Adds new WKAttachment API tests and test support. See comments below for more detail.
+
+ * TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm:
+
+ Tweak the editing test page markup, such that document.body.innerHTML doesn't additionally contain the contents
+ of the script tag.
+
+ (webViewForTestingAttachments):
+ (testImageFileURL):
+ (testImageData):
+ (testPDFFileURL):
+ (testPDFData):
+ (platformCopyRichTextWithAttachment):
+ (platformCopyPNG):
+ (platformImageWithData):
+
+ Add some platform helper functions, which some of the platform-agnostic tests below use to resolve differences
+ between iOS and macOS when writing to the pasteboard and creating an image from data.
+
+ (TestWebKitAPI::TEST):
+
+ Add new API tests to exercise drag and drop/copy and paste of rich content (namely, images and files) on iOS and
+ macOS. iOS attachment tests use DataInteractionSimulator to simulate UIKit drag and drop coordination. On macOS,
+ handling of dropped content is much closer to handling of pasted content (they use the same codepaths to read
+ from the platform pasteboard), so exercising paste codepaths on Mac is sufficient.
+
+ * TestWebKitAPI/ios/DataInteractionSimulator.h:
+ * TestWebKitAPI/ios/DataInteractionSimulator.mm:
+ (-[DataInteractionSimulator _resetSimulatedState]):
+ (-[DataInteractionSimulator insertedAttachments]):
+ (-[DataInteractionSimulator removedAttachments]):
+
+ Teach the iOS drag and drop simulator to keep track of attachment elements that are inserted during a drop. We
+ also keep track of removed attachments here too, though no default drop handling scenario should trigger
+ attachment removal, so we simply use this to check that no _WKAttachments were removed during a drop.
+
+ (-[DataInteractionSimulator _webView:didInsertAttachment:]):
+ (-[DataInteractionSimulator _webView:didRemoveAttachment:]):
+
2017-12-18 Fujii Hironori <hironori.fu...@sony.com>
[WinCairo] Move the destination of WinCairoRequirements.zip into WebKitLibraries and register it and related files as git ignore files.
Modified: trunk/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm (226084 => 226085)
--- trunk/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm 2017-12-19 00:15:33 UTC (rev 226084)
+++ trunk/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm 2017-12-19 00:19:17 UTC (rev 226085)
@@ -25,13 +25,19 @@
#import "config.h"
+#import "DataInteractionSimulator.h"
#import "PlatformUtilities.h"
#import "TestWKWebView.h"
+#import <WebKit/WKPreferencesRefPrivate.h>
#import <WebKit/WKWebViewPrivate.h>
#import <WebKit/WebKit.h>
#import <WebKit/WebKitPrivate.h>
#import <wtf/RetainPtr.h>
+#if PLATFORM(IOS)
+#import <MobileCoreServices/MobileCoreServices.h>
+#endif
+
#if WK_API_ENABLED
@interface AttachmentUpdateObserver : NSObject <WKUIDelegatePrivate>
@@ -122,9 +128,10 @@
{
auto configuration = adoptNS([[WKWebViewConfiguration alloc] init]);
[configuration _setAttachmentElementEnabled:YES];
+ WKPreferencesSetCustomPasteboardDataEnabled((WKPreferencesRef)[configuration preferences], YES);
auto webView = adoptNS([[TestWKWebView alloc] initWithFrame:CGRectMake(0, 0, 320, 500) configuration:configuration.get()]);
- [webView synchronouslyLoadHTMLString:@"<body contenteditable></body><script>document.body.focus();</script>"];
+ [webView synchronouslyLoadHTMLString:@"<script>focus = () => document.body.focus()</script><body _onload_=focus() contenteditable></body>"];
return webView;
}
@@ -134,10 +141,14 @@
return [@"<a href=''>This is some HTML data</a>" dataUsingEncoding:NSUTF8StringEncoding];
}
+static NSURL *testImageFileURL()
+{
+ return [[NSBundle mainBundle] URLForResource:@"icon" withExtension:@"png" subdirectory:@"TestWebKitAPI.resources"];
+}
+
static NSData *testImageData()
{
- NSURL *url = "" mainBundle] URLForResource:@"icon" withExtension:@"png" subdirectory:@"TestWebKitAPI.resources"];
- return [NSData dataWithContentsOfURL:url];
+ return [NSData dataWithContentsOfURL:testImageFileURL()];
}
static NSData *testVideoData()
@@ -146,10 +157,14 @@
return [NSData dataWithContentsOfURL:url];
}
+static NSURL *testPDFFileURL()
+{
+ return [[NSBundle mainBundle] URLForResource:@"test" withExtension:@"pdf" subdirectory:@"TestWebKitAPI.resources"];
+}
+
static NSData *testPDFData()
{
- NSURL *url = "" mainBundle] URLForResource:@"test" withExtension:@"pdf" subdirectory:@"TestWebKitAPI.resources"];
- return [NSData dataWithContentsOfURL:url];
+ return [NSData dataWithContentsOfURL:testPDFFileURL()];
}
static _WKAttachmentDisplayOptions *displayOptionsWithMode(_WKAttachmentDisplayMode mode)
@@ -291,8 +306,67 @@
@end
+#pragma mark - Platform testing helper functions
+
+void platformCopyRichTextWithImage()
+{
+ auto richText = adoptNS([[NSMutableAttributedString alloc] init]);
+ auto image = adoptNS([[NSTextAttachment alloc] initWithData:testImageData() ofType:(NSString *)kUTTypePNG]);
+
+ [richText appendAttributedString:[[[NSAttributedString alloc] initWithString:@"Lorem ipsum "] autorelease]];
+ [richText appendAttributedString:[NSAttributedString attributedStringWithAttachment:image.get()]];
+ [richText appendAttributedString:[[[NSAttributedString alloc] initWithString:@" dolor sit amet."] autorelease]];
+
+#if PLATFORM(MAC)
+ NSPasteboard *pasteboard = [NSPasteboard generalPasteboard];
+ [pasteboard clearContents];
+ [pasteboard writeObjects:@[ richText.get() ]];
+#elif PLATFORM(IOS)
+ auto item = adoptNS([[NSItemProvider alloc] init]);
+ [item registerObject:richText.get() visibility:NSItemProviderRepresentationVisibilityAll];
+ [UIPasteboard generalPasteboard].itemProviders = @[ item.get() ];
+#endif
+}
+
+typedef void(^ItemProviderDataLoadHandler)(NSData *, NSError *);
+
+void platformCopyPNG()
+{
+#if PLATFORM(MAC)
+ NSPasteboard *pasteboard = [NSPasteboard generalPasteboard];
+ [pasteboard declareTypes:@[NSPasteboardTypePNG] owner:nil];
+ [pasteboard setData:testImageData() forType:NSPasteboardTypePNG];
+#elif PLATFORM(IOS)
+ UIPasteboard *pasteboard = [UIPasteboard generalPasteboard];
+ auto item = adoptNS([[UIItemProvider alloc] init]);
+ [item setPreferredPresentationStyle:UIPreferredPresentationStyleAttachment];
+ [item registerDataRepresentationForTypeIdentifier:(NSString *)kUTTypePNG visibility:NSItemProviderRepresentationVisibilityAll loadHandler:[] (ItemProviderDataLoadHandler completionHandler) -> NSProgress * {
+ completionHandler(testImageData(), nil);
+ return nil;
+ }];
+ pasteboard.itemProviders = @[ item.get() ];
+#endif
+}
+
+#if PLATFORM(MAC)
+typedef NSImage PlatformImage;
+#else
+typedef UIImage PlatformImage;
+#endif
+
+PlatformImage *platformImageWithData(NSData *data)
+{
+#if PLATFORM(MAC)
+ return [[[NSImage alloc] initWithData:data] autorelease];
+#else
+ return [UIImage imageWithData:data];
+#endif
+}
+
namespace TestWebKitAPI {
+#pragma mark - Platform-agnostic tests
+
TEST(WKAttachmentTests, AttachmentElementInsertion)
{
auto webView = webViewForTestingAttachments();
@@ -675,6 +749,157 @@
[webView expectUpdatesAfterCommand:@"DeleteBackward" withArgument:nil expectedRemovals:@[attachment.get()] expectedInsertions:@[]];
}
+TEST(WKAttachmentTests, InsertPastedImageAsAttachment)
+{
+ platformCopyPNG();
+
+ RetainPtr<_WKAttachment> attachment;
+ auto webView = webViewForTestingAttachments();
+ {
+ ObserveAttachmentUpdatesForScope observer(webView.get());
+ [webView _synchronouslyExecuteEditCommand:@"Paste" argument:nil];
+ EXPECT_EQ(1U, observer.observer().inserted.count);
+ attachment = observer.observer().inserted[0];
+ }
+
+ auto size = platformImageWithData([attachment synchronouslyRequestData:nil]).size;
+ EXPECT_EQ(215., size.width);
+ EXPECT_EQ(174., size.height);
+
+ {
+ ObserveAttachmentUpdatesForScope observer(webView.get());
+ [webView _synchronouslyExecuteEditCommand:@"SelectAll" argument:nil];
+ [webView _synchronouslyExecuteEditCommand:@"DeleteBackward" argument:nil];
+ observer.expectAttachmentUpdates(@[attachment.get()], @[]);
+ }
+}
+
+TEST(WKAttachmentTests, InsertPastedAttributedStringContainingImage)
+{
+ platformCopyRichTextWithImage();
+
+ RetainPtr<_WKAttachment> attachment;
+ auto webView = webViewForTestingAttachments();
+ {
+ ObserveAttachmentUpdatesForScope observer(webView.get());
+ [webView _synchronouslyExecuteEditCommand:@"Paste" argument:nil];
+ EXPECT_EQ(0U, observer.observer().removed.count);
+ EXPECT_EQ(1U, observer.observer().inserted.count);
+ attachment = observer.observer().inserted[0];
+ }
+
+ [attachment expectRequestedDataToBe:testImageData()];
+ EXPECT_WK_STREQ("Lorem ipsum dolor sit amet.", [webView stringByEvaluatingJavaScript:@"document.body.textContent"]);
+ EXPECT_WK_STREQ("image/png", [webView valueOfAttribute:@"type" forQuerySelector:@"attachment"]);
+
+ {
+ ObserveAttachmentUpdatesForScope observer(webView.get());
+ [webView _synchronouslyExecuteEditCommand:@"SelectAll" argument:nil];
+ [webView _synchronouslyExecuteEditCommand:@"DeleteBackward" argument:nil];
+ observer.expectAttachmentUpdates(@[attachment.get()], @[]);
+ }
+}
+
+#pragma mark - Platform-specific tests
+
+#if PLATFORM(MAC)
+
+TEST(WKAttachmentTestsMac, InsertPastedFileURLsAsAttachments)
+{
+ NSPasteboard *pasteboard = [NSPasteboard generalPasteboard];
+ [pasteboard clearContents];
+ [pasteboard declareTypes:@[NSFilenamesPboardType] owner:nil];
+ [pasteboard setPropertyList:@[testPDFFileURL().path, testImageFileURL().path] forType:NSFilenamesPboardType];
+
+ RetainPtr<NSArray<_WKAttachment *>> insertedAttachments;
+ auto webView = webViewForTestingAttachments();
+ {
+ ObserveAttachmentUpdatesForScope observer(webView.get());
+ [webView _synchronouslyExecuteEditCommand:@"Paste" argument:nil];
+ insertedAttachments = [observer.observer() inserted];
+ EXPECT_EQ(2U, [insertedAttachments count]);
+ }
+
+ NSArray<NSData *> *expectedAttachmentData = @[ testPDFData(), testImageData() ];
+ EXPECT_TRUE([expectedAttachmentData containsObject:[[insertedAttachments firstObject] synchronouslyRequestData:nil]]);
+ EXPECT_TRUE([expectedAttachmentData containsObject:[[insertedAttachments lastObject] synchronouslyRequestData:nil]]);
+ EXPECT_WK_STREQ("application/pdf", [webView stringByEvaluatingJavaScript:@"document.querySelectorAll('attachment')[0].getAttribute('type')"]);
+ EXPECT_WK_STREQ("test.pdf", [webView stringByEvaluatingJavaScript:@"document.querySelectorAll('attachment')[0].getAttribute('title')"]);
+ EXPECT_WK_STREQ("image/png", [webView stringByEvaluatingJavaScript:@"document.querySelectorAll('attachment')[1].getAttribute('type')"]);
+ EXPECT_WK_STREQ("icon.png", [webView stringByEvaluatingJavaScript:@"document.querySelectorAll('attachment')[1].getAttribute('title')"]);
+
+ {
+ ObserveAttachmentUpdatesForScope observer(webView.get());
+ [webView _synchronouslyExecuteEditCommand:@"SelectAll" argument:nil];
+ [webView _synchronouslyExecuteEditCommand:@"DeleteBackward" argument:nil];
+ NSArray<_WKAttachment *> *removedAttachments = [observer.observer() removed];
+ EXPECT_EQ(2U, removedAttachments.count);
+ EXPECT_TRUE([removedAttachments containsObject:[insertedAttachments firstObject]]);
+ EXPECT_TRUE([removedAttachments.lastObject isEqual:[insertedAttachments lastObject]]);
+ }
+}
+
+#endif // PLATFORM(MAC)
+
+#if PLATFORM(IOS)
+
+TEST(WKAttachmentTestsIOS, InsertDroppedImageAsAttachment)
+{
+ auto webView = webViewForTestingAttachments();
+ auto draggingSimulator = adoptNS([[DataInteractionSimulator alloc] initWithWebView:webView.get()]);
+ auto item = adoptNS([[NSItemProvider alloc] init]);
+ [item setPreferredPresentationStyle:UIPreferredPresentationStyleAttachment];
+ [item registerDataRepresentationForTypeIdentifier:(NSString *)kUTTypePNG visibility:NSItemProviderRepresentationVisibilityAll loadHandler:[] (ItemProviderDataLoadHandler completionHandler) -> NSProgress * {
+ completionHandler(testImageData(), nil);
+ return nil;
+ }];
+ [draggingSimulator setExternalItemProviders:@[ item.get() ]];
+ [draggingSimulator runFrom:CGPointZero to:CGPointMake(50, 50)];
+
+ EXPECT_EQ(1U, [draggingSimulator insertedAttachments].count);
+ EXPECT_EQ(0U, [draggingSimulator removedAttachments].count);
+ auto attachment = retainPtr([draggingSimulator insertedAttachments].firstObject);
+ [attachment expectRequestedDataToBe:testImageData()];
+ EXPECT_WK_STREQ("public.png", [webView valueOfAttribute:@"type" forQuerySelector:@"attachment"]);
+
+ {
+ ObserveAttachmentUpdatesForScope observer(webView.get());
+ [webView _synchronouslyExecuteEditCommand:@"SelectAll" argument:nil];
+ [webView _synchronouslyExecuteEditCommand:@"DeleteBackward" argument:nil];
+ observer.expectAttachmentUpdates(@[attachment.get()], @[]);
+ }
+}
+
+TEST(WKAttachmentTestsIOS, InsertDroppedAttributedStringContainingAttachment)
+{
+ auto webView = webViewForTestingAttachments();
+ auto draggingSimulator = adoptNS([[DataInteractionSimulator alloc] initWithWebView:webView.get()]);
+ auto image = adoptNS([[NSTextAttachment alloc] initWithData:testImageData() ofType:(NSString *)kUTTypePNG]);
+ auto item = adoptNS([[NSItemProvider alloc] init]);
+ [item registerObject:[NSAttributedString attributedStringWithAttachment:image.get()] visibility:NSItemProviderRepresentationVisibilityAll];
+
+ [draggingSimulator setExternalItemProviders:@[ item.get() ]];
+ [draggingSimulator runFrom:CGPointZero to:CGPointMake(50, 50)];
+
+ EXPECT_EQ(1U, [draggingSimulator insertedAttachments].count);
+ EXPECT_EQ(0U, [draggingSimulator removedAttachments].count);
+ auto attachment = retainPtr([draggingSimulator insertedAttachments].firstObject);
+
+ auto size = platformImageWithData([attachment synchronouslyRequestData:nil]).size;
+ EXPECT_EQ(215., size.width);
+ EXPECT_EQ(174., size.height);
+ EXPECT_WK_STREQ("image/png", [webView valueOfAttribute:@"type" forQuerySelector:@"attachment"]);
+
+ {
+ ObserveAttachmentUpdatesForScope observer(webView.get());
+ [webView _synchronouslyExecuteEditCommand:@"SelectAll" argument:nil];
+ [webView _synchronouslyExecuteEditCommand:@"DeleteBackward" argument:nil];
+ observer.expectAttachmentUpdates(@[attachment.get()], @[]);
+ }
+}
+
+#endif // PLATFORM(IOS)
+
} // namespace TestWebKitAPI
#endif // WK_API_ENABLED
Modified: trunk/Tools/TestWebKitAPI/ios/DataInteractionSimulator.h (226084 => 226085)
--- trunk/Tools/TestWebKitAPI/ios/DataInteractionSimulator.h 2017-12-19 00:15:33 UTC (rev 226084)
+++ trunk/Tools/TestWebKitAPI/ios/DataInteractionSimulator.h 2017-12-19 00:19:17 UTC (rev 226085)
@@ -138,6 +138,9 @@
RetainPtr<NSMutableArray<NSValue *>>_queuedAdditionalItemRequestLocations;
RetainPtr<NSMutableArray<UITargetedDragPreview *>> _liftPreviews;
+ RetainPtr<NSMutableArray<_WKAttachment *>> _insertedAttachments;
+ RetainPtr<NSMutableArray<_WKAttachment *>> _removedAttachments;
+
bool _isDoneWaitingForInputSession;
BOOL _shouldPerformOperation;
double _currentProgress;
@@ -168,6 +171,9 @@
@property (nonatomic, readonly) CGRect lastKnownDragCaretRect;
@property (nonatomic, readonly) NSArray<UITargetedDragPreview *> *liftPreviews;
+@property (nonatomic, readonly) NSArray<_WKAttachment *> *insertedAttachments;
+@property (nonatomic, readonly) NSArray<_WKAttachment *> *removedAttachments;
+
@end
#endif // ENABLE(DATA_INTERACTION)
Modified: trunk/Tools/TestWebKitAPI/ios/DataInteractionSimulator.mm (226084 => 226085)
--- trunk/Tools/TestWebKitAPI/ios/DataInteractionSimulator.mm 2017-12-19 00:15:33 UTC (rev 226084)
+++ trunk/Tools/TestWebKitAPI/ios/DataInteractionSimulator.mm 2017-12-19 00:19:17 UTC (rev 226085)
@@ -317,6 +317,8 @@
_currentProgress = 0;
_isDoneWithCurrentRun = false;
_observedEventNames = adoptNS([[NSMutableArray alloc] init]);
+ _insertedAttachments = adoptNS([[NSMutableArray alloc] init]);
+ _removedAttachments = adoptNS([[NSMutableArray alloc] init]);
_finalSelectionRects = @[ ];
_dragSession = nil;
_dropSession = nil;
@@ -576,6 +578,16 @@
Util::run(&_isDoneWaitingForInputSession);
}
+- (NSArray<_WKAttachment *> *)insertedAttachments
+{
+ return _insertedAttachments.get();
+}
+
+- (NSArray<_WKAttachment *> *)removedAttachments
+{
+ return _removedAttachments.get();
+}
+
#pragma mark - WKUIDelegatePrivate
- (void)_webView:(WKWebView *)webView dataInteractionOperationWasHandled:(BOOL)handled forSession:(id)session itemProviders:(NSArray<UIItemProvider *> *)itemProviders
@@ -617,6 +629,16 @@
return self.overridePerformDropBlock ? self.overridePerformDropBlock(session) : session.items;
}
+- (void)_webView:(WKWebView *)webView didInsertAttachment:(_WKAttachment *)attachment
+{
+ [_insertedAttachments addObject:attachment];
+}
+
+- (void)_webView:(WKWebView *)webView didRemoveAttachment:(_WKAttachment *)attachment
+{
+ [_removedAttachments addObject:attachment];
+}
+
#pragma mark - _WKInputDelegate
- (BOOL)_webView:(WKWebView *)webView focusShouldStartInputSession:(id <_WKFocusedElementInfo>)info
