- Revision
- 268299
- Author
- wenson_hs...@apple.com
- Date
- 2020-10-09 16:39:34 -0700 (Fri, 09 Oct 2020)
Log Message
[GPU Process] Add additional support for painting video elements to 2D contexts
https://bugs.webkit.org/show_bug.cgi?id=217532
Reviewed by Tim Horton.
Source/WebCore:
Add support for using `video` elements as sources for `createImageBitmap` and `createPattern` by ensuring that
we use display-list-backed image buffers for painting when using the GPU process for media. See below for more
details.
Tests: fast/canvas/canvas-createPattern-video-loading.html
fast/canvas/canvas-createPattern-video-modify.html
imported/w3c/web-platform-tests/html/canvas/element/imagebitmap/canvas-createImageBitmap-video-resize.html
imported/w3c/web-platform-tests/html/canvas/element/imagebitmap/createImageBitmap-drawImage.html
imported/w3c/web-platform-tests/html/canvas/element/imagebitmap/createImageBitmap-flipY.html
* html/HTMLVideoElement.cpp:
(WebCore::HTMLVideoElement::createBufferForPainting const):
* html/HTMLVideoElement.h:
Add a helper function to create an `ImageBuffer` compatible with this video element, for the purpose of
painting video frames. In particular, if GPU process for media is enabled, we'll create a display-list-backed
remote image buffer.
* html/ImageBitmap.cpp:
(WebCore::ImageBitmap::createPromise):
Call into `HTMLVideoElement::createBufferForPainting` to construct the `ImageBuffer`.
* html/canvas/CanvasRenderingContext2DBase.cpp:
(WebCore::CanvasRenderingContext2DBase::createPattern):
Call into `HTMLVideoElement::createBufferForPainting` to construct the `ImageBuffer`.
* platform/graphics/RenderingMode.h:
Add a "purpose" enum value for media painting.
* platform/graphics/displaylists/DisplayListImageBuffer.h:
Adjust this to avoid forcing the platform context to be created in the case where there are no display list
items to flush. Needed to avoid crashing when creating `CanvasPattern` with a video element.
Source/WebKit:
See WebCore/ChangeLog for more details.
* WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::shouldUseRemoteRenderingFor):
LayoutTests:
Unskip several layout tests that pass after this change.
* gpu-process/TestExpectations:
Modified Paths
Diff
Modified: trunk/LayoutTests/ChangeLog (268298 => 268299)
--- trunk/LayoutTests/ChangeLog 2020-10-09 23:34:11 UTC (rev 268298)
+++ trunk/LayoutTests/ChangeLog 2020-10-09 23:39:34 UTC (rev 268299)
@@ -1,3 +1,14 @@
+2020-10-09 Wenson Hsieh <wenson_hs...@apple.com>
+
+ [GPU Process] Add additional support for painting video elements to 2D contexts
+ https://bugs.webkit.org/show_bug.cgi?id=217532
+
+ Reviewed by Tim Horton.
+
+ Unskip several layout tests that pass after this change.
+
+ * gpu-process/TestExpectations:
+
2020-10-09 Hector Lopez <hector_i_lo...@apple.com>
imported/w3c/web-platform-tests/mediacapture-record/idlharness.window.html is a flaky crash
Modified: trunk/LayoutTests/gpu-process/TestExpectations (268298 => 268299)
--- trunk/LayoutTests/gpu-process/TestExpectations 2020-10-09 23:34:11 UTC (rev 268298)
+++ trunk/LayoutTests/gpu-process/TestExpectations 2020-10-09 23:39:34 UTC (rev 268299)
@@ -26,8 +26,6 @@
fast/replaced/replaced-element-with-percentage-width-inside-flow-asserts.html [ Skip ]
fast/replaced/border-radius-clip-content-edge.html [ Skip ]
fast/canvas/canvas-drawImage-composite-copy.html [ ImageOnlyFailure ]
-fast/canvas/canvas-createPattern-video-loading.html [ Failure ]
-fast/canvas/canvas-createPattern-video-modify.html [ Failure ]
fast/canvas/canvas-large-dimensions.html [ Failure ]
fast/canvas/canvas-render-layer.html [ Failure ]
fast/canvas/check-stale-putImageData.html [ Failure ]
@@ -91,9 +89,6 @@
imported/w3c/web-platform-tests/imagebitmap-renderingcontext/transferFromImageBitmap-detached.html [ Skip ]
imported/w3c/web-platform-tests/imagebitmap-renderingcontext/tranferFromImageBitmap-null.html [ Skip ]
imported/w3c/web-platform-tests/html/canvas/element/fill-and-stroke-styles/2d.pattern.animated.gif.html [ Failure ]
-imported/w3c/web-platform-tests/html/canvas/element/imagebitmap/canvas-createImageBitmap-video-resize.html [ Failure ]
-imported/w3c/web-platform-tests/html/canvas/element/imagebitmap/createImageBitmap-drawImage.html [ Failure ]
-imported/w3c/web-platform-tests/html/canvas/element/imagebitmap/createImageBitmap-flipY.html [ Failure ]
imported/w3c/web-platform-tests/html/canvas/element/imagebitmap/createImageBitmap-origin.sub.html [ Failure ]
imported/w3c/web-platform-tests/html/canvas/element/path-objects/2d.path.stroke.scale2.html [ Failure ]
imported/w3c/web-platform-tests/html/dom/idlharness.https.html [ Skip ]
Modified: trunk/Source/WebCore/ChangeLog (268298 => 268299)
--- trunk/Source/WebCore/ChangeLog 2020-10-09 23:34:11 UTC (rev 268298)
+++ trunk/Source/WebCore/ChangeLog 2020-10-09 23:39:34 UTC (rev 268299)
@@ -1,3 +1,47 @@
+2020-10-09 Wenson Hsieh <wenson_hs...@apple.com>
+
+ [GPU Process] Add additional support for painting video elements to 2D contexts
+ https://bugs.webkit.org/show_bug.cgi?id=217532
+
+ Reviewed by Tim Horton.
+
+ Add support for using `video` elements as sources for `createImageBitmap` and `createPattern` by ensuring that
+ we use display-list-backed image buffers for painting when using the GPU process for media. See below for more
+ details.
+
+ Tests: fast/canvas/canvas-createPattern-video-loading.html
+ fast/canvas/canvas-createPattern-video-modify.html
+ imported/w3c/web-platform-tests/html/canvas/element/imagebitmap/canvas-createImageBitmap-video-resize.html
+ imported/w3c/web-platform-tests/html/canvas/element/imagebitmap/createImageBitmap-drawImage.html
+ imported/w3c/web-platform-tests/html/canvas/element/imagebitmap/createImageBitmap-flipY.html
+
+ * html/HTMLVideoElement.cpp:
+ (WebCore::HTMLVideoElement::createBufferForPainting const):
+ * html/HTMLVideoElement.h:
+
+ Add a helper function to create an `ImageBuffer` compatible with this video element, for the purpose of
+ painting video frames. In particular, if GPU process for media is enabled, we'll create a display-list-backed
+ remote image buffer.
+
+ * html/ImageBitmap.cpp:
+ (WebCore::ImageBitmap::createPromise):
+
+ Call into `HTMLVideoElement::createBufferForPainting` to construct the `ImageBuffer`.
+
+ * html/canvas/CanvasRenderingContext2DBase.cpp:
+ (WebCore::CanvasRenderingContext2DBase::createPattern):
+
+ Call into `HTMLVideoElement::createBufferForPainting` to construct the `ImageBuffer`.
+
+ * platform/graphics/RenderingMode.h:
+
+ Add a "purpose" enum value for media painting.
+
+ * platform/graphics/displaylists/DisplayListImageBuffer.h:
+
+ Adjust this to avoid forcing the platform context to be created in the case where there are no display list
+ items to flush. Needed to avoid crashing when creating `CanvasPattern` with a video element.
+
2020-10-09 Zalan Bujtas <za...@apple.com>
[LFC][Integration] Enable overflow hidden.
Modified: trunk/Source/WebCore/html/HTMLVideoElement.cpp (268298 => 268299)
--- trunk/Source/WebCore/html/HTMLVideoElement.cpp 2020-10-09 23:34:11 UTC (rev 268298)
+++ trunk/Source/WebCore/html/HTMLVideoElement.cpp 2020-10-09 23:39:34 UTC (rev 268299)
@@ -37,6 +37,7 @@
#include "HTMLImageLoader.h"
#include "HTMLNames.h"
#include "HTMLParserIdioms.h"
+#include "ImageBuffer.h"
#include "Logging.h"
#include "Page.h"
#include "PictureInPictureSupport.h"
@@ -290,6 +291,13 @@
setDisplayMode(Poster);
}
+std::unique_ptr<ImageBuffer> HTMLVideoElement::createBufferForPainting(const FloatSize& size, ShouldAccelerate shouldAccelerate) const
+{
+ auto* hostWindow = document().view() && document().view()->root() ? document().view()->root()->hostWindow() : nullptr;
+ auto shouldUseDisplayList = document().settings().displayListDrawingEnabled() ? ShouldUseDisplayList::Yes : ShouldUseDisplayList::No;
+ return ImageBuffer::create(size, shouldAccelerate, shouldUseDisplayList, RenderingPurpose::MediaPainting, 1, ColorSpace::SRGB, hostWindow);
+}
+
void HTMLVideoElement::paintCurrentFrameInContext(GraphicsContext& context, const FloatRect& destRect)
{
RefPtr<MediaPlayer> player = HTMLMediaElement::player();
Modified: trunk/Source/WebCore/html/HTMLVideoElement.h (268298 => 268299)
--- trunk/Source/WebCore/html/HTMLVideoElement.h 2020-10-09 23:34:11 UTC (rev 268298)
+++ trunk/Source/WebCore/html/HTMLVideoElement.h 2020-10-09 23:39:34 UTC (rev 268299)
@@ -34,9 +34,12 @@
namespace WebCore {
class HTMLImageLoader;
+class ImageBuffer;
class RenderVideo;
class PictureInPictureObserver;
+enum class ShouldAccelerate : bool;
+
class HTMLVideoElement final : public HTMLMediaElement, public Supplementable<HTMLVideoElement> {
WTF_MAKE_ISO_ALLOCATED(HTMLVideoElement);
public:
@@ -67,6 +70,8 @@
void webkitRequestFullscreen() override;
#endif
+ std::unique_ptr<ImageBuffer> createBufferForPainting(const FloatSize&, ShouldAccelerate) const;
+
// Used by canvas to gain raw pixel access
void paintCurrentFrameInContext(GraphicsContext&, const FloatRect&);
Modified: trunk/Source/WebCore/html/ImageBitmap.cpp (268298 => 268299)
--- trunk/Source/WebCore/html/ImageBitmap.cpp 2020-10-09 23:34:11 UTC (rev 268298)
+++ trunk/Source/WebCore/html/ImageBitmap.cpp 2020-10-09 23:39:34 UTC (rev 268299)
@@ -487,8 +487,7 @@
auto sourceRectangle = maybeSourceRectangle.releaseReturnValue();
auto outputSize = outputSizeForSourceRectangle(sourceRectangle, options);
- auto bitmapData = ImageBuffer::create(FloatSize(outputSize.width(), outputSize.height()), bufferRenderingMode);
-
+ auto bitmapData = video->createBufferForPainting(outputSize, bufferRenderingMode == RenderingMode::Accelerated ? ShouldAccelerate::Yes : ShouldAccelerate::No);
if (!bitmapData) {
resolveWithBlankImageBuffer(!taintsOrigin(scriptExecutionContext.securityOrigin(), *video), WTFMove(promise));
return;
Modified: trunk/Source/WebCore/html/canvas/CanvasRenderingContext2DBase.cpp (268298 => 268299)
--- trunk/Source/WebCore/html/canvas/CanvasRenderingContext2DBase.cpp 2020-10-09 23:34:11 UTC (rev 268298)
+++ trunk/Source/WebCore/html/canvas/CanvasRenderingContext2DBase.cpp 2020-10-09 23:39:34 UTC (rev 268299)
@@ -1954,7 +1954,8 @@
return RefPtr<CanvasPattern> { CanvasPattern::create(BitmapImage::create(WTFMove(nativeImage)), repeatX, repeatY, originClean) };
#endif
- auto imageBuffer = ImageBuffer::create(size(videoElement), drawingContext() ? drawingContext()->renderingMode() : RenderingMode::Accelerated);
+ auto shouldAccelerate = !drawingContext() || drawingContext()->isAcceleratedContext() ? ShouldAccelerate::Yes : ShouldAccelerate::No;
+ auto imageBuffer = videoElement.createBufferForPainting(size(videoElement), shouldAccelerate);
if (!imageBuffer)
return nullptr;
Modified: trunk/Source/WebCore/platform/graphics/RenderingMode.h (268298 => 268299)
--- trunk/Source/WebCore/platform/graphics/RenderingMode.h 2020-10-09 23:34:11 UTC (rev 268298)
+++ trunk/Source/WebCore/platform/graphics/RenderingMode.h 2020-10-09 23:39:34 UTC (rev 268299)
@@ -29,7 +29,8 @@
enum class RenderingPurpose : uint8_t {
Unspecified,
- Canvas
+ Canvas,
+ MediaPainting
};
enum class ShouldAccelerate : bool { No, Yes };
Modified: trunk/Source/WebCore/platform/graphics/displaylists/DisplayListImageBuffer.h (268298 => 268299)
--- trunk/Source/WebCore/platform/graphics/displaylists/DisplayListImageBuffer.h 2020-10-09 23:34:11 UTC (rev 268298)
+++ trunk/Source/WebCore/platform/graphics/displaylists/DisplayListImageBuffer.h 2020-10-09 23:39:34 UTC (rev 268299)
@@ -71,7 +71,8 @@
void flushDrawingContext() override
{
- m_drawingContext.replayDisplayList(BaseConcreteImageBuffer::context());
+ if (m_drawingContext.displayList().itemCount())
+ m_drawingContext.replayDisplayList(BaseConcreteImageBuffer::context());
}
protected:
Modified: trunk/Source/WebKit/ChangeLog (268298 => 268299)
--- trunk/Source/WebKit/ChangeLog 2020-10-09 23:34:11 UTC (rev 268298)
+++ trunk/Source/WebKit/ChangeLog 2020-10-09 23:39:34 UTC (rev 268299)
@@ -1,3 +1,15 @@
+2020-10-09 Wenson Hsieh <wenson_hs...@apple.com>
+
+ [GPU Process] Add additional support for painting video elements to 2D contexts
+ https://bugs.webkit.org/show_bug.cgi?id=217532
+
+ Reviewed by Tim Horton.
+
+ See WebCore/ChangeLog for more details.
+
+ * WebProcess/WebPage/WebPage.cpp:
+ (WebKit::WebPage::shouldUseRemoteRenderingFor):
+
2020-10-09 Peng Liu <peng.l...@apple.com>
[Media in GPU Process] Enable audio routing arbitration on Mac
Modified: trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp (268298 => 268299)
--- trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp 2020-10-09 23:34:11 UTC (rev 268298)
+++ trunk/Source/WebKit/WebProcess/WebPage/WebPage.cpp 2020-10-09 23:39:34 UTC (rev 268299)
@@ -7220,6 +7220,8 @@
switch (purpose) {
case RenderingPurpose::Canvas:
return m_shouldRenderCanvasInGPUProcess;
+ case RenderingPurpose::MediaPainting:
+ return m_page->settings().useGPUProcessForMediaEnabled();
default:
break;
}
