- Revision
- 286265
- Author
- wenson_hs...@apple.com
- Date
- 2021-11-29 14:47:33 -0800 (Mon, 29 Nov 2021)
Log Message
Add a basic heuristic for sizing text in image overlay blocks
https://bugs.webkit.org/show_bug.cgi?id=233343
rdar://85755755
Reviewed by Tim Horton.
Source/WebCore:
Address a FIXME by implementing a simple heuristic for sizing text in image overlay blocks. Each image overlay
block has a predetermined target width and height; currently, we hard-code the font size in each of these blocks
to be 80% of the target height. While this works in some cases, it also causes content in these blocks to
overflow the container if there's too much text.
Avoid this problem by finding the largest font size, such that text in each container fits within the target
bounds without overflowing vertically (we don't need to worry about horizontal overflow because horizontally
overflowing text will simply wrap to the next line).
Because of line wrapping, it's difficult to analytically compute this font size, so I'm instead using binary
search (up to a fixed number of iterations) to converge on this largest size that avoids exceeding the target
height. See below for more details.
Test: fast/images/text-recognition/image-overlay-blocks.html
* dom/ImageOverlay.cpp:
(WebCore::ImageOverlay::updateWithTextRecognitionResult):
Implement the main logic of the binary search. Establish a goal of ending up with text that fills at least 90%
of the available space in the container (with a max of just slightly over 100%, so that we stop early if we
happen to only barely exceed the target height). Then, starting with a font size that's 80% of the target
height, converge on a font size between 0% and 100% of the target height that achieves this goal. In practice,
most of the injected blocks of text will stop after a single iteration (due to the "80%-of-height" font size
being sufficient to mostly fill available vertical space).
Each FontSizeAdjustmentState here represents a single block container. At the beginning of each iteration, we
ensure that layout is up to date, and set the font size to be between the current min and max value (scaling by
target height to get font size). Using `linesBoundingBox()`, we compute the height of the rendered text relative
to its parent (which we know a-priori to be as tall as the target height). If this height is at least 90% of the
target height, we finish and remove the font size adjustment state from the vector of pending font adjustments;
otherwise, we adjust the min and max to either increase or decrease the font size as needed, and continue the
loop. We finally bail after an arbitrarily-chosen 10 iterations, if the text in the container is still too
small.
* testing/Internals.cpp:
(WebCore::Internals::installImageOverlay):
Augment this testing-only internals hook so that we can inject arbitrary image overlay block content inside
image overlays.
* testing/Internals.h:
* testing/Internals.idl:
LayoutTests:
Add a new layout test. See WebCore/ChangeLog for more details.
* fast/images/text-recognition/image-overlay-blocks-expected.html: Added.
* fast/images/text-recognition/image-overlay-blocks.html: Added.
Modified Paths
Added Paths
Diff
Modified: trunk/LayoutTests/ChangeLog (286264 => 286265)
--- trunk/LayoutTests/ChangeLog 2021-11-29 22:38:41 UTC (rev 286264)
+++ trunk/LayoutTests/ChangeLog 2021-11-29 22:47:33 UTC (rev 286265)
@@ -1,3 +1,16 @@
+2021-11-29 Wenson Hsieh <wenson_hs...@apple.com>
+
+ Add a basic heuristic for sizing text in image overlay blocks
+ https://bugs.webkit.org/show_bug.cgi?id=233343
+ rdar://85755755
+
+ Reviewed by Tim Horton.
+
+ Add a new layout test. See WebCore/ChangeLog for more details.
+
+ * fast/images/text-recognition/image-overlay-blocks-expected.html: Added.
+ * fast/images/text-recognition/image-overlay-blocks.html: Added.
+
2021-11-29 Chris Dumez <cdu...@apple.com>
REGRESSION(r283855) [GTK][WPE] imported/w3c/web-platform-tests/webaudio/the-audio-api/the-pannernode-interface/panner-equalpower.html fails
Added: trunk/LayoutTests/fast/images/text-recognition/image-overlay-blocks-expected.html (0 => 286265)
--- trunk/LayoutTests/fast/images/text-recognition/image-overlay-blocks-expected.html (rev 0)
+++ trunk/LayoutTests/fast/images/text-recognition/image-overlay-blocks-expected.html 2021-11-29 22:47:33 UTC (rev 286265)
@@ -0,0 +1,23 @@
+<!DOCTYPE html>
+<html>
+<head>
+<style>
+body, html {
+ margin: 0;
+}
+
+.cover-block-container {
+ position: absolute;
+ top: 0;
+ left: 0;
+ width: 200px;
+ height: 200px;
+ background-color: black;
+}
+</style>
+</head>
+<body>
+<img src=""
+<div class="cover-block-container"></div>
+</body>
+</html>
\ No newline at end of file
Added: trunk/LayoutTests/fast/images/text-recognition/image-overlay-blocks.html (0 => 286265)
--- trunk/LayoutTests/fast/images/text-recognition/image-overlay-blocks.html (rev 0)
+++ trunk/LayoutTests/fast/images/text-recognition/image-overlay-blocks.html 2021-11-29 22:47:33 UTC (rev 286265)
@@ -0,0 +1,39 @@
+<!DOCTYPE html>
+<html>
+<head>
+<style>
+body, html {
+ margin: 0;
+}
+
+.cover-block-container {
+ position: absolute;
+ top: 0;
+ left: 0;
+ width: 200px;
+ height: 200px;
+ background-color: black;
+}
+</style>
+</head>
+<body>
+<img src=""
+<div class="cover-block-container"></div>
+<script>
+addEventListener("load", () => {
+ let image = document.querySelector("img");
+ internals.installImageOverlay(image, [], [
+ {
+ topLeft : new DOMPointReadOnly(0.1, 0.1),
+ topRight : new DOMPointReadOnly(0.4, 0.1),
+ bottomRight : new DOMPointReadOnly(0.4, 0.4),
+ bottomLeft : new DOMPointReadOnly(0.1, 0.4),
+ text : "This is a lot of text that might overflow the overlay block unless we do something about it.",
+ }
+ ]);
+ const block = internals.shadowRoot(image).querySelector("div.image-overlay-block");
+ block.style.overflow = "visible";
+});
+</script>
+</body>
+</html>
\ No newline at end of file
Modified: trunk/Source/WebCore/ChangeLog (286264 => 286265)
--- trunk/Source/WebCore/ChangeLog 2021-11-29 22:38:41 UTC (rev 286264)
+++ trunk/Source/WebCore/ChangeLog 2021-11-29 22:47:33 UTC (rev 286265)
@@ -1,3 +1,54 @@
+2021-11-29 Wenson Hsieh <wenson_hs...@apple.com>
+
+ Add a basic heuristic for sizing text in image overlay blocks
+ https://bugs.webkit.org/show_bug.cgi?id=233343
+ rdar://85755755
+
+ Reviewed by Tim Horton.
+
+ Address a FIXME by implementing a simple heuristic for sizing text in image overlay blocks. Each image overlay
+ block has a predetermined target width and height; currently, we hard-code the font size in each of these blocks
+ to be 80% of the target height. While this works in some cases, it also causes content in these blocks to
+ overflow the container if there's too much text.
+
+ Avoid this problem by finding the largest font size, such that text in each container fits within the target
+ bounds without overflowing vertically (we don't need to worry about horizontal overflow because horizontally
+ overflowing text will simply wrap to the next line).
+
+ Because of line wrapping, it's difficult to analytically compute this font size, so I'm instead using binary
+ search (up to a fixed number of iterations) to converge on this largest size that avoids exceeding the target
+ height. See below for more details.
+
+ Test: fast/images/text-recognition/image-overlay-blocks.html
+
+ * dom/ImageOverlay.cpp:
+ (WebCore::ImageOverlay::updateWithTextRecognitionResult):
+
+ Implement the main logic of the binary search. Establish a goal of ending up with text that fills at least 90%
+ of the available space in the container (with a max of just slightly over 100%, so that we stop early if we
+ happen to only barely exceed the target height). Then, starting with a font size that's 80% of the target
+ height, converge on a font size between 0% and 100% of the target height that achieves this goal. In practice,
+ most of the injected blocks of text will stop after a single iteration (due to the "80%-of-height" font size
+ being sufficient to mostly fill available vertical space).
+
+ Each FontSizeAdjustmentState here represents a single block container. At the beginning of each iteration, we
+ ensure that layout is up to date, and set the font size to be between the current min and max value (scaling by
+ target height to get font size). Using `linesBoundingBox()`, we compute the height of the rendered text relative
+ to its parent (which we know a-priori to be as tall as the target height). If this height is at least 90% of the
+ target height, we finish and remove the font size adjustment state from the vector of pending font adjustments;
+ otherwise, we adjust the min and max to either increase or decrease the font size as needed, and continue the
+ loop. We finally bail after an arbitrarily-chosen 10 iterations, if the text in the container is still too
+ small.
+
+ * testing/Internals.cpp:
+ (WebCore::Internals::installImageOverlay):
+
+ Augment this testing-only internals hook so that we can inject arbitrary image overlay block content inside
+ image overlays.
+
+ * testing/Internals.h:
+ * testing/Internals.idl:
+
2021-11-29 Chris Dumez <cdu...@apple.com>
REGRESSION(r283855) [GTK][WPE] imported/w3c/web-platform-tests/webaudio/the-audio-api/the-pannernode-interface/panner-equalpower.html fails
Modified: trunk/Source/WebCore/dom/ImageOverlay.cpp (286264 => 286265)
--- trunk/Source/WebCore/dom/ImageOverlay.cpp 2021-11-29 22:38:41 UTC (rev 286264)
+++ trunk/Source/WebCore/dom/ImageOverlay.cpp 2021-11-29 22:47:33 UTC (rev 286265)
@@ -41,6 +41,7 @@
#include "Page.h"
#include "Quirks.h"
#include "RenderImage.h"
+#include "RenderText.h"
#include "ShadowRoot.h"
#include "SimpleRange.h"
#include "Text.h"
@@ -498,6 +499,25 @@
}
#endif // ENABLE(DATA_DETECTION)
+ constexpr float minScaleForFontSize = 0;
+ constexpr float initialScaleForFontSize = 0.8;
+ constexpr float maxScaleForFontSize = 1;
+ constexpr unsigned iterationLimit = 10;
+ constexpr float minTargetScore = 0.9;
+ constexpr float maxTargetScore = 1.02;
+
+ struct FontSizeAdjustmentState {
+ Ref<HTMLElement> container;
+ float targetHeight;
+ float scale { initialScaleForFontSize };
+ float minScale { minScaleForFontSize };
+ float maxScale { maxScaleForFontSize };
+ bool mayRequireAdjustment { true };
+ };
+
+ Vector<FontSizeAdjustmentState> elementsToAdjust;
+ elementsToAdjust.reserveInitialCapacity(result.blocks.size());
+
ASSERT(result.blocks.size() == elements.blocks.size());
for (size_t index = 0; index < result.blocks.size(); ++index) {
auto& block = result.blocks[index];
@@ -506,11 +526,55 @@
auto blockContainer = elements.blocks[index];
auto bounds = fitElementToQuad(blockContainer.get(), convertToContainerCoordinates(block.normalizedQuad));
- // FIXME: We'll need a smarter algorithm here that chooses the largest font size for the container without
- // vertically overflowing the container.
- blockContainer->setInlineStyleProperty(CSSPropertyFontSize, std::round(0.8 * bounds.size.height()), CSSUnitType::CSS_PX);
+ blockContainer->setInlineStyleProperty(CSSPropertyFontSize, initialScaleForFontSize * bounds.size.height(), CSSUnitType::CSS_PX);
+ elementsToAdjust.uncheckedAppend({ WTFMove(blockContainer), bounds.size.height() });
}
+ unsigned currentIteration = 0;
+ while (!elementsToAdjust.isEmpty()) {
+ document->updateLayoutIgnorePendingStylesheets();
+
+ for (auto& state : elementsToAdjust) {
+ RefPtr textNode = state.container->firstChild();
+ if (!is<Text>(textNode)) {
+ ASSERT_NOT_REACHED();
+ state.mayRequireAdjustment = false;
+ continue;
+ }
+
+ auto* textRenderer = downcast<Text>(*textNode).renderer();
+ if (!textRenderer) {
+ ASSERT_NOT_REACHED();
+ state.mayRequireAdjustment = false;
+ continue;
+ }
+
+ auto currentScore = textRenderer->linesBoundingBox().height() / state.targetHeight;
+ if (currentScore < minTargetScore)
+ state.minScale = state.scale;
+ else if (currentScore > maxTargetScore)
+ state.maxScale = state.scale;
+ else {
+ state.mayRequireAdjustment = false;
+ continue;
+ }
+
+ state.scale = (state.minScale + state.maxScale) / 2;
+ state.container->setInlineStyleProperty(CSSPropertyFontSize, state.targetHeight * state.scale, CSSUnitType::CSS_PX);
+ }
+
+ elementsToAdjust.removeAllMatching([](auto& state) {
+ return !state.mayRequireAdjustment;
+ });
+
+ if (++currentIteration > iterationLimit) {
+ // Fall back to the largest font size that still vertically fits within the container.
+ for (auto& state : elementsToAdjust)
+ state.container->setInlineStyleProperty(CSSPropertyFontSize, state.targetHeight * state.minScale, CSSUnitType::CSS_PX);
+ break;
+ }
+ }
+
if (RefPtr frame = document->frame())
frame->eventHandler().scheduleCursorUpdate();
Modified: trunk/Source/WebCore/testing/Internals.cpp (286264 => 286265)
--- trunk/Source/WebCore/testing/Internals.cpp 2021-11-29 22:38:41 UTC (rev 286264)
+++ trunk/Source/WebCore/testing/Internals.cpp 2021-11-29 22:47:33 UTC (rev 286265)
@@ -5753,6 +5753,7 @@
Internals::ImageOverlayLine::~ImageOverlayLine() = default;
Internals::ImageOverlayText::~ImageOverlayText() = default;
+Internals::ImageOverlayBlock::~ImageOverlayBlock() = default;
#if ENABLE(IMAGE_ANALYSIS)
@@ -5801,7 +5802,7 @@
#endif // ENABLE(IMAGE_ANALYSIS)
-void Internals::installImageOverlay(Element& element, Vector<ImageOverlayLine>&& lines)
+void Internals::installImageOverlay(Element& element, Vector<ImageOverlayLine>&& lines, Vector<ImageOverlayBlock>&& blocks)
{
if (!is<HTMLElement>(element))
return;
@@ -5814,9 +5815,12 @@
#if ENABLE(DATA_DETECTION)
, Vector<TextRecognitionDataDetector>()
#endif
- , Vector<TextRecognitionBlockData>()
+ , blocks.map([] (auto& block) {
+ return TextRecognitionBlockData { block.text, getQuad<ImageOverlayBlock>(block) };
+ })
});
#else
+ UNUSED_PARAM(blocks);
UNUSED_PARAM(lines);
#endif
}
Modified: trunk/Source/WebCore/testing/Internals.h (286264 => 286265)
--- trunk/Source/WebCore/testing/Internals.h 2021-11-29 22:38:41 UTC (rev 286264)
+++ trunk/Source/WebCore/testing/Internals.h 2021-11-29 22:47:33 UTC (rev 286265)
@@ -918,8 +918,19 @@
~ImageOverlayLine();
};
- void installImageOverlay(Element&, Vector<ImageOverlayLine>&&);
+ struct ImageOverlayBlock {
+ String text;
+ RefPtr<DOMPointReadOnly> topLeft;
+ RefPtr<DOMPointReadOnly> topRight;
+ RefPtr<DOMPointReadOnly> bottomRight;
+ RefPtr<DOMPointReadOnly> bottomLeft;
+
+ ~ImageOverlayBlock();
+ };
+
+ void installImageOverlay(Element&, Vector<ImageOverlayLine>&&, Vector<ImageOverlayBlock>&& = { });
+
#if ENABLE(IMAGE_ANALYSIS)
void requestTextRecognition(Element&, RefPtr<VoidCallback>&&);
RefPtr<Element> textRecognitionCandidate() const;
Modified: trunk/Source/WebCore/testing/Internals.idl (286264 => 286265)
--- trunk/Source/WebCore/testing/Internals.idl 2021-11-29 22:38:41 UTC (rev 286264)
+++ trunk/Source/WebCore/testing/Internals.idl 2021-11-29 22:47:33 UTC (rev 286265)
@@ -288,6 +288,17 @@
[
ExportMacro=WEBCORE_TESTSUPPORT_EXPORT,
+ JSGenerateToJSObject,
+] dictionary ImageOverlayBlock {
+ required DOMString text;
+ required DOMPointReadOnly topLeft;
+ required DOMPointReadOnly topRight;
+ required DOMPointReadOnly bottomRight;
+ required DOMPointReadOnly bottomLeft;
+};
+
+[
+ ExportMacro=WEBCORE_TESTSUPPORT_EXPORT,
LegacyNoInterfaceObject,
] interface Internals {
DOMString address(Node node);
@@ -941,7 +952,7 @@
[Conditional=IMAGE_ANALYSIS] readonly attribute Element? textRecognitionCandidate;
[Conditional=IMAGE_ANALYSIS] undefined requestTextRecognition(Element element, VoidCallback callback);
- undefined installImageOverlay(Element element, sequence<ImageOverlayLine> lines);
+ undefined installImageOverlay(Element element, sequence<ImageOverlayLine> lines, optional sequence<ImageOverlayBlock> blocks = []);
boolean usingAppleInternalSDK();
boolean usingGStreamer();
