Branch: refs/heads/main
Home: https://github.com/WebKit/WebKit
Commit: 49b049f82d0832b67368eae3cf8a8caa2d50ac2d
https://github.com/WebKit/WebKit/commit/49b049f82d0832b67368eae3cf8a8caa2d50ac2d
Author: Wenson Hsieh <[email protected]>
Date: 2026-08-03 (Mon, 03 Aug 2026)
Changed paths:
M Source/WebKit/Shared/TextExtractionToStringConversion.cpp
M Source/WebKit/Shared/TextExtractionToStringConversion.h
M Source/WebKit/UIProcess/TextExtractionCache.cpp
M Source/WebKit/UIProcess/TextExtractionCache.h
M Tools/TestWebKitAPI/Tests/WebKit/WKWebView/TextExtractionTests.mm
Log Message:
-----------
[Text Extraction] Make stale node UID resolution heuristics more lenient
https://bugs.webkit.org/show_bug.cgi?id=320808
rdar://183825688
Reviewed by Abrar Rahman Protyasha.
The heuristics added in 315981@main and 316094@main introduced a way for the
agent to interact with
stale node UIDs by storing a rolling buffer of recent text extractions
delivered to the agent; if a
target node UID issued by the agent is stale (i.e. references an old target
node from a previous
page that no longer exists), we can often still fulfill the agent's intent by
observing what the
stale node UID corresponded to on the prior text extraction, and checking
whether there's any
content from a more recent text extraction where the target element and its
surrounding context is
identical. If so, we pretend that the agent had actually issued a click
targeting the similar node
in the latest text extraction.
This can often happen when an agent tries (for example) to explore different
navigation links on a
website in search for a change password form, and tries to backtrack to another
link it's already
seen, but by referencing a node UID that no longer exists in the DOM.
The issue is that this clashes with another heuristic that avoids surfacing
links that point to the
current URL in text extractions (in order to save on tokens), as well as ARIA
attributes like
`aria-selected=true`, which break the stale node resolution heuristics even
though (semantically)
the rest of the text is a match.
To fix this, we change the stale node resolution algorithm to:
1. Include link URLs that point to the same page.
2. Ignore changes in both ARIA `selected` and `expanded` attributes.
See below for more details.
Test: TextExtractionTests.InteractionRemapsStaleNodeIdentifierWithURL
* Source/WebKit/Shared/TextExtractionToStringConversion.cpp:
(WebKit::TextExtractionAggregator::takeResults):
(WebKit::TextExtractionAggregator::addResult):
(WebKit::TextExtractionAggregator::appendToLine):
Refactor how text extraction results are accumulated, such that we build up
separate strings while
traversing the text extraction item tree (i.e. intermediate representation of
the semantic DOM);
instead of just keeping track of the extracted text per line, we separately
track the text per line
that will be reported to the WebKit client (`textForClient`) and the text that
will be destined for
the internal text extraction result cache, used to resolve stale node UIDs.
This allows us to cleanly avoid adding certain attributes (`uid`, `selected`,
`expanded`) to the
text saved in the cache, and change the way we surface other attributes (`url`).
(WebKit::isTransientAriaAttribute):
(WebKit::partsForItem):
(WebKit::addPartsForText):
(WebKit::addPartsForItem):
(WebKit::addTextRepresentationRecursive):
* Source/WebKit/Shared/TextExtractionToStringConversion.h:
* Source/WebKit/UIProcess/TextExtractionCache.cpp:
(WebKit::TextExtractionCache::contextWindowBefore):
(WebKit::TextExtractionCache::contextWindowAfter):
(WebKit::TextExtractionCache::resolve const):
* Source/WebKit/UIProcess/TextExtractionCache.h:
Also bump up the number of cached extractions from 6 → 20, to increase
potential coverage of stale
node UIDs.
* Tools/TestWebKitAPI/Tests/WebKit/WKWebView/TextExtractionTests.mm:
(TestWebKitAPI::TEST(TextExtractionTests,
InteractionRemapsStaleNodeIdentifierWithURL)):
Add an API test to cover these changes, inspired by an agent failure on Etsy
that would've been
salvaged with relaxed heuristics around stale node UID resolution here.
Canonical link: https://commits.webkit.org/318499@main
To unsubscribe from these emails, change your notification settings at
https://github.com/WebKit/WebKit/settings/notifications
