Richard Zowalla created OPENNLP-1946:
----------------------------------------
Summary: UrlCharSequenceNormalizer misses or partially removes
many URLs from the WPT URL test suite
Key: OPENNLP-1946
URL: https://issues.apache.org/jira/browse/OPENNLP-1946
Project: OpenNLP
Issue Type: Improvement
Reporter: Richard Zowalla
h3. Background
UrlCharSequenceNormalizer replaces URLs and email addresses with a space. Its
URL rule is {{https?://}} followed by {{\[\-_.?&\~;\+=/#0\-9A\-Za\-z\]\+}}. Its
email rule is
{{\(?<\!\[\-\+_.0\-9A\-Za\-z\]\)\[\-\+\_.0\-9A\-Za\-z\]\+@\[\-0\-9A\-Za\-z\]\+\[\-.0\-9A\-Za\-z\]\+}}.
It is part of the default normalizer chain in LanguageDetectorFactory.
We ran all 893 cases from the WHATWG URL test suite
\(web\-platform\-tests/wpt@3e62dcd8d2968379e0a77fd7b7763de2304883cc,
url/resources/urltestdata.json\) through the normalizer. The regex version
\(2.5.12\) and the cursor\-scan version \(main, OPENNLP\-1876\) produce
identical output for every input, so the gaps below are in the original design,
not in the port.
h3. Results
||Bucket||WPT verdict||Expected||Pass||Fail||
|A1|valid http\(s\), written as scheme://|whole URL removed|54|114|
|A2|valid http\(s\), unusual form \(http:foo, http:\, http:/x\)|whole URL
removed|0|24|
|B|invalid URL|input unchanged|123|39 fully \+ 107 partly removed|
|C|valid URL with another scheme|not partly removed|269|9|
|D|relative reference|input unchanged|154|0|
h3. Findings
_1. Valid http\(s\) URLs are cut short \(114\)._ The match ends at the first
character outside the body set:
\* {{:}} for a port or user:pass \(27\): {{[http://f:21/}|http://f:21/}]} \->
{{" :21/"}}
\* {{%}} escapes \(25\):
{{[http://example.com/%20foo}|http://example.com/%20foo}]} \-> {{" %20foo"}}
\* non\-ASCII / IDN hostnames and paths \(21\)
\* ASCII punctuation such as {{" \ < > ` ' \( { \!}} \(15\)
\* tabs, newlines and control characters \(13\)
\* {{\[}} in IPv6 hosts \(7\): {{http://\[::1\]/}} is not matched at all
\* {{@}} in userinfo \(6\)
_2. Matches start inside other schemes \(no left boundary\)._
\* {{git\+[https://github.com/foo/bar}|https://github.com/foo/bar}]} \->
{{"git\+ "}}
\* {{blob:[https://example.com:443/}|https://example.com:443/}]} \-> {{"blob:
:443/"}}
_3. The email pass removes the password and host from userinfo._
\* {{telnet://user:[email protected]:23/}} \-> {{"telnet://user: :23/"}}
\* the same happens for redis://, rsync:// and tag:
_4. Invalid hosts are removed as URLs \(39\)._ Examples:
{{[http://256.256.256.256}|http://256.256.256.256}]},
{{[http://1.2.3.08}|http://1.2.3.08}]}, {{[http://foo.0x4}|http://foo.0x4}]},
{{http://?}}, {{http://#}}. This is low priority, because a text normalizer
does not need to validate hosts.
h3. Proposal
Changing the default output would change the features that existing language
detector models were trained on. Therefore:
h1. Keep {{UrlCharSequenceNormalizer.getInstance\(\)}} and the
LanguageDetectorFactory chain byte\-for\-byte unchanged. Document the known
limits \(findings 1\-3\) in the class Javadoc.
h1. Add an opt\-in variant \(for example a separate instance or a builder
flag\) that:
#\* requires a non\-word character or start of text before the scheme \(finding
2\)
#\* matches the scheme case\-insensitively
#\* accepts a port, userinfo, percent escapes, bracketed IPv6 hosts, and
non\-ASCII hostnames and paths in the body \(finding 1\)
#\* stops at whitespace, control characters, and {{< > "}} as delimiters, which
is how URLs are bounded in running text
#\* runs the URL pass before the email pass without email\-matching inside URL
userinfo \(finding 3\)
h1. Add a test that runs the WPT data \(pinned commit; the data is 3\-clause
BSD, so vendoring it needs a LICENSE/NOTICE entry, or a curated subset is kept
instead\) against both variants: the default pins current output, the opt\-in
variant asserts buckets A1 and C.
Out of scope: validating hosts \(finding 4\), the unusual forms in bucket A2,
and schemes other than http\(s\).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
