[
https://issues.apache.org/jira/browse/OPENNLP-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristian Rickert reassigned OPENNLP-1946:
-----------------------------------------
Assignee: Kristian Rickert
> UrlCharSequenceNormalizer misses or partially removes many URLs from the WPT
> URL test suite
> -------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1946
> URL: https://issues.apache.org/jira/browse/OPENNLP-1946
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Richard Zowalla
> Assignee: Kristian Rickert
> Priority: Major
> Attachments: failures.json, report.md
>
>
> h3. Background
> UrlCharSequenceNormalizer replaces URLs and email addresses with a space. Its
> URL rule is {{https?://}} followed by {{\[\-_.?&\~;\+=/#0\-9A\-Za\-z\]\+}}.
> Its email rule is
> {{\(?<\!\[\-\+_.0\-9A\-Za\-z\]\)\[\-\+\_.0\-9A\-Za\-z\]\+@\[\-0\-9A\-Za\-z\]\+\[\-.0\-9A\-Za\-z\]\+}}.
> It is part of the default normalizer chain in LanguageDetectorFactory.
> We ran all 893 cases from the WHATWG URL test suite
> \(web\-platform\-tests/wpt@3e62dcd8d2968379e0a77fd7b7763de2304883cc,
> url/resources/urltestdata.json\) through the normalizer. The regex version
> \(2.5.12\) and the cursor\-scan version \(main, OPENNLP\-1876\) produce
> identical output for every input, so the gaps below are in the original
> design, not in the port.
> h3. Results
> ||Bucket||WPT verdict||Expected||Pass||Fail||
> |A1|valid http\(s\), written as scheme://|whole URL removed|54|114|
> |A2|valid http\(s\), unusual form \(http:foo, http:\, http:/x\)|whole URL
> removed|0|24|
> |B|invalid URL|input unchanged|123|39 fully \+ 107 partly removed|
> |C|valid URL with another scheme|not partly removed|269|9|
> |D|relative reference|input unchanged|154|0|
> h3. Findings
> _1. Valid http\(s\) URLs are cut short \(114\)._ The match ends at the first
> character outside the body set:
> \* {{:}} for a port or user:pass \(27\): {{[http://f:21/}|http://f:21/}]} \->
> {{" :21/"}}
> \* {{%}} escapes \(25\):
> {{[http://example.com/%20foo}|http://example.com/%20foo}]} \-> {{" %20foo"}}
> \* non\-ASCII / IDN hostnames and paths \(21\)
> \* ASCII punctuation such as {{" \ < > ` ' \( { \!}} \(15\)
> \* tabs, newlines and control characters \(13\)
> \* {{\[}} in IPv6 hosts \(7\): {{http://\[::1\]/}} is not matched at all
> \* {{@}} in userinfo \(6\)
> _2. Matches start inside other schemes \(no left boundary\)._
> \* {{git\+[https://github.com/foo/bar}|https://github.com/foo/bar}]} \->
> {{"git\+ "}}
> \* {{blob:[https://example.com:443/}|https://example.com:443/}]} \-> {{"blob:
> :443/"}}
> _3. The email pass removes the password and host from userinfo._
> \* {{telnet://user:[email protected]:23/}} \-> {{"telnet://user: :23/"}}
> \* the same happens for redis://, rsync:// and tag:
> _4. Invalid hosts are removed as URLs \(39\)._ Examples:
> {{[http://256.256.256.256}|http://256.256.256.256}]},
> {{[http://1.2.3.08}|http://1.2.3.08}]}, {{[http://foo.0x4}|http://foo.0x4}]},
> {{http://?}}, {{http://#}}. This is low priority, because a text normalizer
> does not need to validate hosts.
> h3. Proposal
> Changing the default output would change the features that existing language
> detector models were trained on. Therefore:
> h1. Keep {{UrlCharSequenceNormalizer.getInstance\(\)}} and the
> LanguageDetectorFactory chain byte\-for\-byte unchanged. Document the known
> limits \(findings 1\-3\) in the class Javadoc.
> h1. Add an opt\-in variant \(for example a separate instance or a builder
> flag\) that:
> #\* requires a non\-word character or start of text before the scheme
> \(finding 2\)
> #\* matches the scheme case\-insensitively
> #\* accepts a port, userinfo, percent escapes, bracketed IPv6 hosts, and
> non\-ASCII hostnames and paths in the body \(finding 1\)
> #\* stops at whitespace, control characters, and {{< > "}} as delimiters,
> which is how URLs are bounded in running text
> #\* runs the URL pass before the email pass without email\-matching inside
> URL userinfo \(finding 3\)
> h1. Add a test that runs the WPT data \(pinned commit; the data is 3\-clause
> BSD, so vendoring it needs a LICENSE/NOTICE entry, or a curated subset is
> kept instead\) against both variants: the default pins current output, the
> opt\-in variant asserts buckets A1 and C.
> Out of scope: validating hosts \(finding 4\), the unusual forms in bucket A2,
> and schemes other than http\(s\).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
