Hello friends.
Let me take you through a bug, my current work and the little dilemma I'm
facing in regards to how to "canonicalize" host names in URLs! I'll end the
mail with a question about a possible solution I've thought of.
# Not parsing percent-encoded host names in URLs
$ curl https://%63url.se/
curl: (6) Could not resolve host: %63url.se
instead of:
$ curl https://%63url.se/
[content from https://curl.se]
Issue: https://github.com/curl/curl/issues/7830
PR: https://github.com/curl/curl/pull/7834
## Obvious first take
Make sure that the URL parser **decodes** percent-encoded host names. %41
becomes `A` etc.
The parser rejects "control codes" while decoding. %00, %0a and %0d makes the
host name illegal.
## Canonical host name
The URL API can also *extract* the full URL so it needs to be able to reverse
the process and here begins the challenges.
My first simplistic (or maybe *naive*) approach works like this:
Setting `https://%63url.se/` is extracted again as `https://curl.se/` but
setting `https://%c0.se/` is extracted as `https://%c0.se/` (since anything
non-ASCII is not "URL compliant").
## IDN input
Enter IDN. Internation Domain Names. They are specified outside of the
regular URL spec (RFC 3986) and they are specified using non-ASCII byte
codes.
Example name: `räksmörgås.se` (clients puny-encode this name to
`xn--rksmrgs-5wao1o.se` for DNS etc).
Since this host/URL uses non-ASCII letters, the naive approch mentioned above
would then, when the URL API is used to extract this again, use a sequence of
percent-encoded UTF-8 `r%C3%A4ksm%C3%B6rg%C3A5s.se`.
It would **not** extract back to `räksmörgås.se`, which probably is what a
user will expect.
Next-level complication: mix in percent-encoding to the IDN name:
`r%c3%a4ksmörgås.se`
The two percent-encoded bytes is UTF-8 sequence for `ä`, which makes this
host name work the same way.
## IDN output
How do we know how to encode the host name when the user wants to extract it?
Alternatives I can think of:
### A) Don't
Store the originally provided name and use that for retrieval as well. This
is bad as then the same URL with differently encoded host names will appear
as two different ones. Users probably will not expect nor appreciate this.
### B) Always
Always percent-encode (this is what the PR currently does). It makes the host
name canonical and it still works IDN wise, but the retrieved URL is ugly and
user hostile.
### C) Puny-encode
Return the **puny-encoded** version of the name if it was an IDN name,
otherwise percent-encode. Makes the host name canonical, it still works IDN
wise, but the retrieved URL is ugly and user hostile. Just possibly a little
less hostile than version B. An upside could be that a puny-code version of
the host name works even with clients that don't speak IDN. This method then
works differently if libcurl was built with or without IDN support.
### D) Heuristics
If the host name was a valid IDN name, then return that name without
encoding, otherwise perecent-encode. This makes `r%c3%a4ksmörgås.se` as input
generate `räksmörgås.se` as output. This method then works differently if
libcurl was built with or without IDN support.
Can we make version (D) work and would that be preferred?
--
/ daniel.haxx.se
| Commercial curl support up to 24x7 is available!
| Private help, bug fixes, support, ports, new features
| https://curl.se/support.html--
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
