Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string
Jakub Narebski writes: > I think the problem is not with aligning, otherwise we would simply get > bad aling, and not visible corruption. The ACTUAL PROBLEM is most > probably because of concatenating strings marked as UTF-8 and strings > not marked as UTF-8. Strange things happen then in Perl, unfortunetaly. Sounds quite right---the patch needs to be retitled, if the bug is not about "measuring offset", which I think is what fooled me when I sent my response.
Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string
> One solution would be to force conversion to UTF-8 on input via "open" > pragma (e.g. "use open ':encoding(UTF-8)';"). But there is no > UTF-8-with_fallback encoding available - we would have to write one, and > install it as module (or fake it via Perl trickery). This mechanism is > almost the same to what we currently use in gitwbe. Yes, I tried using `Encode::Guess` with "open" pragma, but no luck. https://perldoc.perl.org/Encode/Guess.html I'm also afraid of "open" pragma does not work properly while using git_blame_common(). Let's say someone using non-ASCII characters in his/her name, committing non-UTF8 encoded characters. git-blame will combine them in the same line. Following is an example: $ git blame dummy | xxd : 3461 6464 3565 6331 2028 e585 90e5 b3b6 4add5ec1 (.. 0010: 20e6 96b0 2032 3031 382d 3035 2d30 3320 ... 2018-05-03 0020: 3232 3a34 383a 3432 202b 3039 3030 2031 22:48:42 +0900 1 0030: 2920 8367 8389 8343 0a ) .g...C. * e585 90e5 b3b6 20e6 96b0 : my name, encoded with UTF-8 * 8367 8389 8343 : "トライ" encoded with Shift_JIS It means I need to split each lines of git-blame output at the very beginning, then convert the first-half as UTF-8 and the second-half as Shift_JIS. Sincerely, -- Shin Kojima
Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string
Junio C Hamano writes: > Shin Kojima writes: > >> Offset positions should not be counted by byte length, but by actual >> character length. >> ... >> # escape tabs (convert tabs to spaces) >> sub untabify { >> -my $line = shift; >> +my $line = to_utf8(shift); >> >> while ((my $pos = index($line, "\t")) != -1) { >> if (my $count = (8 - ($pos % 8))) { > > Some codepaths in the resulting codeflow look even hackier than they > already are. For example, format_rem_add_lines_pair() calls > untabify() and then feeds its result to esc_html(). The first thing > done in esc_html() is to call to_utf8(). I know that to_utf8() > cheats and leaves the incoming string as-is if it is already UTF-8, > so this may be a safe conversion, but ideally we should be able to > say "function X takes non-UTF8 and works on it", "function Y takes > UTF8 and works on it", and "function Z takes non-UTF8 and gives UTF8 > data back" for each functions clearly, not "function W can take > either UTF8 or any other garbage and tries to return UTF8". The problem with handling encoding in sane way, that is encode it on input (to UTF-8), and decode on output (to plain text or HTML) is the $fallback_encoding. Gitweb assumes that everything uses UTF-8 encoding. If the source is not in UTF-8, but for example uses latin-1 encoding, then there we could stumble upon byte sequences which are not valid UTF-8. If that happens, then gitweb tries to convert it to UTF-8 using $fallback_encoding. If $fallback_encoding is single-byte encoding, like latin-1, where any byte sequence is valid, then that's all. If there is an error during conversion to UTF-8, then Unicode REPLACEMENT CHARACTER, code point U+FFFD, is used. But there are places where gitweb outputs plain text; the intention is to use source data as is - to have it as one would have in the console. Some input paths are common for plain text and HTML output; because of that problem the data is not converted to UTF-8 on input. The to_utf8() function tries to be clever, and do not convert alredy converted data. > Also, does it even "fix" the problem to use to_utf8() here in the > first place? Untabify is about aligning the character after a HT to > multiple of 8 display position, so we'd want measure display width, > which is not the same as either byte count or char count. I think the problem is not with aligning, otherwise we would simply get bad aling, and not visible corruption. The ACTUAL PROBLEM is most probably because of concatenating strings marked as UTF-8 and strings not marked as UTF-8. Strange things happen then in Perl, unfortunetaly. One solution would be to force conversion to UTF-8 on input via "open" pragma (e.g. "use open ':encoding(UTF-8)';"). But there is no UTF-8-with_fallback encoding available - we would have to write one, and install it as module (or fake it via Perl trickery). This mechanism is almost the same to what we currently use in gitwbe. Another would be to use the trick that Perl 6 uses when encountering byte sequence that is invalid UTF-8 - encode it using private plane in Unicode using UTF-8, thus achieving lossless conversion / decoding. But this also as far as I know is not available from CPAN, so we would have to implement it ourself. Best, -- Jakub Narębski
Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string
> ideally we should be able to say "function X takes non-UTF8 and > works on it", "function Y takes UTF8 and works on it", and "function > Z takes non-UTF8 and gives UTF8 data back" for each functions > clearly, not "function W can take either UTF8 or any other garbage > and tries to return UTF8". Yes, I totally agree with that. > Some codepaths in the resulting codeflow look even harder than they > already are. For example, format_rem_add_lines_pair() calls > untabify() and then feeds its result to esc_html(). Honestly, I discovered this problem exactly from format_rem_add_lines_pair(). In my environment($fallback_encoding = 'cp932'), some commitdiff shows garbled text only inside color-words portions. I added a reproduce process at the end of this message. After my investigation, I thought format_rem_add_lines_pair() tries to use split()/index()/substr()/etc against raw blob contents and that produces funny characters. These builtin functions should be used to decoded string. untabify() looks proper place for me to decode blob contents beforehand, as it definitely is not to be used for binary contests like images and compressed snapshot files. I'm sure using to_utf8() in untabify() fixes this problem. In fact, there is also another similar problem in blame function that assumes blob contents as if utf8 encoded: binmode $fd, ':utf8'; I personally consider text blob contents should be decoded as soon as possible and esc_html() should not contain to_utf8(), but the codeflow is slightly vast and I couldn't eliminate the possibility of calls from somewhere else that does not care character encodings. So yes, I would appreciate hearing your thoughts. > Also, does it even "fix" the problem to use to_utf8() here in the > first place? Untabify is about aligning the character after a HT to > multiple of 8 display position, so we'd want measure display width, > which is not the same as either byte count or char count. Following is a reproduce process: $ git --version git version 2.17.0 $ mkdir test $ cd test $ git init $ echo 'モバイル' | iconv -f UTF-8 -t Shift_JIS > dummy $ git add . $ git commit -m 'init' $ echo 'インスタント' | iconv -f UTF-8 -t Shift_JIS > dummy $ git commit -am 'change' $ git instaweb $ echo 'our $fallback_encoding = "cp932";' >> .git/gitweb/gitweb_config.perl $ w3m -dump 'http://127.0.0.1:1234/?p=.git;a=commitdiff' What I got: gitprojects / .git / commitdiff [commit ] ? search: [] [ ]re summary | shortlog | log | commit | commitdiff | tree raw | patch | inline | side by side (parent: 79e26fe) change master authorShin Kojima Wed, 2 May 2018 10:55:01 + (19:55 +0900) committer Shin Kojima Wed, 2 May 2018 10:55:01 + (19:55 +0900) dummy patch | blob | history diff --git a/dummy b/dummy index ac37f38..31fb96a 100644 (file) --- a/dummy +++ b/dummy @@ -1 +1 @@ -coイル +Cンスタント Unnamed repository; edit this file 'description' to name the repository. RSS Atom What I expected: gitprojects / .git / commitdiff [commit ] ? search: [] [ ]re summary | shortlog | log | commit | commitdiff | tree raw | patch | inline | side by side (parent: 79e26fe) change master authorShin Kojima Wed, 2 May 2018 10:55:01 + (19:55 +0900) committer Shin Kojima Wed, 2 May 2018 10:55:01 + (19:55 +0900) dummy patch | blob | history diff --git a/dummy b/dummy index ac37f38..31fb96a 100644 (file) --- a/dummy +++ b/dummy @@ -1 +1 @@ -モバイル +インスタント Unnamed repository; edit this file 'description' to name the repository. RSS Atom -- Shin Kojima
Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string
Shin Kojima writes: > Offset positions should not be counted by byte length, but by actual > character length. > ... > # escape tabs (convert tabs to spaces) > sub untabify { > - my $line = shift; > + my $line = to_utf8(shift); > > while ((my $pos = index($line, "\t")) != -1) { > if (my $count = (8 - ($pos % 8))) { Some codepaths in the resulting codeflow look even hackier than they already are. For example, format_rem_add_lines_pair() calls untabify() and then feeds its result to esc_html(). The first thing done in esc_html() is to call to_utf8(). I know that to_utf8() cheats and leaves the incoming string as-is if it is already UTF-8, so this may be a safe conversion, but ideally we should be able to say "function X takes non-UTF8 and works on it", "function Y takes UTF8 and works on it", and "function Z takes non-UTF8 and gives UTF8 data back" for each functions clearly, not "function W can take either UTF8 or any other garbage and tries to return UTF8". Also, does it even "fix" the problem to use to_utf8() here in the first place? Untabify is about aligning the character after a HT to multiple of 8 display position, so we'd want measure display width, which is not the same as either byte count or char count.