Bug#501851: zsh: Completion fails if a directory name contains '(', ')' and 'Ą '.
On Wed, 15 Oct 2008 13:34:47 +0900 Morita Sho <[EMAIL PROTECTED]> wrote: > Thanks for detailed explanation! But I'm still confusing. Even with the old > syntax, the completion Ą/[TAB] won't work. You're right. My explanation wasn't correct; it doesn't matter if you use the m:{[lower]}={[upper]} syntax or the older one, so I've restored the advice that the newer one is preferable. > BTW, to find out why replacing _path_files with the old one solves the problem > for me, I put a debug output into _path_files as below to watch the value of > $tmp1 before the $~tmp1 expansion. > print "DEBUG: LINENO=$LINENO tmp1=$tmp1" > /dev/stderr > tmp1=( $~tmp1 ) 2> /dev/null > > When I type Ą/[TAB], I got this debug output. > DEBUG: LINENO=400 tmp1=ă�*(-/) > DEBUG: LINENO=400 tmp1=ă�*(-/) Thanks, this tells me that the compfiles utility isn't handling Meta characters properly when performing matching. Luckily this seems to be a local problem within that command that I think the patch addresses. Index: Doc/Zsh/compwid.yo === RCS file: /cvsroot/zsh/zsh/Doc/Zsh/compwid.yo,v retrieving revision 1.44 diff -u -r1.44 compwid.yo --- Doc/Zsh/compwid.yo 14 Oct 2008 12:59:04 - 1.44 +++ Doc/Zsh/compwid.yo 18 Oct 2008 18:44:08 - @@ -942,16 +942,14 @@ completion you can use `tt(m:{[:lower:]}={[:upper:]})'. Although the matching system does not yet handle multibyte characters, this is likely to be a future extension, at which point this syntax will handle -arbitrary alphabets; until then it is safer to use the older syntax -that only handles ASCII characters, `tt(m:{a-z}={A-Z}) as this does -not have side effects in the case of multibyte characters. - -In other cases `tt([:)var(name)tt(:])' forms are allowed. If the two forms -on the left and right are the same, the characters must match exactly. In -remaining cases, the corresponding tests are applied to both characters, -but they are not otherwise constrained; any matching character in one set -goes with any matching character in the other set: this is equivalent to -the behaviour of ordinary character classes. +arbitrary alphabets; hence this form, rather than the use of explicit +ranges, is the recommended form. In other cases +`tt([:)var(name)tt(:])' forms are allowed. If the two forms on the left +and right are the same, the characters must match exactly. In remaining +cases, the corresponding tests are applied to both characters, but they +are not otherwise constrained; any matching character in one set goes +with any matching character in the other set: this is equivalent to the +behaviour of ordinary character classes. The pattern var(tpat) may also be one or two stars, `tt(*)' or `tt(**)'. This means that the pattern on the command line can match Index: Src/Zle/computil.c === RCS file: /cvsroot/zsh/zsh/Src/Zle/computil.c,v retrieving revision 1.110 diff -u -r1.110 computil.c --- Src/Zle/computil.c 14 Oct 2008 22:09:11 - 1.110 +++ Src/Zle/computil.c 18 Oct 2008 18:44:09 - @@ -4024,7 +4024,7 @@ * management is difficult. */ for (;;) { - for (mp = ms; *add; add++, mp++) { + for (mp = ms; *add; ) { if (!(m = *mp)) { /* * No matcher, so just match the character @@ -4034,13 +4034,13 @@ * metacharacter? */ if (ret) { - if (imeta(*add)) { + if (*add == Meta) { *p++ = Meta; - *p++ = *add ^ 32; + *p++ = add[1]; } else *p++ = *add; } else - len += imeta(*add) ? 2 : 1; + len += (*add == Meta) ? 2 : 1; } else if (m->flags & CMF_RIGHT) { /* * Right-anchored: match anything followed @@ -4049,15 +4049,16 @@ if (ret) { *p++ = '*'; /* TODO: quote again? */ - if (imeta(*add)) { + if (*add == Meta) { *p++ = Meta; - *p++ = *add ^ 32; + *p++ = add[1]; } else *p++ = *add; } else - len += imeta(*add) ? 3 : 2; + len += (*add == Meta) ? 3 : 2; } else { /* The usual set of matcher possibilities. */ + int chr = (*add == Meta) ? add[1] ^ 32 : *add; int ind; if (m->line->tp == CPAT_EQUIV && m->word->tp == CPAT_EQUIV) { @@ -4072,21 +4073,21 @@ */ if (ret) { *p++ = '['; - if (imeta(*add)) { + if (*add == Meta)
Bug#501851: zsh: Completion fails if a directory name contains '(', ')' and 'Ą '.
On 10/19/2008 03:48 AM, Peter Stephenson wrote: > Thanks, this tells me that the compfiles utility isn't handling Meta > characters properly when performing matching. Luckily this seems to be > a local problem within that command that I think the patch addresses. I can confirm that the completion problem has been solved on cvs head. Thanks for your work! -- Morita Sho <[EMAIL PROTECTED]> -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#501851: zsh: Completion fails if a directory name contains '(', ')' and 'Ą '.
Package: zsh Version: 4.3.6-7 Severity: normal Hi, Completion fails if a directory name contains '(', ')' and 'Ą '. 'Ą ' is U+0104 in unicode. Following is a small example to reproduce the problem. % mkdir '()Ą' % touch '()Ą'/foo % ls '()Ą'/[TAB] => I expected it completes "foo", but it completes nothing. If I renamed the directory name slightly, e.g. remove a parenthesis or change 'Ą' to 'Ā ' (U+0100), it correctly completes "foo". Regards, -- System Information: Debian Release: lenny/sid APT prefers unstable APT policy: (500, 'unstable') Architecture: i386 (i686) Kernel: Linux 2.6.26-1-686 (SMP w/1 CPU core) Locale: LANG=ja_JP.UTF-8, LC_CTYPE=ja_JP.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/bash Versions of packages zsh depends on: ii libc6 2.7-14 GNU C Library: Shared libraries ii libcap2 2.11-2 support for getting/setting POSIX. ii libncursesw5 5.6+20080925-1 shared libraries for terminal hand Versions of packages zsh recommends: ii libpcre3 7.8-2 Perl 5 Compatible Regular Expressi Versions of packages zsh suggests: pn zsh-doc(no description available) -- no debconf information -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#501851: zsh: Completion fails if a directory name contains '(', ')' and 'Ą '.
On Sat, Oct 11, 2008 at 08:06:11AM +0900, Morita Sho wrote: > Completion fails if a directory name contains '(', ')' and 'Ą '. > 'Ą ' is U+0104 in unicode. > > Following is a small example to reproduce the problem. > > % mkdir '()Ą' > % touch '()Ą'/foo > % ls '()Ą'/[TAB] > => I expected it completes "foo", but it completes nothing. > > > If I renamed the directory name slightly, e.g. remove a parenthesis or > change 'Ą' to 'Ā ' (U+0100), it correctly completes "foo". I get different but also unuseful behavior trying to reproduce that. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#501851: zsh: Completion fails if a directory name contains '(', ')' and 'Ą '.
On Sat, Oct 11, 2008 at 01:17, Clint Adams <[EMAIL PROTECTED]> wrote: >> => I expected it completes "foo", but it completes nothing. Confirmed for 4.3.6 on Debian unstable. Richard -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#501851: zsh: Completion fails if a directory name contains '(', ')' and 'Ą '.
Thanks for your quick reply. I found that the difference between '()Ą'/[TAB] and '()Ā'/[TAB] is at line 372 of /usr/share/zsh/functions/Completion/Unix/_path_files. tmp1=( $~tmp1 ) 2> /dev/null When I typed '()Ą'/[TAB], tmp1 will be '\(\)Ą/*', and $~tmp1 is expanded to nothing. When I typed '()Ā'/[TAB], tmp1 will be '\(\)Ā/*', and $~tmp1 is expanded to ()Ā/foo. $~tmp1 expansion seems not working correctly for a pattern '\(\)Ą/*'. % ls -R .: ()Ā ()Ą ./()Ā: foo ./()Ą: foo % tmp1='\(\)Ą/*'; print $~tmp1; zsh: no matches found: \(\)Ą/* % tmp1='\(\)Ā/*'; print $~tmp1; ()Ā/foo But `print \(\)Ą/*` is correctly expanded to ()Ą/foo. % print \(\)Ą/* ()Ą/foo Regards, -- Morita Sho <[EMAIL PROTECTED]> -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#501851: zsh: Completion fails if a directory name contains '(', ')' and 'Ą '.
On Sat, 11 Oct 2008 22:47:52 +0900 Morita Sho <[EMAIL PROTECTED]> wrote: > $~tmp1 expansion seems not working correctly for a pattern '\(\)Ą/*'. Thanks for the clear analysis. The pattern gets divided up into chunks because of the backslashed characters, but we don't report that it wasn't a pure string on that basis, so the directory trise to match the wrong string. This was triggered in this case because of the character in a range that zsh has to quote internally to avoid clashing with tokens, which is why it occurred with some forms of accent but not others. I could optimise this better but this is a simple change for now. Index: Src/pattern.c === RCS file: /cvsroot/zsh/zsh/Src/pattern.c,v retrieving revision 1.47 diff -u -r1.47 pattern.c --- Src/pattern.c 8 Jun 2008 17:53:55 - 1.47 +++ Src/pattern.c 11 Oct 2008 21:44:35 - @@ -1446,7 +1446,13 @@ * Marker for restoring a backslash in output: * does not match a character. */ - return patcomppiece(flagp); + next = patcomppiece(flagp); + /* +* Can't match a pure string since we need to do this +* as multiple chunks. +*/ + *flagp &= ~P_PURESTR; + return next; break; #ifdef DEBUG default: Index: Test/D07multibyte.ztst === RCS file: /cvsroot/zsh/zsh/Test/D07multibyte.ztst,v retrieving revision 1.25 diff -u -r1.25 D07multibyte.ztst --- Test/D07multibyte.ztst 26 Mar 2008 15:26:20 - 1.25 +++ Test/D07multibyte.ztst 11 Oct 2008 21:44:35 - @@ -395,3 +395,14 @@ >OK >OK >OK + + mkdir glob + mkdir glob/'()Ą' glob/'()Ā' + mkdir glob/'()Ą'/foo glob/'()Ā'/bar + tmp1=('glob/\(\)Ą/*') + print $~tmp1 + tmp1=('glob/\(\)Ā/*') + print $~tmp1 +0:Backslashes and metafied characters in patterns +>glob/()Ą/foo +>glob/()Ā/bar -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#501851: zsh: Completion fails if a directory name contains '(', ')' and 'Ą '.
On 10/12/2008 06:52 AM, Peter Stephenson wrote: > The pattern gets divided up into chunks because of the backslashed > characters, but we don't report that it wasn't a pure string on that > basis, so the directory trise to match the wrong string. This was triggered > in this case because of the character in a range that zsh has to quote > internally to avoid clashing with tokens, which is why it occurred with > some forms of accent but not others. > > I could optimise this better but this is a simple change for now. Thanks for your work. I have checkouted the latest source code from cvs head. I can see $~tmp1 expansion works correctly, but '()Ą'/[TAB] completion still not works. % mkdir '()Ą' % touch '()Ą'/foo % ls '()Ą'/[TAB] => Nothing completes. % tmp1='\(\)Ą/*' % print $~tmp1 ()Ą/foo Additionally, Ą/[TAB] completion doesn't work on cvs head. (It was working on zsh 4.3.6) % mkdir Ą % touch Ą/foo % ls Ą/[TAB] => Nothing completes. Replacing _path_files with previous one (zsh 4.3.6) seems to solve the problem. I'm not sure, but it might be a regression in _path_files. Regards, -- Morita Sho <[EMAIL PROTECTED]> -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#501851: zsh: Completion fails if a directory name contains '(', ')' and 'Ą '.
On Mon, Oct 13, 2008 at 03:32, Morita Sho <[EMAIL PROTECTED]> wrote: > Replacing _path_files with previous one (zsh 4.3.6) seems to solve the > problem. > I'm not sure, but it might be a regression in _path_files. Just to be sure, you are talking about the second part here, not both ones, correct? Richard -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#501851: zsh: Completion fails if a directory name contains '(', ')' and 'Ą '.
On Mon, Oct 13, 2008 at 11:30, Peter Stephenson <[EMAIL PROTECTED]> wrote: > Please send the output from running "^X?" if there no obvious difference. Whoah, neat! I 'fear' I will never stop being amazed at what the ZSH devs think of.. :) Richard -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#501851: zsh: Completion fails if a directory name contains '(', ')' and 'Ą '.
On Mon, 13 Oct 2008 10:32:05 +0900 Morita Sho <[EMAIL PROTECTED]> wrote: > Additionally, Ą/[TAB] completion doesn't work on cvs head. (It was working on > zsh 4.3.6) > > % mkdir Ą > % touch Ą/foo > % ls Ą/[TAB] >=> Nothing completes. This works fine for me (as does the version with "()" after the previous fix). Does the same still happen after "zsh -f; autoload -U compinit; compinit"? Is the character the composed version, i.e. Unicode 0x0104, which is 0xc4 0x84 in UTF-8, or could it be some decomposed version with combining characters? Please send the output from running "^X?" if there no obvious difference. -- Peter Stephenson <[EMAIL PROTECTED]> Software Engineer CSR PLC, Churchill House, Cambridge Business Park, Cowley Road Cambridge, CB4 0WZ, UK Tel: +44 (0)1223 692070 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#501851: zsh: Completion fails if a directory name contains '(', ')' and 'Ą '.
On 10/13/2008 03:25 PM, Richard Hartmann wrote: > Just to be sure, you are talking about the second part here, not both ones, > correct? Sorry for ambiguous. I'm talking about both ones. Replacing _path_files solves both '()Ą'/[TAB] and Ą/[TAB] problems. On 10/13/2008 06:30 PM, Peter Stephenson wrote: > This works fine for me (as does the version with "()" after the previous > fix). Does the same still happen after "zsh -f; autoload -U compinit; > compinit"? Is the character the composed version, i.e. Unicode 0x0104, > which is 0xc4 0x84 in UTF-8, or could it be some decomposed version with > combining characters? > > Please send the output from running "^X?" if there no obvious difference. Thanks! The problem gone after "zsh -f; autoload -U compinit; compinit". The directory 'Ą' is exactly 0xc4 0x84. % ls Ą % ls | hexdump -C c4 84 0a |...| I removed and reconstructed ~/.zshrc to find out what settings in my ~/.zshrc break the completion. The completion Ą/[TAB] works fine with the following .zshrc. % cat ~/.zshrc # Lines configured by zsh-newuser-install HISTFILE=~/.histfile HISTSIZE=1000 SAVEHIST=1000 bindkey -e # End of lines configured by zsh-newuser-install # The following lines were added by compinstall zstyle :compinstall filename '/home/qw/.zshrc' autoload -Uz compinit compinit # End of lines added by compinstall However, if I run `compinstall` and enable "Case-insensitive completion (lowercase matches uppercase)" from "2. Matching control" menu, the completion Ą/[TAB] stop to working. I see the following line were added to ~/.zshrc. zstyle ':completion:*' matcher-list 'm:{[:lower:]}={[:upper:]}' After removing the above line, the completion Ą/[TAB] works again. -- Morita Sho <[EMAIL PROTECTED]> -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#501851: zsh: Completion fails if a directory name contains '(', ')' and 'Ą '.
On Tue, 14 Oct 2008 10:46:29 +0900 Morita Sho <[EMAIL PROTECTED]> wrote: > zstyle ':completion:*' matcher-list 'm:{[:lower:]}={[:upper:]}' > > After removing the above line, the completion Ą/[TAB] works again. Thanks for narrowing it down... I'm aware that that syntax still needs some work on it, it's only half-converted to supporting non-ASCII characters, so it's probably not surprising this sort of thing happens. The problem is the way characters are stored is quite specific to a single-byte representation; I've made it so that it can be extended to multibyte/wide characters, but it hasn't been. I'm not going to get it properly converted before the next release since it's a big job (I started it once back in June and got bogged down). When I get a chance, I'll look and see if there's any thing obvious I can do with Meta-encoded characters that will work round the problem for now. Until then you should be able to get away with the old syntax, zstyle ':completion:*' matcher-list 'm:{a-z}={A-Z}' since before I finish the job it won't actually handle non-ASCII character conversions properly anyway. I will document my way round this as follows... Index: Doc/Zsh/compwid.yo === RCS file: /cvsroot/zsh/zsh/Doc/Zsh/compwid.yo,v retrieving revision 1.43 diff -u -r1.43 compwid.yo --- Doc/Zsh/compwid.yo 14 Jul 2008 17:39:26 - 1.43 +++ Doc/Zsh/compwid.yo 14 Oct 2008 12:01:54 - @@ -942,14 +942,16 @@ completion you can use `tt(m:{[:lower:]}={[:upper:]})'. Although the matching system does not yet handle multibyte characters, this is likely to be a future extension, at which point this syntax will handle -arbitrary alphabets; hence this form, rather than the use of explicit -ranges, is the recommended form. In other cases -`tt([:)var(name)tt(:])' forms are allowed. If the two forms on the left -and right are the same, the characters must match exactly. In remaining -cases, the corresponding tests are applied to both characters, but they -are not otherwise constrained; any matching character in one set goes -with any matching character in the other set: this is equivalent to the -behaviour of ordinary character classes. +arbitrary alphabets; until then it is safer to use the older syntax +that only handles ASCII characters, `tt(m:{a-z}={A-Z}) as this does +not have side effects in the case of multibyte characters. + +In other cases `tt([:)var(name)tt(:])' forms are allowed. If the two forms +on the left and right are the same, the characters must match exactly. In +remaining cases, the corresponding tests are applied to both characters, +but they are not otherwise constrained; any matching character in one set +goes with any matching character in the other set: this is equivalent to +the behaviour of ordinary character classes. The pattern var(tpat) may also be one or two stars, `tt(*)' or `tt(**)'. This means that the pattern on the command line can match -- Peter Stephenson <[EMAIL PROTECTED]> Software Engineer CSR PLC, Churchill House, Cambridge Business Park, Cowley Road Cambridge, CB4 0WZ, UK Tel: +44 (0)1223 692070 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#501851: zsh: Completion fails if a directory name contains '(', ')' and 'Ą '.
On 10/14/2008 09:04 PM, Peter Stephenson wrote: > Until then you should be able to get away with the old syntax, > > zstyle ':completion:*' matcher-list 'm:{a-z}={A-Z}' > > since before I finish the job it won't actually handle non-ASCII character > conversions properly anyway. Thanks for detailed explanation! But I'm still confusing. Even with the old syntax, the completion Ą/[TAB] won't work. As far as I understand, the following configuration should fix the completion problem, but it won't. % cat ~/.zshrc # Lines configured by zsh-newuser-install HISTFILE=~/.histfile HISTSIZE=1000 SAVEHIST=1000 bindkey -e # End of lines configured by zsh-newuser-install # The following lines were added by compinstall #zstyle ':completion:*' matcher-list 'm:{[:lower:]}={[:upper:]}' zstyle ':completion:*' matcher-list 'm:{a-z}={A-Z}' zstyle :compinstall filename '/home/qw/.zshrc' autoload -Uz compinit compinit # End of lines added by compinstall BTW, to find out why replacing _path_files with the old one solves the problem for me, I put a debug output into _path_files as below to watch the value of $tmp1 before the $~tmp1 expansion. print "DEBUG: LINENO=$LINENO tmp1=$tmp1" > /dev/stderr tmp1=( $~tmp1 ) 2> /dev/null When I type Ą/[TAB], I got this debug output. DEBUG: LINENO=400 tmp1=ă�*(-/) DEBUG: LINENO=400 tmp1=ă�*(-/) Afterward, I replaced _path_files with the old one (which is from zsh 4.3.6), and put the same debug output, then I got this debug output. DEBUG: LINENO=372 tmp1=ă�*(-/) DEBUG: LINENO=372 tmp1=Ą/* It seems that tmp1 has a invalid (meta-encoded?) character. In _path_files which is from zsh 4.3.6, they seems to retry with a normal (non-meta-encoded?) character. Unfortunately, I couldn't understand why they are differ -- Morita Sho <[EMAIL PROTECTED]> -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]