On Wed, Oct 31, 2012 at 12:13:56AM +0000, Nicholas Marriott wrote: > That is: > > - libedit has a wchar_t* buffer (el->el_line.buffer) and el_line calls > ct_encode_string to convert it to a char*. > > - ct_encode_string calls wctomb which it expects to make UTF-8 but in > fact because setlocale has not been called it outputs ASCII. > > - el_line then uses ct_enc_width which assumes UTF-8 and returns 2. So > the offset is adjusted by 2 even though only 1 byte was filled in. > > - ftp obviously isn't happy about having a position after a \0, so it > goes boom. > > The setlocale() change below will only fix the problem if LC_CTYPE or > LC_ALL is set to UTF-8. ftp still cores if pasting UTF-8 in C locale. > > I think the right fix is for libedit to use the return value of wctomb > to adjust the offset rather than assuming UTF-8 and working out the > width itself. > > Perhaps something like this (very lightly tested): > > Index: chartype.c > =================================================================== > RCS file: /cvs/src/lib/libedit/chartype.c,v > retrieving revision 1.4 > diff -u -p -r1.4 chartype.c > --- chartype.c 17 Nov 2011 20:14:24 -0000 1.4 > +++ chartype.c 31 Oct 2012 00:13:12 -0000 > @@ -44,6 +44,8 @@ > #define CT_BUFSIZ 1024 > > #ifdef WIDECHAR > +protected ssize_t ct_encode_char1(char *, size_t, Char); > + > protected void > ct_conv_buff_resize(ct_buffer_t *conv, size_t mincsize, size_t minwsize) > { > @@ -178,27 +180,25 @@ ct_decode_argv(int argc, const char *arg > protected size_t > ct_enc_width(Char c) > { > - /* UTF-8 encoding specific values */ > - if (c < 0x80) > - return 1; > - else if (c < 0x0800) > - return 2; > - else if (c < 0x10000) > - return 3; > - else if (c < 0x110000) > - return 4; > - else > - return 0; /* not a valid codepoint */ > + char s[MB_CUR_MAX]; > + > + return ct_encode_char1(s, sizeof s, c); > } > > protected ssize_t > ct_encode_char(char *dst, size_t len, Char c) > { > - ssize_t l = 0; > if (len < ct_enc_width(c)) > return -1; > - l = ct_wctomb(dst, c); > + return ct_encode_char1(dst, len, c); > +} > > +protected ssize_t > +ct_encode_char1(char *dst, size_t len, Char c) > +{ > + ssize_t l = 0; > + > + l = ct_wctomb(dst, c); > if (l < 0) { > ct_wctomb_reset; > l = 0; >
With this patch (and without the patch for ftp), the behavior of ftp is a little weird. > cd pub/Open ^ Here I write https://en.wikipedia.org/wiki/%C2%BA and press tab. I can see the character. I press backspace and the completion doesn't work. I press backspace again and tab. ftp completes to "cd pub/OpeBSD". > > On Tue, Oct 30, 2012 at 11:56:18PM +0000, Nicholas Marriott wrote: > > Hi > > > > The buffer isn't zero-terminated, it's the result of calling wctomb to > > convert the internal wchar_t* that libedit has into a char*. > > > > libedit works out the offset in el_line with ct_enc_width which rather > > foolishly makes the assumption that wctomb will convert to UTF-8, but > > ftp doesn't call setlocale so it just leaves it as ASCII. > > > > Try this: > > > > Index: main.c > > =================================================================== > > RCS file: /cvs/src/usr.bin/ftp/main.c,v > > retrieving revision 1.85 > > diff -u -p -r1.85 main.c > > --- main.c 26 Aug 2012 02:16:02 -0000 1.85 > > +++ main.c 30 Oct 2012 23:52:34 -0000 > > @@ -67,6 +67,7 @@ > > > > #include <ctype.h> > > #include <err.h> > > +#include <locale.h> > > #include <netdb.h> > > #include <pwd.h> > > #include <stdio.h> > > @@ -90,6 +91,8 @@ main(volatile int argc, char *argv[]) > > char *outfile = NULL; > > const char *errstr; > > int dumb_terminal = 0; > > + > > + setlocale(LC_CTYPE, ""); > > > > ftpport = "ftp"; > > httpport = "http"; > > > > > > > > > > > > On Tue, Oct 30, 2012 at 10:31:16PM +0100, Otto Moerbeek wrote: > > > On Tue, Oct 30, 2012 at 10:17:12PM +0100, Otto Moerbeek wrote: > > > > > > > On Tue, Oct 30, 2012 at 08:59:27PM +0100, Juan Francisco Cantero > > > > Hurtado wrote: > > > > > > > > > On Tue, Oct 30, 2012 at 09:31:58AM +0100, Otto Moerbeek wrote: > > > > > > On Mon, Oct 29, 2012 at 06:43:13PM +0100, Juan Francisco Cantero > > > > > > Hurtado wrote: > > > > > > > > > > > > > Chris Cappuccio sent me a mail saying he can't see the > > > > > > > characters, only > > > > > > > a question mark. > > > > > > > > > > > > > > I'm linking each character to their wikipedia page, so you can > > > > > > > copy-paste the character. > > > > > > > > > > > > > > On Thu, Oct 25, 2012 at 05:07:34AM +0200, Juan Francisco Cantero > > > > > > > Hurtado wrote: > > > > > > > > This afternoon I was downloading a tarball from a OpenBSD > > > > > > > > mirror. I > > > > > > > > press the key "?" and after the tab key. ftp crashed with a > > > > > > > > segfault. > > > > > > > > > > > > Please also include your environment settings. It is likely locale > > > > > > plays a role here. > > > > > > > > > > > > At least env | grep LC > > > > > > > > > > > > > > > > I've tried the bug in amd64 without locales and also with > > > > > LC_TIME="es_ES.ISO8859-1" LC_CTYPE="en_US.UTF-8". > > > > > > > > > > The i386 system was a clean installation in a virtual machine. > > > > > > > > I can now reproduce using a terminal that accepts more than just low > > > > ascii. > > > > > > > > What I see is that when complete() is called the cursor position in > > > > the EditLine struct is not what it is supposed to be, it points a > > > > couple of bytes beyond the terminating NUL while it is supposed to > > > > point to the NUL. That causes confusing in the scanner, getting the > > > > argument list count wrong. > > > > > > Ehh, the buffer is not NUL terminated, but observation still holds: > > > the cursor position is a couple of bytes further than it > > > should be. > > > > > > > > > > > The root of the problem seems to be inside the editline lib. > > > > > > > > Cc:ing nicm@, maybe he has a clue > > > > > > > > -Otto > > > > > > > > > > > > > > > > > > > > > > > > > > https://en.wikipedia.org/wiki/%C2%BA > > > > > > > > > > > > > > > > Steps for reproduce: > > > > > > > > # ftp ftp.fr.openbsd.org > > > > > > > > user and password > > > > > > > > ascii art > > > > > > > > ftp> cd pub/Open? <- Here press the tab key > > > > > > > https://en.wikipedia.org/wiki/%C2%BA > > > > > > > > segmentation fault (core dumped) ftp ftp.fr.openbsd.org > > > > > > > > > > > > > > > > It also crashes with the letter "?" and "?". > > > > > > > https://en.wikipedia.org/wiki/%C3%81 > > > > > > > https://en.wikipedia.org/wiki/%C3%91 > > > > > > > > > > > > > > > > Tested in: > > > > > > > > - A snapshot from yesterday. i386. root account. console/ksh > > > > > > > > without > > > > > > > > locales. > > > > > > > > - A snapshot from a few days ago. amd64. user. urxvt/zsh with > > > > > > > > utf8 > > > > > > > > locales. > > > > > > > > > > > > > > > > I also tested the bug in a remote session with OpenBSD 4.7 and > > > > > > > > ftp works > > > > > > > > without problems. > > > > > > > > > > > > > > > > I've updated the code of usr.bin/ftp to 2012-10-01 and > > > > > > > > 2012-01-01 and > > > > > > > > tried both versions. ftp also crashes. > > > > > > > > > > > > > > > > Backtrace: > > > > > > > > Thread 1 (process 3436): > > > > > > > > #0 memcpy (dst0=0x9d4160, src0=Variable "src0" is not > > > > > > > > available. > > > > > > > > ) at /usr/src/lib/libc/string/bcopy.c:115 > > > > > > > > #1 0x000000000040432b in complete (el=Variable "el" is not > > > > > > > > available. > > > > > > > > ) at /usr/src/usr.bin/ftp/complete.c:313 > > > > > > > > #2 0x000000000041eb84 in el_wgets (el=0x20da64800, > > > > > > > > nread=0x7f7ffffe3ebc) at read.c:612 > > > > > > > > #3 0x000000000041ef8d in el_gets (el=0x20da64800, > > > > > > > > nread=Variable "nread" is not available. > > > > > > > > ) at eln.c:78 > > > > > > > > #4 0x000000000040e55f in cmdscanner (top=Variable "top" is not > > > > > > > > available. > > > > > > > > ) at /usr/src/usr.bin/ftp/main.c:465 > > > > > > > > #5 0x000000000040eb7c in main (argc=1, argv=0x7f7ffffe4398) at > > > > > > > > /usr/src/usr.bin/ftp/main.c:369 > > > > > > > > > > > > > > > > Let me know if it's necessary more info or whatever :) > > > > > > > > > > > > > > > > Cheers. > > > > > > > > > > > > > > > -- Juan Francisco Cantero Hurtado http://juanfra.info