Re: [patch] file encoding detection doesn't work with process substitution
Ken Takata wrote: > 2016/8/9 Tue 5:27:12 UTC+9 Bram Moolenaar wrote: > > Ken Takata wrote: > > > > > 2016/8/6 Sat 22:30:05 UTC+9 Bram Moolenaar wrote: > > > > Ken Takata wrote: > > > > > > > > > When process substitution (e.g. "<(cmd)") is used on bash (or zsh), > > > > > file > > > > > encoding detection doesn't work, and an empty buffer is opened. > > > > > E.g.: > > > > > > > > > > $ vim -u NONE --cmd "set fencs=cp932,utf-8" <(echo テスト) > > > > > # Assuming the system encoding is utf-8. > > > > > > > > > > Vim will be executed with the command line like: > > > > > > > > > > $ vim -u NONE --cmd "set fencs=cp932,utf-8" /dev/fd/63 > > > > > > > > > > "/dev/fd/63" is a fifo and it is not seekable. > > > > > The echoed string "テスト" (utf-8) cannot be converted from cp932 to > > > > > utf-8, so > > > > > Vim retries with utf-8, but it fails because the input is not > > > > > seekable. > > > > > > > > > > However reading from stdin can detect the encoding. E.g.: > > > > > > > > > > $ echo テスト | vim -u NONE --cmd "set fencs=cp932,utf-8" - > > > > > > > > > > I think file encoding detection should also work with process > > > > > substitution. > > > > > Attached patch fixes the problem. > > > > > > > > Thanks. It would be good to have a test for this. The encoding > > > > detection is important, but I don't think there is a test for it. > > > > > > > > Reading from stdin and fifo require starting Vim. Test49 does this, it > > > > just uses ../vim for that. That doesn't work on Windows though. But > > > > the fifo won't work on Windows either. > > > > > > I added a test for this. I also wanted to add a test for stdin on Windows, > > > but Windows' echo command cannot write Unicode characters to a pipe. So > > > the > > > test is disabled on Windows. > > > > Thanks. It would be better to use "cat" instead of "echo". Does this > > also exist on MS-Windows? Or perhaps we can check if it exists. > > Then we can also use multiple lines in the test. > > OK, "type" can be used on Windows. > Please check the attached patch. Thanks. Glad you could make it work. I verified that without your patch the utf8 fifo test fails. -- Normal people believe that if it ain't broke, don't fix it. Engineers believe that if it ain't broke, it doesn't have enough features yet. (Scott Adams - The Dilbert principle) /// Bram Moolenaar -- b...@moolenaar.net -- http://www.Moolenaar.net \\\ ///sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\ \\\ an exciting new programming language -- http://www.Zimbu.org/// \\\help me help AIDS victims -- http://ICCF-Holland.org/// -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [patch] file encoding detection doesn't work with process substitution
Hi, 2016/8/9 Tue 5:27:12 UTC+9 Bram Moolenaar wrote: > Ken Takata wrote: > > > 2016/8/6 Sat 22:30:05 UTC+9 Bram Moolenaar wrote: > > > Ken Takata wrote: > > > > > > > When process substitution (e.g. "<(cmd)") is used on bash (or zsh), file > > > > encoding detection doesn't work, and an empty buffer is opened. > > > > E.g.: > > > > > > > > $ vim -u NONE --cmd "set fencs=cp932,utf-8" <(echo テスト) > > > > # Assuming the system encoding is utf-8. > > > > > > > > Vim will be executed with the command line like: > > > > > > > > $ vim -u NONE --cmd "set fencs=cp932,utf-8" /dev/fd/63 > > > > > > > > "/dev/fd/63" is a fifo and it is not seekable. > > > > The echoed string "テスト" (utf-8) cannot be converted from cp932 to > > > > utf-8, so > > > > Vim retries with utf-8, but it fails because the input is not seekable. > > > > > > > > However reading from stdin can detect the encoding. E.g.: > > > > > > > > $ echo テスト | vim -u NONE --cmd "set fencs=cp932,utf-8" - > > > > > > > > I think file encoding detection should also work with process > > > > substitution. > > > > Attached patch fixes the problem. > > > > > > Thanks. It would be good to have a test for this. The encoding > > > detection is important, but I don't think there is a test for it. > > > > > > Reading from stdin and fifo require starting Vim. Test49 does this, it > > > just uses ../vim for that. That doesn't work on Windows though. But > > > the fifo won't work on Windows either. > > > > I added a test for this. I also wanted to add a test for stdin on Windows, > > but Windows' echo command cannot write Unicode characters to a pipe. So the > > test is disabled on Windows. > > Thanks. It would be better to use "cat" instead of "echo". Does this > also exist on MS-Windows? Or perhaps we can check if it exists. > Then we can also use multiple lines in the test. OK, "type" can be used on Windows. Please check the attached patch. Regards, Ken Takata -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. # HG changeset patch # Parent f574310bd638067986318c3d8f0ecd77dae84748 diff --git a/src/Makefile b/src/Makefile --- a/src/Makefile +++ b/src/Makefile @@ -2114,6 +2114,7 @@ test_arglist \ test_signs \ test_sort \ test_startup \ + test_startup_utf8 \ test_stat \ test_statusline \ test_syn_attr \ diff --git a/src/buffer.c b/src/buffer.c --- a/src/buffer.c +++ b/src/buffer.c @@ -70,6 +70,64 @@ static char *e_auabort = N_("E855: Autoc /* Number of times free_buffer() was called. */ static int buf_free_count = 0; +/* Read data from buffer for retrying. */ +static int +read_buffer( +int read_stdin, /* read file from stdin, otherwise fifo */ +exarg_T *eap, /* for forced 'ff' and 'fenc' or NULL */ +int flags) /* extra flags for readfile() */ +{ +int retval = OK; +linenr_T line_count; + +/* + * Read from the buffer which the text is already filled in and append at + * the end. This makes it possible to retry when 'fileformat' or + * 'fileencoding' was guessed wrong. + */ +line_count = curbuf->b_ml.ml_line_count; +retval = readfile( + read_stdin ? NULL : curbuf->b_ffname, + read_stdin ? NULL : curbuf->b_fname, + (linenr_T)line_count, (linenr_T)0, (linenr_T)MAXLNUM, eap, + flags | READ_BUFFER); +if (retval == OK) +{ + /* Delete the binary lines. */ + while (--line_count >= 0) + ml_delete((linenr_T)1, FALSE); +} +else +{ + /* Delete the converted lines. */ + while (curbuf->b_ml.ml_line_count > line_count) + ml_delete(line_count, FALSE); +} +/* Put the cursor on the first line. */ +curwin->w_cursor.lnum = 1; +curwin->w_cursor.col = 0; + +if (read_stdin) +{ + /* Set or reset 'modified' before executing autocommands, so that + * it can be changed there. */ + if (!readonlymode && !bufempty()) + changed(); + else if (retval != FAIL) + unchanged(curbuf, FALSE); + +#ifdef FEAT_AUTOCMD +# ifdef FEAT_EVAL + apply_autocmds_retval(EVENT_STDINREADPOST, NULL, NULL, FALSE, + curbuf, ); +# else + apply_autocmds(EVENT_STDINREADPOST, NULL, NULL, FALSE, curbuf); +# endif +#endif +} +return retval; +} + /* * Open current buffer, that is: open the memfile and read the file into * memory. @@ -88,6 +146,7 @@ open_buffer( #ifdef FEAT_SYN_HL long old_tw = curbuf->b_p_tw; #endif +int read_fifo = FALSE; /* * The 'readonly' flag is only set when BF_NEVERLOADED is being reset. @@ -143,17 +202,42 @@ open_buffer( ) { int old_msg_silent
Re: [patch] file encoding detection doesn't work with process substitution
Ken Takata wrote: > 2016/8/6 Sat 22:30:05 UTC+9 Bram Moolenaar wrote: > > Ken Takata wrote: > > > > > When process substitution (e.g. "<(cmd)") is used on bash (or zsh), file > > > encoding detection doesn't work, and an empty buffer is opened. > > > E.g.: > > > > > > $ vim -u NONE --cmd "set fencs=cp932,utf-8" <(echo テスト) > > > # Assuming the system encoding is utf-8. > > > > > > Vim will be executed with the command line like: > > > > > > $ vim -u NONE --cmd "set fencs=cp932,utf-8" /dev/fd/63 > > > > > > "/dev/fd/63" is a fifo and it is not seekable. > > > The echoed string "テスト" (utf-8) cannot be converted from cp932 to utf-8, > > > so > > > Vim retries with utf-8, but it fails because the input is not seekable. > > > > > > However reading from stdin can detect the encoding. E.g.: > > > > > > $ echo テスト | vim -u NONE --cmd "set fencs=cp932,utf-8" - > > > > > > I think file encoding detection should also work with process > > > substitution. > > > Attached patch fixes the problem. > > > > Thanks. It would be good to have a test for this. The encoding > > detection is important, but I don't think there is a test for it. > > > > Reading from stdin and fifo require starting Vim. Test49 does this, it > > just uses ../vim for that. That doesn't work on Windows though. But > > the fifo won't work on Windows either. > > I added a test for this. I also wanted to add a test for stdin on Windows, > but Windows' echo command cannot write Unicode characters to a pipe. So the > test is disabled on Windows. Thanks. It would be better to use "cat" instead of "echo". Does this also exist on MS-Windows? Or perhaps we can check if it exists. Then we can also use multiple lines in the test. > BTW, I found that RunVim() in shared.vim didn't work as expected. I also > fixed this. You mean the return value? Strange that everything worked anyway... Oh, it just skipped the actual test! -- If the Universe is constantly expanding, why can't I ever find a parking space? /// Bram Moolenaar -- b...@moolenaar.net -- http://www.Moolenaar.net \\\ ///sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\ \\\ an exciting new programming language -- http://www.Zimbu.org/// \\\help me help AIDS victims -- http://ICCF-Holland.org/// -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [patch] file encoding detection doesn't work with process substitution
Hi Bram, 2016/8/6 Sat 22:30:05 UTC+9 Bram Moolenaar wrote: > Ken Takata wrote: > > > When process substitution (e.g. "<(cmd)") is used on bash (or zsh), file > > encoding detection doesn't work, and an empty buffer is opened. > > E.g.: > > > > $ vim -u NONE --cmd "set fencs=cp932,utf-8" <(echo テスト) > > # Assuming the system encoding is utf-8. > > > > Vim will be executed with the command line like: > > > > $ vim -u NONE --cmd "set fencs=cp932,utf-8" /dev/fd/63 > > > > "/dev/fd/63" is a fifo and it is not seekable. > > The echoed string "テスト" (utf-8) cannot be converted from cp932 to utf-8, so > > Vim retries with utf-8, but it fails because the input is not seekable. > > > > However reading from stdin can detect the encoding. E.g.: > > > > $ echo テスト | vim -u NONE --cmd "set fencs=cp932,utf-8" - > > > > I think file encoding detection should also work with process substitution. > > Attached patch fixes the problem. > > Thanks. It would be good to have a test for this. The encoding > detection is important, but I don't think there is a test for it. > > Reading from stdin and fifo require starting Vim. Test49 does this, it > just uses ../vim for that. That doesn't work on Windows though. But > the fifo won't work on Windows either. I added a test for this. I also wanted to add a test for stdin on Windows, but Windows' echo command cannot write Unicode characters to a pipe. So the test is disabled on Windows. BTW, I found that RunVim() in shared.vim didn't work as expected. I also fixed this. Regards, Ken Takata -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. # HG changeset patch # Parent fe67d6106f686209c616b7dc937457c909b0ee31 diff --git a/src/Makefile b/src/Makefile --- a/src/Makefile +++ b/src/Makefile @@ -2111,6 +2111,7 @@ test_arglist \ test_reltime \ test_ruby \ test_startup \ + test_startup_utf8 \ test_searchpos \ test_set \ test_sort \ diff --git a/src/buffer.c b/src/buffer.c --- a/src/buffer.c +++ b/src/buffer.c @@ -70,6 +70,64 @@ static char *e_auabort = N_("E855: Autoc /* Number of times free_buffer() was called. */ static int buf_free_count = 0; +/* Read data from buffer for retrying. */ +static int +read_buffer( +int read_stdin, /* read file from stdin, otherwise fifo */ +exarg_T *eap, /* for forced 'ff' and 'fenc' or NULL */ +int flags) /* extra flags for readfile() */ +{ +int retval = OK; +linenr_T line_count; + +/* + * Read from the buffer which the text is already filled in and append at + * the end. This makes it possible to retry when 'fileformat' or + * 'fileencoding' was guessed wrong. + */ +line_count = curbuf->b_ml.ml_line_count; +retval = readfile( + read_stdin ? NULL : curbuf->b_ffname, + read_stdin ? NULL : curbuf->b_fname, + (linenr_T)line_count, (linenr_T)0, (linenr_T)MAXLNUM, eap, + flags | READ_BUFFER); +if (retval == OK) +{ + /* Delete the binary lines. */ + while (--line_count >= 0) + ml_delete((linenr_T)1, FALSE); +} +else +{ + /* Delete the converted lines. */ + while (curbuf->b_ml.ml_line_count > line_count) + ml_delete(line_count, FALSE); +} +/* Put the cursor on the first line. */ +curwin->w_cursor.lnum = 1; +curwin->w_cursor.col = 0; + +if (read_stdin) +{ + /* Set or reset 'modified' before executing autocommands, so that + * it can be changed there. */ + if (!readonlymode && !bufempty()) + changed(); + else if (retval != FAIL) + unchanged(curbuf, FALSE); + +#ifdef FEAT_AUTOCMD +# ifdef FEAT_EVAL + apply_autocmds_retval(EVENT_STDINREADPOST, NULL, NULL, FALSE, + curbuf, ); +# else + apply_autocmds(EVENT_STDINREADPOST, NULL, NULL, FALSE, curbuf); +# endif +#endif +} +return retval; +} + /* * Open current buffer, that is: open the memfile and read the file into * memory. @@ -88,6 +146,7 @@ open_buffer( #ifdef FEAT_SYN_HL long old_tw = curbuf->b_p_tw; #endif +int read_fifo = FALSE; /* * The 'readonly' flag is only set when BF_NEVERLOADED is being reset. @@ -143,17 +202,42 @@ open_buffer( ) { int old_msg_silent = msg_silent; - +#ifdef UNIX + int save_bin = curbuf->b_p_bin; + int perm; +#endif #ifdef FEAT_NETBEANS_INTG int oldFire = netbeansFireChanges; netbeansFireChanges = 0; #endif +#ifdef UNIX + perm = mch_getperm(curbuf->b_ffname); + if (perm >= 0 && (0 +# ifdef S_ISFIFO + || S_ISFIFO(perm) +# endif +# ifdef S_ISSOCK + || S_ISSOCK(perm) +# endif + )) + read_fifo = TRUE; + if
Re: [patch] file encoding detection doesn't work with process substitution
Ken Takata wrote: > When process substitution (e.g. "<(cmd)") is used on bash (or zsh), file > encoding detection doesn't work, and an empty buffer is opened. > E.g.: > > $ vim -u NONE --cmd "set fencs=cp932,utf-8" <(echo テスト) > # Assuming the system encoding is utf-8. > > Vim will be executed with the command line like: > > $ vim -u NONE --cmd "set fencs=cp932,utf-8" /dev/fd/63 > > "/dev/fd/63" is a fifo and it is not seekable. > The echoed string "テスト" (utf-8) cannot be converted from cp932 to utf-8, so > Vim retries with utf-8, but it fails because the input is not seekable. > > However reading from stdin can detect the encoding. E.g.: > > $ echo テスト | vim -u NONE --cmd "set fencs=cp932,utf-8" - > > I think file encoding detection should also work with process substitution. > Attached patch fixes the problem. Thanks. It would be good to have a test for this. The encoding detection is important, but I don't think there is a test for it. Reading from stdin and fifo require starting Vim. Test49 does this, it just uses ../vim for that. That doesn't work on Windows though. But the fifo won't work on Windows either. -- Mrs Abbott: I'm a paediatrician. Basil: Feet? Mrs Abbott: Children. Sybil: Oh, Basil! Basil: Well, children have feet, don't they? That's how they move around, my dear. You must take a look next time, it's most interesting. (Fawlty Towers) /// Bram Moolenaar -- b...@moolenaar.net -- http://www.Moolenaar.net \\\ ///sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\ \\\ an exciting new programming language -- http://www.Zimbu.org/// \\\help me help AIDS victims -- http://ICCF-Holland.org/// -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[patch] file encoding detection doesn't work with process substitution
Hi, When process substitution (e.g. "<(cmd)") is used on bash (or zsh), file encoding detection doesn't work, and an empty buffer is opened. E.g.: $ vim -u NONE --cmd "set fencs=cp932,utf-8" <(echo テスト) # Assuming the system encoding is utf-8. Vim will be executed with the command line like: $ vim -u NONE --cmd "set fencs=cp932,utf-8" /dev/fd/63 "/dev/fd/63" is a fifo and it is not seekable. The echoed string "テスト" (utf-8) cannot be converted from cp932 to utf-8, so Vim retries with utf-8, but it fails because the input is not seekable. However reading from stdin can detect the encoding. E.g.: $ echo テスト | vim -u NONE --cmd "set fencs=cp932,utf-8" - I think file encoding detection should also work with process substitution. Attached patch fixes the problem. Regards, Ken Takata -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. # HG changeset patch # Parent cc54db6e0d02246ac4ec02cfbe3edccbb6b47872 diff --git a/src/buffer.c b/src/buffer.c --- a/src/buffer.c +++ b/src/buffer.c @@ -70,6 +70,64 @@ static char *e_auabort = N_("E855: Autoc /* Number of times free_buffer() was called. */ static int buf_free_count = 0; +/* Read data from buffer for retrying. */ +static int +read_buffer( +int read_stdin, /* read file from stdin, otherwise fifo */ +exarg_T *eap, /* for forced 'ff' and 'fenc' or NULL */ +int flags) /* extra flags for readfile() */ +{ +int retval = OK; +linenr_T line_count; + +/* + * Read from the buffer which the text is already filled in and append at + * the end. This makes it possible to retry when 'fileformat' or + * 'fileencoding' was guessed wrong. + */ +line_count = curbuf->b_ml.ml_line_count; +retval = readfile( + read_stdin ? NULL : curbuf->b_ffname, + read_stdin ? NULL : curbuf->b_fname, + (linenr_T)line_count, (linenr_T)0, (linenr_T)MAXLNUM, eap, + flags | READ_BUFFER); +if (retval == OK) +{ + /* Delete the binary lines. */ + while (--line_count >= 0) + ml_delete((linenr_T)1, FALSE); +} +else +{ + /* Delete the converted lines. */ + while (curbuf->b_ml.ml_line_count > line_count) + ml_delete(line_count, FALSE); +} +/* Put the cursor on the first line. */ +curwin->w_cursor.lnum = 1; +curwin->w_cursor.col = 0; + +if (read_stdin) +{ + /* Set or reset 'modified' before executing autocommands, so that + * it can be changed there. */ + if (!readonlymode && !bufempty()) + changed(); + else if (retval != FAIL) + unchanged(curbuf, FALSE); + +#ifdef FEAT_AUTOCMD +# ifdef FEAT_EVAL + apply_autocmds_retval(EVENT_STDINREADPOST, NULL, NULL, FALSE, + curbuf, ); +# else + apply_autocmds(EVENT_STDINREADPOST, NULL, NULL, FALSE, curbuf); +# endif +#endif +} +return retval; +} + /* * Open current buffer, that is: open the memfile and read the file into * memory. @@ -88,6 +146,7 @@ open_buffer( #ifdef FEAT_SYN_HL long old_tw = curbuf->b_p_tw; #endif +int read_fifo = FALSE; /* * The 'readonly' flag is only set when BF_NEVERLOADED is being reset. @@ -143,17 +202,42 @@ open_buffer( ) { int old_msg_silent = msg_silent; - +#ifdef UNIX + int save_bin = curbuf->b_p_bin; + int perm; +#endif #ifdef FEAT_NETBEANS_INTG int oldFire = netbeansFireChanges; netbeansFireChanges = 0; #endif +#ifdef UNIX + perm = mch_getperm(curbuf->b_ffname); + if (perm >= 0 && (0 +# ifdef S_ISFIFO + || S_ISFIFO(perm) +# endif +# ifdef S_ISSOCK + || S_ISSOCK(perm) +# endif + )) + read_fifo = TRUE; + if (read_fifo) + curbuf->b_p_bin = TRUE; +#endif if (shortmess(SHM_FILEINFO)) msg_silent = 1; retval = readfile(curbuf->b_ffname, curbuf->b_fname, (linenr_T)0, (linenr_T)0, (linenr_T)MAXLNUM, eap, - flags | READ_NEW); + flags | READ_NEW | (read_fifo ? READ_FIFO : 0)); +#ifdef UNIX + if (read_fifo) + { + curbuf->b_p_bin = save_bin; + if (retval == OK) + retval = read_buffer(FALSE, eap, flags); + } +#endif msg_silent = old_msg_silent; #ifdef FEAT_NETBEANS_INTG netbeansFireChanges = oldFire; @@ -164,8 +248,7 @@ open_buffer( } else if (read_stdin) { - int save_bin = curbuf->b_p_bin; - linenr_T line_count; + int save_bin = curbuf->b_p_bin; /* * First read the text in binary mode into the buffer. @@ -179,42 +262,7 @@ open_buffer( flags | (READ_NEW + READ_STDIN)); curbuf->b_p_bin = save_bin; if (retval == OK) - { - line_count = curbuf->b_ml.ml_line_count; - retval = readfile(NULL, NULL, (linenr_T)line_count, -