Re: [patch] Respect file's EOL/NOEOL settings
Hi all, I don't like the name 'respecteol', from just seeing it I don't know what it means. 'fixeol' probably would do the same, only has the inverted value. 'fixeol' is much better. Thank you, I was not sure about the name since the very beginning. Please find the updated patch in attachment. -- Best Regards, Pavel -- -- You received this message from the vim_dev maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups vim_dev group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. fixeol_v4.src.patch Description: Binary data fixeol_v4.runtime.patch Description: Binary data
Re: [patch] Respect file's EOL/NOEOL settings
Hi Bram, Christian, James, Olaf, Pavel
On 2015-07-11 Saturday at 00:22 +0200 Christian Brabandt wrote:
On Fr, 10 Jul 2015, Bram Moolenaar wrote:
The 'addeol' option indicates adding an EOL, also when there already was
one, that's confusing. We could call it 'fixeol', which would normally
be on. For some file formats for some systems it could be switched off,
so that the missing EOL is not added.
I might be missing something (or I did not quite understand all mails
posted), but what is the problem with the 'respecteol' patch that has
been posted here in the beginning?
What about just dropping the dependency of option “endofline” on option
“binary”?
Yes, in theory cases of breaking backwards compatibility by this change
could be constructed, but do they matter for _reasonable_ usage in
already written code?
--- runtime/doc/options.txt~
+++ runtime/doc/options.txt
@@ -2661,15 +2661,15 @@
*'endofline'* *'eol'* *'noendofline'* *'noeol'*
'endofline' 'eol' boolean (default on)
local to buffer
{not in Vi}
- When writing a file and this option is off and the 'binary' option
- is on, no EOL will be written for the last line in the file. This
+ When writing a file and this option is off, no EOL will be written
+ for the last line in the file. Thus denominating this option
+ “endoflastline” would more intuitively tell its purpose. This
option is automatically set when starting to edit a new file, unless
the file does not have an EOL for the last line in the file, in
which case it is reset. Normally you don't have to set or reset this
- option. When 'binary' is off the value is not used when writing the
- file. When 'binary' is on it is used to remember the presence of a
+ option. It is used to remember the presence of an
EOL for the last line in the file, so that when you write the file
the situation from the original file can be kept. But you can change
it if you want to.
--
Best regards,
Roland Eggner
--
--
You received this message from the vim_dev maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php
---
You received this message because you are subscribed to the Google Groups
vim_dev group.
To unsubscribe from this group and stop receiving emails from it, send an email
to vim_dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
pgpQOCN5HP6fS.pgp
Description: PGP signature
Re: [patch] Respect file's EOL/NOEOL settings
Christian wrote: On Fr, 10 Jul 2015, Bram Moolenaar wrote: The 'addeol' option indicates adding an EOL, also when there already was one, that's confusing. We could call it 'fixeol', which would normally be on. For some file formats for some systems it could be switched off, so that the missing EOL is not added. I might be missing something (or I did not quite understand all mails posted), but what is the problem with the 'respecteol' patch that has been posted here in the beginning? I don't like the name 'respecteol', from just seeing it I don't know what it means. 'fixeol' probably would do the same, only has the inverted value. -- hundred-and-one symptoms of being an internet addict: 208. Your goals for the future are obtaining an T1 connection and a 130 gig hard drive. /// Bram Moolenaar -- b...@moolenaar.net -- http://www.Moolenaar.net \\\ ///sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\ \\\ an exciting new programming language -- http://www.Zimbu.org/// \\\help me help AIDS victims -- http://ICCF-Holland.org/// -- -- You received this message from the vim_dev maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups vim_dev group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [patch] Respect file's EOL/NOEOL settings
On 10-Jul-15, Bram Moolenaar wrote:
Olaf Dabrunz wrote:
I think this is one way to do this.
However, I believe the final EOL in a file should be made visible and
editable.
Consider this:
file A file B
,---. ,---.
|...| |...|
|Last line | |Last line |
|~ | | | -- lookie here
|~ | |~ |
`---´ `---´
ends on 'Last line' ends on an empty line
No, this looks like there is an extra empty line. Note that EOL means
End Of Line, not line separator.
Yes.
Apparently, using EOL for either the NL or CR character or the CR+NL
sequence that vim recognises as line terminators caused confusion.
The files we talk about here use NL or CR+NL as line separators.
So even if vim internally calls them EOL, which is what I meant when
saying EOL, they are not used as EOLs in these files, and saying EOL
can be misunderstood.
The problem is that there seems to be no neutral word for NL or CR or
CR+NL that does not also say how they are interpreted.
Line end hints at line termination, newline, next line and
line break suggest line separation.
Maybe it is better to use something like line terminator / separator
and LT/S or LTS for short?
On many systems a missing EOL
indicates an incomplete file, it was truncated.
And appending an EOL to the last line fixes the format at least, so it
complies with POSIX again.
Unfortunately some
programs, especially on MS-Windows, have the habit of forgetting the EOL
and try to change the rules by saying it's supposed to be this way.
Yes.
If vim had a mode where it *never* adds a newline on the last line when
writing a file, then vim would write file A without a final EOL, and
file B with a final EOL, and no other differences.
What I mean here and below is an LT/S (as defined above).
And when vim is in that mode, when reading a file, vim would make it
show up like file A when the file has no newline at the end, and like
file B when it has a newline at the end.
This way, it becomes clearly visible in the editor what the file
contains, and the user can change it too.
If you want to prevent that for a certain filetype the file contains an
EOL at the end, you can either highlight this as an error to the user:
:call matchadd('ErrorMsg', '^\%$')
or automatically remove that line with a BufWritePre autocommand:
:augroup CProj
:au!
:autocmd BufWritePre *.cproj if addeol == 0 | /^\%$/d | endif
:augroup END
(assuming the option controlling this mode is called 'addeol')
or both.
Of course, the option could also be set depending on filetype:
:augroup MyFiletype
:autocmd FileType cproj set noaddeol
:augroup END
When the 'addeol' option is turned off ('noaddeol'), if the 'eol' option
is set, an empty line is appended to the buffer. When 'addeol' is
turned on, and the buffer ends in an empty line, that empty line is
removed and the 'eol' option is set, or otherwise 'eol' is cleared
('noeol').
Of course, if 'addeol' is off ('noaddeol') when a file is read, then
'eol' is always cleared.
Writing a file when 'binary' is set must ignore 'eol' when 'addeol' is
off -- the EOL is already in the buffer, no need to add another one
('eol' should be off for 'noaddeol', it is bogus anyway).
It think it should be fairly easy to implement an option like 'addeol'
in vim.
What do you think?
The 'addeol' option indicates adding an EOL, also when there already was
one, that's confusing.
Right.
We could call it 'fixeol', which would normally
'fixeol' is much better.
be on. For some file formats for some systems it could be switched off,
so that the missing EOL is not added.
Yes.
--
Olaf Dabrunz (oda at fctrace.org)
--
--
You received this message from the vim_dev maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php
---
You received this message because you are subscribed to the Google Groups
vim_dev group.
To unsubscribe from this group and stop receiving emails from it, send an email
to vim_dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [patch] Respect file's EOL/NOEOL settings
Olaf Dabrunz wrote:
I think this is one way to do this.
However, I believe the final EOL in a file should be made visible and
editable.
Consider this:
file A file B
,---. ,---.
|...| |...|
|Last line | |Last line |
|~ | | | -- lookie here
|~ | |~ |
`---´ `---´
ends on 'Last line' ends on an empty line
No, this looks like there is an extra empty line. Note that EOL means
End Of Line, not line separator. On many systems a missing EOL
indicates an incomplete file, it was truncated. Unfortunately some
programs, especially on MS-Windows, have the habit of forgetting the EOL
and try to change the rules by saying it's supposed to be this way.
If vim had a mode where it *never* adds a newline on the last line when
writing a file, then vim would write file A without a final EOL, and
file B with a final EOL, and no other differences.
And when vim is in that mode, when reading a file, vim would make it
show up like file A when the file has no newline at the end, and like
file B when it has a newline at the end.
This way, it becomes clearly visible in the editor what the file
contains, and the user can change it too.
If you want to prevent that for a certain filetype the file contains an
EOL at the end, you can either highlight this as an error to the user:
:call matchadd('ErrorMsg', '^\%$')
or automatically remove that line with a BufWritePre autocommand:
:augroup CProj
:au!
:autocmd BufWritePre *.cproj if addeol == 0 | /^\%$/d | endif
:augroup END
(assuming the option controlling this mode is called 'addeol')
or both.
Of course, the option could also be set depending on filetype:
:augroup MyFiletype
:autocmd FileType cproj set noaddeol
:augroup END
When the 'addeol' option is turned off ('noaddeol'), if the 'eol' option
is set, an empty line is appended to the buffer. When 'addeol' is
turned on, and the buffer ends in an empty line, that empty line is
removed and the 'eol' option is set, or otherwise 'eol' is cleared
('noeol').
Of course, if 'addeol' is off ('noaddeol') when a file is read, then
'eol' is always cleared.
Writing a file when 'binary' is set must ignore 'eol' when 'addeol' is
off -- the EOL is already in the buffer, no need to add another one
('eol' should be off for 'noaddeol', it is bogus anyway).
It think it should be fairly easy to implement an option like 'addeol'
in vim.
What do you think?
The 'addeol' option indicates adding an EOL, also when there already was
one, that's confusing. We could call it 'fixeol', which would normally
be on. For some file formats for some systems it could be switched off,
so that the missing EOL is not added.
--
Some of the well known MS-Windows errors:
EMULTI Multitasking attempted, system confused
EKEYBOARD Keyboard locked, try getting out of this one!
EXPLAIN Unexplained error, please tell us what happened
EFUTURE Reserved for our future mistakes
/// Bram Moolenaar -- b...@moolenaar.net -- http://www.Moolenaar.net \\\
///sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org///
\\\help me help AIDS victims -- http://ICCF-Holland.org///
--
--
You received this message from the vim_dev maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php
---
You received this message because you are subscribed to the Google Groups
vim_dev group.
To unsubscribe from this group and stop receiving emails from it, send an email
to vim_dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [patch] Respect file's EOL/NOEOL settings
On 08-Jul-15, Pavel Samarkin wrote: Hi, If you use git, then in “git help attributes” the example mentioning *.vcproj files might be useful for you. Possibly the end-of-line normalization offered by git includes the case where just the last line needs normalization. Roland - thank you, I did not know about that feature of git, it might be useful for my *.csproj situation, but generally I don't want my files to be normalized in any way, I want them to be left intact. 1. 'norespecteol' (default) means that an EOL is always added unless 'eol' + 'bin' are set, i.e. Vim's existing behavior. 2. Opening an existing file when 'respecteol' is set honors the current state of the EOL on the last line, and should set 'eol' accordingly. 3. If a user wishes a new, unsaved buffer to not have an EOL on the last line, she can set 'respecteol' and 'noeol'. James - I think in #1 you meant 'noeol' + 'bin' - if yes, then 1-3 is exactly what my patch adds. I believe the final EOL in a file should be made visible and editable. Olaf - James is right, according to the POSIX standard, EOL means EOL, not 'newline', regardless of how it's treated by some other text editors. According to the POSIX standard you cited, 3.206 Line A sequence of zero or more non- newline characters plus a terminating newline character. a line is terminated by a newline character. It does not mention EOL. In the other mail I tried to hint at the different meanings and interpretations. They can be confusing. As a small overview: Unicode says LF / NL and EOL are the same, at codepoint 0x000A: http://unicode.org/charts/PDF/U.pdf. This extends the meaning of 0x0A from the source character set ISO 646 / ECMA-6 and ISO 6429 / ECMA-48, which defines this as LF. Unicode invented a different code point for line separator LS at 0x2028 to represent this semantic unambiguously: http://unicode.org/charts/PDF/U2000.pdf Note that there is also NEL (next line) at 0x0085, invented by ISO 6429 / ECMA-48. See the Unicode Newline Guidelines at http://unicode.org/standard/reports/tr13/tr13-5.html for a discussion and recommendations around newline / EOL etc. treatment. The original definitions of characters like NL and CR are operational (e.g. move to the next line, or to the beginning of the line, see ISO 6429 / ECMA-48). But EOL is the concept of a line terminator, which needs to be represented by one or more codes. A standard like POSIX simply adopted NL as the code for a line terminator. Others have adopted NL, CR+NL or CR as line terminators or as line separators. Have a look at https://en.wikipedia.org/wiki/Newline to find more representations for next line, line terminator or line separator, both historic and current. The reason why in vim we can conveniently use the concept of EOL is that vim does such a good job of converting the different representations into a single internal representation (lines terminated by NUL '\0'). But the problem here is not the different representations for EOL, but that there is software that uses the concept of a line separator rather than a line terminator like EOL, yet represents this with a similar selection of codes (NL, CR+NL, maybe even CR). Besides, I don't want to break the existing Vim functionality, I just want to expand the horizons of its usage beyond the Unix world it has been originally created for. Fully agreed. I do not think that a suggestion like '(no)addeol' breaks vim. It is quite comparable to your approach, but IMHO even simpler, and more visible and editable to the user: When saving a file, do not append EOL (or in this case line separator) representation characters to the last line. To make the file end on a line separator, an empty line must be at the end of the buffer. This behavior can be turned on or off with a single option. Only when switching this option on or off, an empty last line is converted to 'eol' or back, and if the last line is not empty, 'noeol' is set or no empty line is appended to the buffer when converting backwards. And when a file is read, then the setting of the option determines if the state of the last EOL will show up in 'eol' or in the buffer. That's it. The main advantage of my approach, as I see it, is that when a file uses the line separator concept instead of the EOL concept, it does not hide the empty line at the end in an option. Instead, it makes that line visible and malleable in a very natural way inside the buffer. This makes the user see what she gets, and she can manipulate it easily. I may not have convinced you, so maybe it helps if we could compare the two approaches side-by-side. But I will be very busy with other things in the next days, so I am not sure if I will get around to write an implementation
Re: [patch] Respect file's EOL/NOEOL settings
Hi Roland, Thanks for the suggestion - you're absolutely right, most of the problems can be solved with various scripts - in this particular case, it's possible to do even better - analyze the output of git diff and remove the extra EOL where needed. We can work around each application that breaks the standard, or we can add support for non-standard files directly to the text editor, thus solving this problem once and for all. It's Bram's choice now. -- Best regards, Pavel -- -- You received this message from the vim_dev maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups vim_dev group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [patch] Respect file's EOL/NOEOL settings
On Thu, Jul 09, 2015 at 03:09:58AM +0200, Olaf Dabrunz wrote: However, I believe the final EOL in a file should be made visible and editable. Consider this: file A file B ,---. ,---. |...| |...| |Last line | |Last line | |~ | | | -- lookie here |~ | |~ | `---´ `---´ ends on 'Last line' ends on an empty line If vim had a mode where it *never* adds a newline on the last line when writing a file, then vim would write file A without a final EOL, and file B with a final EOL, and no other differences. This is just propagating the misconception that the EOL is a new line, instead of an indicator of the end of the current line. Why should I add a blank line, that isn't really a line, to get an EOL on the just previous non-blank line? And when vim is in that mode, when reading a file, vim would make it show up like file A when the file has no newline at the end, and like file B when it has a newline at the end. This way, it becomes clearly visible in the editor what the file contains, and the user can change it too. While making it easier to accomodate the myriad applications that don't understand EOL is useful, misrepresenting the contents of the file doesn't seem like the right way to do it. The exact behavior you're describing is one of the things that annoys me about Emacs, because I still haven't internalized the fact that it allows me to place the cursor past the last line in the buffer. This leads to the fun behavior of me deleting the last “line”, saving the buffer, and the last “line” reappearing because it properly re-adds the EOL. I haven't followed the discussion about 'respecteol' closely, but this is the behavior I'd expect based on the name: 1. 'norespecteol' (default) means that an EOL is always added unless 'eol' + 'bin' are set, i.e. Vim's existing behavior. 2. Opening an existing file when 'respecteol' is set honors the current state of the EOL on the last line, and should set 'eol' accordingly. 3. If a user wishes a new, unsaved buffer to not have an EOL on the last line, she can set 'respecteol' and 'noeol'. Cheers, -- James GPG Key: 4096R/331BA3DB 2011-12-05 James McCoy james...@jamessan.com -- -- You received this message from the vim_dev maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups vim_dev group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [patch] Respect file's EOL/NOEOL settings
On 22-Jun-15, Pavel Samarkin wrote:
Hi,
Issue: Vim adds EOL to the end of each text file
Although my change might seem controversial, it looks like a lot of people
(including myself) really want it, especially when using Vim on non-Unix
systems (e.g. Windows).
You can notice it here:
http://vim.wikia.com/wiki/Preserve_missing_end-of-line_at_end_of_text_files
http://www.vim.org/scripts/script.php?script_id=4550
and in several questions on stackoverflow:
http://stackoverflow.com/questions/1050640
http://stackoverflow.com/questions/4133501
http://stackoverflow.com/questions/14171254
http://askubuntu.com/questions/13317
The solution suggested in Vim FAQ (5.4)
http://vimdoc.sourceforge.net/htmldoc/vimfaq.html
includes some side-effects like inability to use 'expandtab', 'textwidth' and
some other options (incl. Unicode BOM).
My proposed change introduces the new option called 'respecteol' (that's the
best name I could come up with) which affects the EOL at the end of file the
same way as the 'binary' option does, while not changing any other aspects.
I think this is one way to do this.
However, I believe the final EOL in a file should be made visible and
editable.
Consider this:
file A file B
,---. ,---.
|...| |...|
|Last line | |Last line |
|~ | | | -- lookie here
|~ | |~ |
`---´ `---´
ends on 'Last line' ends on an empty line
If vim had a mode where it *never* adds a newline on the last line when
writing a file, then vim would write file A without a final EOL, and
file B with a final EOL, and no other differences.
And when vim is in that mode, when reading a file, vim would make it
show up like file A when the file has no newline at the end, and like
file B when it has a newline at the end.
This way, it becomes clearly visible in the editor what the file
contains, and the user can change it too.
If you want to prevent that for a certain filetype the file contains an
EOL at the end, you can either highlight this as an error to the user:
:call matchadd('ErrorMsg', '^\%$')
or automatically remove that line with a BufWritePre autocommand:
:augroup CProj
:au!
:autocmd BufWritePre *.cproj if addeol == 0 | /^\%$/d | endif
:augroup END
(assuming the option controlling this mode is called 'addeol')
or both.
Of course, the option could also be set depending on filetype:
:augroup MyFiletype
:autocmd FileType cproj set noaddeol
:augroup END
When the 'addeol' option is turned off ('noaddeol'), if the 'eol' option
is set, an empty line is appended to the buffer. When 'addeol' is
turned on, and the buffer ends in an empty line, that empty line is
removed and the 'eol' option is set, or otherwise 'eol' is cleared
('noeol').
Of course, if 'addeol' is off ('noaddeol') when a file is read, then
'eol' is always cleared.
Writing a file when 'binary' is set must ignore 'eol' when 'addeol' is
off -- the EOL is already in the buffer, no need to add another one
('eol' should be off for 'noaddeol', it is bogus anyway).
It think it should be fairly easy to implement an option like 'addeol'
in vim.
What do you think?
--
Olaf Dabrunz (oda at fctrace.org)
--
--
You received this message from the vim_dev maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php
---
You received this message because you are subscribed to the Google Groups
vim_dev group.
To unsubscribe from this group and stop receiving emails from it, send an email
to vim_dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [patch] Respect file's EOL/NOEOL settings
Hi Pavel On 2015-07-08 Wednesday at 14:16 -0700 Pavel Samarkin wrote: Thanks for the suggestion - you're absolutely right, most of the problems can be solved with various scripts - in this particular case, it's possible to do even better - analyze the output of git diff and remove the extra EOL where needed. If you use git, then in “git help attributes” the example mentioning *.vcproj files might be useful for you. Possibly the end-of-line normalization offered by git includes the case where just the last line needs normalization. -- Best regards, Roland Eggner -- -- You received this message from the vim_dev maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups vim_dev group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. pgpltG4VvrblA.pgp Description: PGP signature
Re: [patch] Respect file's EOL/NOEOL settings
On 08-Jul-15, James McCoy wrote:
On Thu, Jul 09, 2015 at 03:09:58AM +0200, Olaf Dabrunz wrote:
However, I believe the final EOL in a file should be made visible and
editable.
Consider this:
file A file B
,---. ,---.
|...| |...|
|Last line | |Last line |
|~ | | | -- lookie here
|~ | |~ |
`---´ `---´
ends on 'Last line' ends on an empty line
If vim had a mode where it *never* adds a newline on the last line when
writing a file, then vim would write file A without a final EOL, and
file B with a final EOL, and no other differences.
This is just propagating the misconception that the EOL is a new line,
instead of an indicator of the end of the current line. Why should I
add a blank line, that isn't really a line, to get an EOL on the just
previous non-blank line?
Yes, EOL means end of line, and not new line, in the sense of the words.
And in POSIX, a line must be terminated by a newline character.
But in its literal sense, EOL is only a concept, not a character, or
something that appears in a file. EOL is implemented with characters
like NL, CR+NL and CR (':he EOL'). So technically, a text file could
end on something else but characters that represent an EOL.
And if we like it or not, the strict sense of seeing EOL as a line
terminator is only one view. See e.g.
https://en.wikipedia.org/wiki/Newline:
In computing, a newline, also known as a line ending, end of line
(EOL), or line break, is a special character or sequence of
characters signifying the end of a line of text.
[...]
Two ways to view newlines, both of which have valid internal logic,
are that newlines terminate lines or that they separate lines. If a
newline is considered a separator, there will be no newline after
the last line of a file. Some programs have problems processing the
last line of a file if it is not newline-terminated. On the other
hand, programs that expect newline to be used as a separator will
interpret a final newline as starting a new (empty) line.
Some applications can handle either view, some can only handle one view.
Rather than discussing who is right or wrong on this, or why it has
developed in this way, I would prefer that vim supports editing text
that can readily be consumed by any application, regardless to which
view it adheres.
Unless this would mean to add support for rare use cases, or if the cost
of maintenance would be too high, or if the potential to confuse users
would be significant.
For what it's worth, in this case I am mostly concerned about adding
more code that must be maintained.
The use case does not seem to be that rare (MS Visual Studio project
files were given as an example), and the potential for confusion should
be minimal: users do not need to touch 'addeol', similarly to 'binary'.
If they are likely to touch it, displaying the state of 'addeol' in the
statusline should help.
It has also been mentioned how annoying it can be when vim always adds
an EOL to the last line, while other people on a project use other
editors where this becomes visible as an additional line.
Also, source code control and GNU diff see this as a change.
In such projects, vim can be seen as an annoying citizen.
In the gist of the robustness principle, vim should
not just [...] survive other misbehaving [software], but also [...]
cooperate to limit the amount of disruption such [software] can
cause to the shared [...] facility. [-- Adapted from RFC 1122.]
Or in other words, make it possible to create the file formats others
expect, even if that means accomodating some other view.
On the other hand, if a plugin could *reliably* take care of this, then
changes to vim are not needed.
Using 'binary' when saving a file is the only option that preserves the
last line ending. But it does not revert encoding conversions and
fileformat conversions done when reading the file.
So some other solution is needed.
And when vim is in that mode, when reading a file, vim would make it
show up like file A when the file has no newline at the end, and like
file B when it has a newline at the end.
This way, it becomes clearly visible in the editor what the file
contains, and the user can change it too.
While making it easier to accomodate the myriad applications that don't
understand EOL is useful, misrepresenting the contents of the file
doesn't seem like the right way to do it.
When seen as a separator, EOL (in its character form as NL, CR+NL or CR)
separates two lines. In this view, the representation is correct:
,---. ,---.
|...| |...
Re: [patch] Respect file's EOL/NOEOL settings
Hi Ben thank you for your detailed reply and friendly advices. Sorry if my message has been misunderstood: My intent was to add another view, which could eventually help Pavel and Bram, to save work time or to extend the scope of solved problems. On 2015-06-24 Wednesday at 07:49 -0700 Ben Fritz wrote: On Wednesday, June 24, 2015 at 5:12:01 AM UTC-5, Roland Eggner wrote: There are more cases, where vim tries to fix “errors” or “irregularities” of files and thereby damages them, unless “++binary” has been included in the reading command. Just two examples: (1) viminfo files with register contents resulting from alternating fileencondings, e.g. utf-8, latin1, latin9: When viewing or editing such viminfo files, including “++binary” in the reading command avoids data damage. First of all, .viminfo files are not really intended to be hand-edited, although the format is simple enough that it's certainly possible most of the time. Maybe I am a “villain” because of calling vim via a wrapper script which edits .viminfo files. This gives me a buffer list, which keeps entries until referred files disappear from filesystem or their atime is older than 3 weeks. And it decreases the frequency of losing text marks `a - `z by deletion of jump lists. However, you're expecting something fairly unreasonable. Vim has no way of marking different regions of a file as having different encodings. In fact I am not aware of any text editors that DO allow this. How does the editor know what encoding to apply to any new text? How are the regions delimited, especially if the delimiter could have different representations in different encodings? In the case that you have multiple encodings in a file, the file really and truly *IS* a binary file. My intent was exactly this conclusion to be drawn by readers. If line specific encodings could be implemented properly, it would be of little use in the case of my example (2): the patch files would appear redundant. Bram added this probably related entry to the todo list more than 5 years ago: When a register contains illegal bytes, writing viminfo in utf-8 and reading it back doesn't result in utf-8. (Devin Bayer) “:help viminfo-encoding” appears to give a hint in the last sentence, that different encodings used for different lines under certain circumstances might be intentional: … :set viminfo+=c Vim will then attempt to convert the text in the viminfo file from the 'encoding' value it was written with to the current 'encoding' value. This requires Vim to be compiled with the |+iconv| feature. Filenames are not converted. But why do you have multiple encodings in the file? The encoding of text in the _viminfo file should only depend on the 'encoding' option of Vim, it should not depend on the fileencoding option of the various files. Are you setting 'encoding' differently as you open files in different files? You should not be doing that...you should keep 'encoding' set to utf-8 and change 'fileencoding' as needed. Yes, in theory it should, but praxis differs, despite I am doing for many years exactly what you recommend. Detailed bug reports regarding this topic must wait, until I find more spare time. How can I specify the binary attribute, when vim tries to restore this register contents in a later session? vim-7.4 appears to ignore the line “*encoding=utf-8“ on reading of viminfo files. Vim does not, by default, detect any encoding from any text in the file, except for reading a BOM to detect certain Unicode encodings. If you distinguish the _guessing_ of encodings performed in src/fileio.c:readfile() from _detection_ of declared encodings: fully agreed. For that, you need a plugin. I am fond of Autofenc: http://www.vim.org/scripts/script.php?script_id=2721 Thank you for this reference. I will check it when I update my vim installations the next time (not in the near future). (2) The patch file resulting from the diff between old and new files after a command similar to “iconv -f ISO-8859-1 -t utf-8 …” usually needs to be treated as binary. vim damages such patch files on writing, unless the reading command includes “++binary”. See above. Such a file *is* a binary file, it cannot be anything else. My intent was exactly this conclusion to be drawn by readers. If you forget to read it with ++binary in the first place, you can always :e ++binary after loading. A concept which can be reused _consistently_ for the solution of all problems of this class probably can save a lot of future work time. I disagree that these problems are in the same class at all, … Two points of disagreeing (here and in the last but one paragraph below) besides agreeing in every other point maybe we can carry? … but the option for preventing such issues already exists: :e
Re: [patch] Respect file's EOL/NOEOL settings
Hi Pavel On 2015-06-24 Wednesday at 11:12 -0700 Pavel Samarkin wrote: … I was aiming at something different. Problem (as I see it): Vim is based on a fact that every text file is POSIX-compliant http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_206 which is not necessarily true on all platforms supported by Vim. Because of that, some scenarios are not currently supported by Vim. For example: Given: a text file in UTF-8 encoding, with BOM, with DOS line endings and without trailing EOL. Scenario: edit the contents of this file preserving all the attributes specified in Given. Real-world application: Microsoft Visual Studio project file in source control. Solutions that Vim provides currently: (1) regular :save - will add a trailing EOL; (2) :w ++binary - will replace CRLFs with LFs and lose BOM; (3) Solution from vim.wikia - http://vim.wikia.com/wiki/Preserve_missing_end-of-line_at_end_of_text_files - preserves CRLFs, but loses BOM and adds some overhead to the write process; (4) vim -b filename - displays CRs and BOM as deletable characters and disables some text-related options (like 'expandtab' and 'modeline'), so it's not always applicable; (5) some more complicated solutions I found on the Internet use external scripts (Python or Perl) to remove EOL from already saved file, which seems like an overkill to me - why to write and then remove something if we don't need it originally? Besides, Python and Perl are not always available on all platforms. Solution with my changes: :set reol :w In my particular case, adding the following to the .vimrc: au BufWrite *.csproj set respecteol will once and for all declare all *.csproj files as not necessarily POSIX-compliant. I'm sorry if I confused anyone by my original message and I hope that my point is more clear now. Yes, now it is clear for me. Thank you for the detailed explanation. I assume, the *.csproj files are originally written arbitrarily sometimes without and sometimes with trailing EOL, and can imagine, how nasty this is in the context of version management. Would the following alternative solution work for you? (1) Add a make target to your source code projects, which reads the *.csproj files and writes shadow files *.csproj.EOL_fixed, thereby just adding the trailing EOL if it misses. The “workhorse” utility performing this file writing could be vim, or if the number of your *.csproj files is very large it could be a perl script or a small dedicated binary compiled from a few lines of C code written by you. Optionally if you want to hide the shadow files in directory listings, the make target could set the file attribute “hidden” usually available on Microsoft filesystems. (2) Add a pre-commit hook to the version management system, which calls this make target. (3) Add the shadow files *.csproj.EOL_fixed to the files tracked by the version management system, and add an ignore rule for *.csproj files. Considerations: • Solving problems as close to their source as possible usually results in the least effort required in the long run. • Adding trailing EOL to already saved files probably requires less effort than an editor builtin solution. Human worktime is more expensive than cputime. Thus it is very common in the software industry to save human worktime by combining some already built software modules, rather than to create a new monolithic solution which solves just a single problem. The former usually increases the productivity of software engineers, whereas the latter usually increases the profit of certain big market players. https://en.wikipedia.org/wiki/Unix_philosophy • With (1) … (3) you would be better prepared for the future: If Microsoft some day will break its own standards (already did so regarding other software products), and some of your business partners exchanging *.csproj files with you use a new, backwards incompatible release of the software writing this files, you could save the need to buy a license for the new release just by modifying the utility used by the make target in (1). • You can read a few more considerations near the end of my mail replying to Ben in this thread. Question: Is the patch you propose really the long-term-least-effort solution, or can (1) … (3) or something similar provide a satisfying solution for you with less effort? -- Best regards, Roland Eggner -- -- You received this message from the vim_dev maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups vim_dev group. To unsubscribe from this group and stop receiving
Re: [patch] Respect file's EOL/NOEOL settings
Hi all, Just updating my patch after the latest merges to the vim's master. -- Best regards, Pavel -- -- You received this message from the vim_dev maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups vim_dev group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. respecteol_v3.runtime.patch Description: Binary data respecteol_v3.src.patch Description: Binary data
Re: [patch] Respect file's EOL/NOEOL settings
Hi all, Roland - thank you for the response. Unfortunately, there is no simple solution to the problems that you described: (1) viminfo files with register contents resulting from alternating fileencondings (2) The patch file resulting from the diff between old and new files after a command similar to iconv -f ISO-8859-1 -t utf-8 ... And it's not what I was trying to fix. Neither I was trying to fix gzip/bz2 decompression - you are right, autocmd-based solutions are stable enough. I was aiming at something different. Problem (as I see it): Vim is based on a fact that every text file is POSIX-compliant http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_206 which is not necessarily true on all platforms supported by Vim. Because of that, some scenarios are not currently supported by Vim. For example: Given: a text file in UTF-8 encoding, with BOM, with DOS line endings and without trailing EOL. Scenario: edit the contents of this file preserving all the attributes specified in Given. Real-world application: Microsoft Visual Studio project file in source control. Solutions that Vim provides currently: (1) regular :save - will add a trailing EOL; (2) :w ++binary - will replace CRLFs with LFs and lose BOM; (3) Solution from vim.wikia - http://vim.wikia.com/wiki/Preserve_missing_end-of-line_at_end_of_text_files - preserves CRLFs, but loses BOM and adds some overhead to the write process; (4) vim -b filename - displays CRs and BOM as deletable characters and disables some text-related options (like 'expandtab' and 'modeline'), so it's not always applicable; (5) some more complicated solutions I found on the Internet use external scripts (Python or Perl) to remove EOL from already saved file, which seems like an overkill to me - why to write and then remove something if we don't need it originally? Besides, Python and Perl are not always available on all platforms. Solution with my changes: :set reol :w In my particular case, adding the following to the .vimrc: au BufWrite *.csproj set respecteol will once and for all declare all *.csproj files as not necessarily POSIX-compliant. I'm sorry if I confused anyone by my original message and I hope that my point is more clear now. Thank you. -- Best regards, Pavel -- -- You received this message from the vim_dev maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups vim_dev group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [patch] Respect file's EOL/NOEOL settings
Am 2015-06-23 01:21, schrieb Pavel Samarkin: Hi, Issue: Vim adds EOL to the end of each text file Although my change might seem controversial, it looks like a lot of people (including myself) really want it, especially when using Vim on non-Unix systems (e.g. Windows). You can notice it here: http://vim.wikia.com/wiki/Preserve_missing_end-of-line_at_end_of_text_files http://www.vim.org/scripts/script.php?script_id=4550 and in several questions on stackoverflow: http://stackoverflow.com/questions/1050640 http://stackoverflow.com/questions/4133501 http://stackoverflow.com/questions/14171254 http://askubuntu.com/questions/13317 The solution suggested in Vim FAQ (5.4) http://vimdoc.sourceforge.net/htmldoc/vimfaq.html includes some side-effects like inability to use 'expandtab', 'textwidth' and some other options (incl. Unicode BOM). My proposed change introduces the new option called 'respecteol' (that's the best name I could come up with) which affects the EOL at the end of file the same way as the 'binary' option does, while not changing any other aspects. The option is local to a buffer, so it can be set depending on a filetype. Please let me know if this can be merged. Thank you. +1 This has bitten me just recently, I worked on a team repository and making some changes to the text would lead to the addition of EOL to some files, which was in this case unexpected and unintended. It took me a while to fix this. So adding an option to make the 'eol' behaviour work better was on my list of changes to do next. So yes, I would definitely appreciate such an option. BTW, it would be good to add this option to $VIMRUNTIME/optwin.vim and having tests would also help. Best, Christian -- -- You received this message from the vim_dev maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups vim_dev group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [patch] Respect file's EOL/NOEOL settings
+1 -- Yukihiro Nakadaira - yukihiro.nakada...@gmail.com -- -- You received this message from the vim_dev maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups vim_dev group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [patch] Respect file's EOL/NOEOL settings
I would definitely appreciate such an option. BTW, it would be good to add this option to $VIMRUNTIME/optwin.vim and having tests would also help. Christian - thank you for the quick response. See attachment for the updated patch. I could not find a good test to modify, so I created a new one (test_reol) and added it to all makefiles. I tested it on Windows 7 and Mac OS X. Also, I updated optwin.vim and syntax/vim.vim. I split my patch in two files - one for source code changes, one for the changes to runtime, for easier review. -- Best regards, Pavel -- -- You received this message from the vim_dev maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups vim_dev group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. respecteol_v2.src.patch Description: Binary data respecteol_v2.runtime.patch Description: Binary data
[patch] Respect file's EOL/NOEOL settings
Hi, Issue: Vim adds EOL to the end of each text file Although my change might seem controversial, it looks like a lot of people (including myself) really want it, especially when using Vim on non-Unix systems (e.g. Windows). You can notice it here: http://vim.wikia.com/wiki/Preserve_missing_end-of-line_at_end_of_text_files http://www.vim.org/scripts/script.php?script_id=4550 and in several questions on stackoverflow: http://stackoverflow.com/questions/1050640 http://stackoverflow.com/questions/4133501 http://stackoverflow.com/questions/14171254 http://askubuntu.com/questions/13317 The solution suggested in Vim FAQ (5.4) http://vimdoc.sourceforge.net/htmldoc/vimfaq.html includes some side-effects like inability to use 'expandtab', 'textwidth' and some other options (incl. Unicode BOM). My proposed change introduces the new option called 'respecteol' (that's the best name I could come up with) which affects the EOL at the end of file the same way as the 'binary' option does, while not changing any other aspects. The option is local to a buffer, so it can be set depending on a filetype. Please let me know if this can be merged. Thank you. -- Best regards, Pavel -- -- You received this message from the vim_dev maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups vim_dev group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. respecteol.patch Description: Binary data
