Re: [Monotone-devel] Re: line endings with 0.31

[J Decker]

edit and wordpad work for builtin stuff

I have been using FTE ( http://fte.sourceforge.net/ ) for the last several
years... it shows the carriage return character... visual studio, actually
the only one that doesn't behave well is notepad

On 11/22/06, Brian May <[EMAIL PROTECTED]> wrote:


> "J" == J Decker  writes:

J> On a note from some windows programmer that appreciated the
J> conversions when checking out linux-like sources to have
J> 'notepad' work as a browser; While some days I can agree, in
J> the end, one must make sacrifices and call a goat a goat, and
J> know that you need to use a smarter tool than notepad to deal
J> with text files.

Any there any sane editors for Windows that only support DOS text file
format?

(no - I do not consider notepad a sane editor anymore).
--
Brian May <[EMAIL PROTECTED]>


___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

Re: [Monotone-devel] Re: line endings with 0.31

[Brian May]

> "J" == J Decker  writes:

J> On a note from some windows programmer that appreciated the
J> conversions when checking out linux-like sources to have
J> 'notepad' work as a browser; While some days I can agree, in
J> the end, one must make sacrifices and call a goat a goat, and
J> know that you need to use a smarter tool than notepad to deal
J> with text files.

Any there any sane editors for Windows that only support DOS text file
format?

(no - I do not consider notepad a sane editor anymore).
-- 
Brian May <[EMAIL PROTECTED]>


___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

Re: [Monotone-devel] Re: line endings with 0.31

[J Decker]

I have to state a vote.

I had lengthy discussions with CVS on similar topics, and resorted to
hacking my own CVS to work properly, that is, if a text file has \r\n, keep
\r\n, do not convert to \r\r\n.
If a text file has \r or \n in some sequence [\r\n]* this is an end of line,
if \n's exist multiply, there are blank lines.  (unless you're on a MAC that
uses \r for newlines?!)

If a file is checked in and developed by a developer which is using both
windows and linux, her tools will of course support both cases for files in
both environments which must be developed in either.

There is little requirement for conversion, and eventually, conversion is
self recursive.

On a note from some windows programmer that appreciated the conversions when
checking out linux-like sources to have 'notepad' work as a browser; While
some days I can agree, in the end, one must make sacrifices and call a goat
a goat, and know that you need to use a smarter tool than notepad to deal
with text files.

Linux tools rarely care about the presence or absence of \r's... they're
redundant other than the classic teletype need to send a carriage return
before the line feed.  If you sent the line feed first, then the carriage
return wouldn't have enough time to happen before processing a the next
character, and the letter would strike before the head reached the return
point.  Sometimes I wonder if people actually learn from history at all...
Modern equipment rarely even requires a carriage return, being of a digital
nature, you cannot gain cheap effect like BOLD by sending
'BOLD\rBOLD\rBOLD really ...' to a printer device... thermal and laser
printers will not gain any effect because there is no ribbon.

I actually cannot think of a sane reason that I would ever want a line feed
without returning to the start 

In the end, I'm sure everyone will disagree, the developers will say it
works good enough, and it'll remain broke, lest the break what already works
fixing something that isn't broken.

Unless you expect users to be lame and use notepad to edit text files, never
open a file with fopen("file", "t" );  but always specify the flag
fopen(file, "b");  even under linux environments this does not hurt.  If you
mean text, say text, always read binary and disregard \r as whitespace.
disregard also \n as whitespace.

Nothing seems to have a problem not converting tab charaters into spaces.

Jim
___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

Re: [Monotone-devel] Re: line endings with 0.31

[J Decker]

The logic for determining line endings within reasonable limits is entirely
feasible in practice as well as theory.

On 11/21/06, Richard Levitte - VMS Whacker <[EMAIL PROTECTED]> wrote:


I've been thinking about this issue a number of times, and have
discussed it as well.

A thought that came to my mind a while ago was that we could really
make this really simple:

- always store the file unchanged, and for text files (according to
   some criteria), store the local EOL format in an attribute.
   mtn:eol = CR, mtn:eol = LF or mtn:eol = CRLF
- for text files, always convert to the local EOL format on
   extraction.
   So if something was stored with mtn:eol = CR (on a Mac, typically),
   all CRs would be converted to LF on a Unix system, or to CRLF on a
   Windows system.

One might wonder what happens during a merge of two revisions where
the same file is stored with different EOL formats in the different
revisions.  Quite simple, since they get converted to the local format
on extraction, I'm assuming that what's merged is the result after
extraction.

Now, please, before you start reacting, I *know* that the whole file
isn't stored, only the delta.  Unfortunately, I know diddly over squat
about xdelta, so a question is if storing different EOL formats at
different times creates more trouble than it's worth.

On the plus side, it means that storing a file (or the delta) will
always happen without conversion, whether it's binary or text, and the
user can remember to add the EOL attribute as an afterthought, at any
time.

If we would do this rigorously, we wouldn't need any hooks either,
really, unless someone really wants to override the default function.

Now, the next, and somewhat related thing (or can of worms) is
character code conversion.

Cheers,
Richard

-
Please consider sponsoring my work on free software.
See http://www.free.lp.se/sponsoring.html for details.

--
Richard Levitte [EMAIL PROTECTED]
http://richard.levitte.org/

"When I became a man I put away childish things, including
the fear of childishness and the desire to be very grown up."
-- C.S. Lewis


___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

Re: [Monotone-devel] Re: line endings with 0.31

[Thomas Moschny]

On Tuesday 21 November 2006 23:51, Brian May wrote:
> Recently we got a sourceforge project converted from CVS to
> subversion.
>
> The process automatically checked in the files using text mode
> conversions.
>
> The result was that a tar.gz file (yes, having this in the project is
> somewhat weird) was corrupted.

It would have been useful to read that
  http://cvs2svn.tigris.org/faq.html#eol-fixup
before doing the conversion.

- Thomas


___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

Re: [Monotone-devel] Re: line endings with 0.31

[hendrik]

On Wed, Nov 22, 2006 at 10:05:13AM +1100, Brian May wrote:
> > "hendrik" == hendrik  <[EMAIL PROTECTED]> writes:
> 
> hendrik> Just for historical record, this famous Microsoft
> hendrik> "incompatibility" is actually a point in which they
> hendrik> followed the official standard.  ASCII was designed
> hendrik> without a newline symbol; instead, it had a
> hendrik> carriage-return -- which returned the teletype carriage
> hendrik> to the beginning of the current line, permitting it to be
> hendrik> overwritten, and a line feed character, which moved to
> hendrik> the next line while leaving the carriage alone (thus
> hendrik> effectively moving a cursor straight down).
> 
> Microsoft, also, initially adopted the standard that ^Z would mark the
> end of the file. Apparently this was required (IIRC), because early
> filesystems didn't store the file length anywhere, so the ^Z marker
> was the only way to tell a reader when it got to the end of the file.

The cr-lf convention was an ASA/ISO standard in the formal sense.
The ~Z convention never was a standard.  Just the way it was done on 
some ancient OS.  The actual official standard meaning for ~Z, 
incidentally, was SUB; i.e, "substitute", whatever that meant.  Probably 
not EOD, which may have been more appropriately coded as ~D, the 
standard code for EOT (end of transmission).

Maybe Microsoft got to use the actual cr-lf standard by accident.
I suspect ~Z was chosen because Z is tha last letter of the alphabet.

> 
> Fortunately, this standard is effectively dead now, and ^Z is only
> used when receiving data from the console.

Unix uses ~D for this when reading from the console.

> 
> I think it is only a matter of time before the Windows text format
> dies too.
> 
> Curiously, wordpad on Windows can read Unix format files, but notepad
> can't.
> -- 
> Brian May <[EMAIL PROTECTED]>
> 
> 
> ___
> Monotone-devel mailing list
> Monotone-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/monotone-devel


___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

Re: [Monotone-devel] Re: line endings with 0.31

[hendrik]

On Wed, Nov 22, 2006 at 10:13:08AM +1100, Brian May wrote:
> > "hendrik" == hendrik  <[EMAIL PROTECTED]> writes:
> 
> hendrik> In monotone, I suggest that a file that has been
> hendrik> character-converted on checkout have its line-end codingw
> hendrik> reverted on checkin, on a line-by-line basis.  Thus only
> hendrik> when the user explicitly edits line ends will the
> hendrik> end-of-line coding be changed in the repository.  This
> hendrik> would have the effect that if massive damage is done to a
> hendrik> true binary file if it is mistakenly line-end-converted,
> hendrik> the damage would be mostly undone on subsequent checkin.
> 
> hendrik> It is probably not the most convenient load to dump on
> hendrik> the diff engine, though I can imagine algorithms.
> 
> The big problem I see with this is if a user tries to merge two
> changed versions of a file, which have different end-of-line codes.
> 
> Or maybe one user fixes the line ending attribute for a file and
> another user updates the file to use another line ending. What happens
> when one the users tries to merge the results?

Some chaotic set of line endings.  But if conversion to a local 
convention is done at checkout, it's probably not fatal.  Just a lt of 
unnecessary differences.

> 
> This could get complicated, and might even end up in the very
> corruption problem you are trying to avoid anyway (e.g. if the file
> being merged is binary but marked as ascii in one/more of the
> revisions).

Well. I wasn't actually *intending* bunary files to be 
line-end-converted.  Just pointing out that such a mistake might not 
necessarily be instantly fatal.

-- hendrik


___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

[Monotone-devel] Re: line endings with 0.31

[Bruce Stephens]

Larry Hastings <[EMAIL PROTECTED]> writes:

[...]

> I'm not sure what SVN using two separate values (one as a
> mime-type!)  buys them.

I believe the theory is that you could use different merging programs
for some types.  text/xml is the usual example, where you might have
an XML tree-merging program that worked better for such things than a
text-based merger.  I've no idea how useful that's been in reality.

> I think one setting with three possible values would be best: "it's
> binary", "it's text but don't do eol conversion", and "it's text,
> please do eol conversion".  From the above I take it it's possible
> to declare a file as binary-with-eol-conversions, which strikes me
> as an unintended consequence waiting to happen.

Yeah, probably.  However, I'd guess there's some value in storing EOL
properties separately from information you use when choosing how to
merge.  Whether MIME-type is the right thing to use for the latter I'm
not sure, but it's hard to argue strongly against.


___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

[Monotone-devel] Re: line endings with 0.31

[Bruce Stephens]

Brian May <[EMAIL PROTECTED]> writes:

>> "Bruce" == Bruce Stephens <[EMAIL PROTECTED]> writes:
>
> Bruce> Sure, that may happen.  That's just a bug, though, isn't
> Bruce> it, so you commit a revision containing the changes to that
> Bruce> one file (including attribute change) as a child of the
> Bruce> original, and merge it wherever necessary?
>
> Out of curiosity, what happens in Monotone if you try to merge two
> changed versions of a binary file?

I think the normal lua hook is called.  Presuming that that fails, the
merge fails.


___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

[Monotone-devel] Re: line endings with 0.31

[Bruce Stephens]

Richard Levitte - VMS Whacker <[EMAIL PROTECTED]> writes:

[...]

> Now, please, before you start reacting, I *know* that the whole file
> isn't stored, only the delta.  Unfortunately, I know diddly over
> squat about xdelta, so a question is if storing different EOL
> formats at different times creates more trouble than it's worth.

Not sure.  It would make the deltas bigger (if the delta represented a
single-character change you might need a separate copy for every line
(perhaps plus a record for the EOL) rather than two big copies plus
the changed characters).  I'm not sure whether that would be
significant over the lifetime of a repository.  It might well depend
on usage patterns.

I must admit it feels like the wrong way to do it.  Better (I'd have
thought) to store some canonical form in the repository and let the
clients do conversion.  What's currently missing is a way to specify
what the canonical form ought to be like: whether the file's text or
binary, whether it has special line-ending requirements (like some VS
files seem to have).  Subversion splits this: svn:mime-type specifies
whether a file's binary or not, and svn:eol-style lets you change the
line ending.
,
.

[...]



___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

Re: [Monotone-devel] Re: line endings with 0.31

[Brian May]

> "hendrik" == hendrik  <[EMAIL PROTECTED]> writes:

hendrik> Just for historical record, this famous Microsoft
hendrik> "incompatibility" is actually a point in which they
hendrik> followed the official standard.  ASCII was designed
hendrik> without a newline symbol; instead, it had a
hendrik> carriage-return -- which returned the teletype carriage
hendrik> to the beginning of the current line, permitting it to be
hendrik> overwritten, and a line feed character, which moved to
hendrik> the next line while leaving the carriage alone (thus
hendrik> effectively moving a cursor straight down).

Microsoft, also, initially adopted the standard that ^Z would mark the
end of the file. Apparently this was required (IIRC), because early
filesystems didn't store the file length anywhere, so the ^Z marker
was the only way to tell a reader when it got to the end of the file.

Fortunately, this standard is effectively dead now, and ^Z is only
used when receiving data from the console.

I think it is only a matter of time before the Windows text format
dies too.

Curiously, wordpad on Windows can read Unix format files, but notepad
can't.
-- 
Brian May <[EMAIL PROTECTED]>


___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

Re: [Monotone-devel] Re: line endings with 0.31

[Brian May]

> "Graydon" == Graydon Hoare <[EMAIL PROTECTED]> writes:

Graydon> Woah! Did we really leave it like this? That's
Graydon> terrible. We added an attribute to files called
Graydon> mtn:manual_merge that controls whether the merger
Graydon> believes it can do line-merging on the file or whether
Graydon> the file should be treated as binary. Can't we assume
Graydon> that mtn:manual_merge implies "don't do the linesep /
Graydon> charset conversion on i/o"? Or can we rephrase the
Graydon> attribute as mtn:blob, or such? Users should not have to
Graydon> muck about with this linesep conversion function stuff,
Graydon> if possible.

I have to agree myself, its not very good. Another option would be if
the file already exists, and you can automatically parse it using some
algorithm to guess if it is binary or not (perhaps with the file
command).

I personally prefer the attribute approach, although, admittedly,
there is the risk that you will corrupt a binary file by checking it
in as ascii.

Another option would be to replace the lua function with one that
reads in a list of regular expressions from an external file. I don't
think this would be as good because it wouldn't set per each
repository.
-- 
Brian May <[EMAIL PROTECTED]>


___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

[Monotone-devel] Re: line endings with 0.31

[William Uther]

Hi,
  I'm newish here, so my opinion might not count for much, but I  
think this is one area where subversion got things right.  I highly  
recommend that people read about subversion's system before making up  
their minds.  http://svnbook.red-bean.com/nightly/en/ 
svn.advanced.props.html  In particular, svn:mime-type svn:eol-style  
and svn:keywords are worth knowing about.


  - Having the mime-type stored can be useful.  You could have a  
hook for inferring this.  By default, subversion leaves it unset.   
This property is used to decide what merge algorithm to use.  i.e. it  
is the mechanism svn uses to mark binary files that shouldn't be  
merged.  An unset entry is considered text, and so will use the  
standard textual merge algorithm.  Anything with text/*  in it is  
also considered text.  There was talk of having plugable merge  
algorithms (e.g. tree merge algorithms for XML files), but I don't  
know if that ever happened.


  - Subversion has a second property for eol conversion.  It can  
have one of five values: "CR", "LF", "CRLF", "native" or unset.


 Unset is the default case, and means no eol conversion.  By  
default, your data doesn't get munged.


 "CR", "LF" and "CRLF" are pointers to svn to normalise the line  
endings before checking in.  These settings are there so that stupid  
editors don't cause people to have spurious diffs.


 "native" tells subversion to convert to whatever the standard line  
endings are for the local platform.  Before checkin the file is  
converted back to "LF" and so that is the eol format in the  
repository.  On checkout the "LF"s are converted to the local standard.


There is a third property it is worth bringing up in this context.   
That is subversion's 'keywords' property.  The keyword expansion  
happens at the same time as "native" line-ending conversion.  This  
property contains a list of keywords to be expanded in the text.  By  
default it is empty and no expansion occurs.  There is a list of  
keywords, "Date", "Revision", "Author", "HeadURL" which would be  
something like "Branch" in mtn, and "Id" which is a short string  
combining parts of the others.  On checkout, when the 'native' line  
ending conversion happens, any keywords between $s are expanded.   
e.g. $Id: $ might become $Id: calc.c 148 2002-07-28 21:30:43Z sally  
$.  On checkin, when the line ending conversion happens, any keywords  
are contracted back to the simple form: $Id: $.  By storing canonical  
forms in the repos, there is no special merge algorithm required.


This system works.  By default it doesn't touch your data.  It allows  
people on different platforms to work together, each with their local  
line endings.  It allows line endings to be forced to one value if  
you want them there and someone keeps corrupting them.  It has a  
canonical form in the repository so that the merge algorithm is kept  
simple.  It doesn't confuse the concepts of 'should I convert line  
endings' and 'should I use line-based merge'.


The one place where you might get a large diff is if you change line- 
ending settings.  e.g. you have a file committed with "CRLF" line  
endings and no line ending property.  In that case, svn will not be  
doing any conversion for you.  If you then set the line ending  
property to "native", you will get a one-time diff that includes the  
property change, and the file-diff that changes all the line-endings  
to "LF"s.  If this is done at import time or soon afterwards it isn't  
a problem.  Done later, it is more of a problem :( - you'd need fancy  
merge algorithms to merge past the line-ending change.


Of course, this is only a problem if the repos wasn't already in the  
right format.  Adding a "native" setting to a document made on Unix  
with "LF"s is not a problem at all, neither is adding a "CRLF"  
setting to document already using "CRLF"s.


Be well,

Will:-}



___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

Re: [Monotone-devel] Re: line endings with 0.31

[Timothy Brownawell]

On Tue, 2006-11-21 at 10:34 -0500, [EMAIL PROTECTED] wrote:

> By the way, *does* monotone use a line-by-line diff?
> I thought it used a character-by-character diff based on compression 
> technology.

It uses a normal line-based diff for merging, and uses xdelta
(character-based diff, sorta) for storage compression.

-- 
Timothy

Free (experimental) public monotone hosting: http://mtn-host.prjek.net



___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

[Monotone-devel] Re: line endings with 0.31

[Bruce Stephens]

Larry Hastings <[EMAIL PROTECTED]> writes:

[...]

> FWIW, I'm a Windows programmer, so I actually /use/ eol convention,
> so if I might chime in with my opinion...
>
> I agree with Nathaniel: eol conversion should be /disabled/ by
> default, and should be /enabled/ with a setting.
> Principle-of-least-surprise there.

I'd have thought the least surprising would be to have the conversion
enabled for text files.  That means you could pull net.venge.monotone
and checkouts would have their text files sanely readable on Windows,
and files checked in on Windows and checkout out on Unix would be
readable.

I guess expectations would be different if you worked entirely on
Windows: then it would make sense to just store binary?

[...]



___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

Re: [Monotone-devel] Re: line endings with 0.31

[Brian May]

> "hendrik" == hendrik  <[EMAIL PROTECTED]> writes:

hendrik> In monotone, I suggest that a file that has been
hendrik> character-converted on checkout have its line-end codingw
hendrik> reverted on checkin, on a line-by-line basis.  Thus only
hendrik> when the user explicitly edits line ends will the
hendrik> end-of-line coding be changed in the repository.  This
hendrik> would have the effect that if massive damage is done to a
hendrik> true binary file if it is mistakenly line-end-converted,
hendrik> the damage would be mostly undone on subsequent checkin.

hendrik> It is probably not the most convenient load to dump on
hendrik> the diff engine, though I can imagine algorithms.

The big problem I see with this is if a user tries to merge two
changed versions of a file, which have different end-of-line codes.

Or maybe one user fixes the line ending attribute for a file and
another user updates the file to use another line ending. What happens
when one the users tries to merge the results?

This could get complicated, and might even end up in the very
corruption problem you are trying to avoid anyway (e.g. if the file
being merged is binary but marked as ascii in one/more of the
revisions).
-- 
Brian May <[EMAIL PROTECTED]>


___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

Re: [Monotone-devel] Re: line endings with 0.31

[Brian May]

> "Richard" == Richard Levitte <- VMS Whacker <[EMAIL PROTECTED]>> writes:

Richard> format on extraction.  So if something was stored with
Richard> mtn:eol = CR (on a Mac, typically), all CRs would be

I might be mistaken, but I believe this file format is obsolete as of
Mac OS X, now it is based on BSD. Now Mac OS uses the Unix format by
default.

There might still be people using earlier versions of Mac OS though...
-- 
Brian May <[EMAIL PROTECTED]>


___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

Re: [Monotone-devel] Re: line endings with 0.31

[Brian May]

> "Bruce" == Bruce Stephens <[EMAIL PROTECTED]> writes:

Bruce> Sure, that may happen.  That's just a bug, though, isn't
Bruce> it, so you commit a revision containing the changes to that
Bruce> one file (including attribute change) as a child of the
Bruce> original, and merge it wherever necessary?

Out of curiosity, what happens in Monotone if you try to merge two
changed versions of a binary file?
-- 
Brian May <[EMAIL PROTECTED]>


___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

[Monotone-devel] Re: line endings with 0.31

[Bruce Stephens]

William Uther <[EMAIL PROTECTED]> writes:

> Hi,
>   I'm newish here, so my opinion might not count for much, but I
> think this is one area where subversion got things right.  I highly
> recommend that people read about subversion's system before making up
> their minds.  http://svnbook.red-bean.com/nightly/en/
> svn.advanced.props.html  In particular, svn:mime-type svn:eol-style
> and svn:keywords are worth knowing about.

Yes, that's been mentioned several times.  (First mention was last
year:
.)
There's a bug mentioning it, too:
.

My guess is it just hasn't bothered anyone enough to actually fix it
yet.  (Further, I suspect the first person to write the code,
documentation, and tests, probably has a good chance to choose how it
works.)

[...]



___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel