Re: [all][POLL] what files to fixcrlf for windows distributions
- Original Message - From: Stephen Colebourne [EMAIL PROTECTED] Sent: Sunday, December 04, 2005 12:56 AM I was the one who initially raised this issue. The reason I raised it is that most _users_ run Windows. And the default text editor on Windows is Notepad which doesn't handle LF. Basically if I download a release and oopen the release notes, or project xml then I want CRLF, otherwise I have to close the file, find wordpad and reopen it. I agree with Martin on this and IMO the simplest solution is to make the release notes HTML - besides resolving this line ending issue, they also look alot better than plain text in notepad or wordpad. Niall - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [all][POLL] what files to fixcrlf for windows distributions
I don't think it is feasible to patch the maven dist plugin or the hand-rolled ant scripts to inspect .svn/props for each file and then change line endings accordingly. I also verified that distros created on Windows are in fact shipping CRLF line endings on all files that have eol=native in their svn props. A recent example is the 1.1 IO release. To me, this is a problem, since it adds to the size of the distros and makes the contents of the distro dependent on the OS used to build it. I see three realistic options to fix this, none of which are ideal, listed in order of how I view them. Other ideas - or statements that this is a non-issue - welcome. I have already done the work for 2 and will submit the patch if that is the preferred solution. 1. Change svn props to get rid of eol=native and specify eol=crlf for .txt. IIUC the docs, lf is the default, so this should make all sources default to lf. It does force Windows users to correctly set line endings on patches, commits, etc. This is essentially going back to the way things were in cvs. 2. Patch the maven dist plugin to put lf line endings on all of the file types listed as recommended to have eol=native on the apache svn pages and crlf on .txt files. I would make both properties configurable in the plugin, with the default for the first being empty and the second .txt The problem with this is getting all of the extents fixed in the lf case. 3. Move to centralized release builds, running all release builds from minotaur. Would have to work through the issues here. Phil - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [all][POLL] what files to fixcrlf for windows distributions
On 12/3/05, Phil Steitz [EMAIL PROTECTED] wrote: I don't think it is feasible to patch the maven dist plugin or the hand-rolled ant scripts to inspect .svn/props for each file and then change line endings accordingly. I also verified that distros created on Windows are in fact shipping CRLF line endings on all files that have eol=native in their svn props. A recent example is the 1.1 IO release. To me, this is a problem, since it adds to the size of the distros and makes the contents of the distro dependent on the OS used to build it. I really don't think this is an issue. It's been this way for _years_, without any problems that I've been aware of. I see no reason to suddenly declare it's an issue and go to all the lengths being discussed to fix it. As for the size, given that the distributions are compressed, and text compression algorithms are very good indeed, I seriously doubt that it's going to make any significant difference. I see three realistic options to fix this, none of which are ideal, listed in order of how I view them. Other ideas - or statements that this is a non-issue - welcome. I have already done the work for 2 and will submit the patch if that is the preferred solution. 1. Change svn props to get rid of eol=native and specify eol=crlf for .txt. IIUC the docs, lf is the default, so this should make all sources default to lf. It does force Windows users to correctly set line endings on patches, commits, etc. This is essentially going back to the way things were in cvs. -1 First off, as I pointed out elsewhere, this is *not* going back to the way things were with CVS. We had exactly the same issue when we were using CVS, and nobody seemed to want to fix it then. In fact, we've had this same situation since Commons was born. The Windows CVS command line client defaults to CR-LF on checkout, as does WinCVS (which also has a checkbox you need to explicitly check if you want Unix line ends). The more serious issue, though, is that this can end up causing incorrect line number reporting in diffs, exceptions, and in other cases, which is going to make our job as component developers more painful than it already is. 2. Patch the maven dist plugin to put lf line endings on all of the file types listed as recommended to have eol=native on the apache svn pages and crlf on .txt files. I would make both properties configurable in the plugin, with the default for the first being empty and the second .txt The problem with this is getting all of the extents fixed in the lf case. 3. Move to centralized release builds, running all release builds from minotaur. Would have to work through the issues here. That would hurt the whole of Commons, IMO. It would force all of the Windows developers to learn to do development - or at least building releases - in a new environment, which will be more painful. This at a time where we really need to make the release process as painless as possible, so that more people are willing to take it on. -- Martin Cooper Phil - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [all][POLL] what files to fixcrlf for windows distributions
Thanks, Martin. I stand corrected then and will shut up about this. To close this out, I guess we are agreeing that we have no standard for line endings and will make no attempt to make them consistent. I did not know the Windows cvs client was also converting the files, so thought this was a new problem introduced by svn. Sorry for the noise. I also agree (strongly) that we need to make releasing easier. Sorry if I this discussion seemed like negative progress. Phil - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [all][POLL] what files to fixcrlf for windows distributions
Has everyone been following the bit on http://www.apache.org/dev/version-control.html? -- Committers will need to properly configure their svn client. One particular issue is OS-specific line-endings for text files. When you add a new text file, especially when applying patches from Bugzilla, first ensure that the line-endings are appropriate for *your* system, then do ... svn add test.txt svn propset svn:eol-style native test.txt Your svn client can be configured to do that automatically for some common file types. Add the list to your ~/.subversion/config [http://www.apache.org/dev/svn-eol-style.txt] file. However, you should still pay attention to the messages from your svn client when you do 'svn commit'. -- Just checking :) Hen On 12/3/05, Phil Steitz [EMAIL PROTECTED] wrote: Thanks, Martin. I stand corrected then and will shut up about this. To close this out, I guess we are agreeing that we have no standard for line endings and will make no attempt to make them consistent. I did not know the Windows cvs client was also converting the files, so thought this was a new problem introduced by svn. Sorry for the noise. I also agree (strongly) that we need to make releasing easier. Sorry if I this discussion seemed like negative progress. Phil - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [all][POLL] what files to fixcrlf for windows distributions
Phil Steitz wrote: Thanks, Martin. I stand corrected then and will shut up about this. To close this out, I guess we are agreeing that we have no standard for line endings and will make no attempt to make them consistent. I did not know the Windows cvs client was also converting the files, so thought this was a new problem introduced by svn. Sorry for the noise. I was the one who initially raised this issue. The reason I raised it is that most _users_ run Windows. And the default text editor on Windows is Notepad which doesn't handle LF. Basically if I download a release and oopen the release notes, or project xml then I want CRLF, otherwise I have to close the file, find wordpad and reopen it. Now I understand the argument that our users are developers and shoud know how to change file associations, and so on. But we here are probably at the higher end of Java/programming knowledge. I want us to cater for users who are more Junior. Also, Notepad is very quick to start and open a small file, which is why some of use (ie. me) choose to leave the txt to notepad association. This discussion however, seems to have yielded no real solution to this. I suspect that what is actually happening is that more commons release managers now use Unix than previously, thus the problem is more apparant. Currently, I setup a custom ant script for each project I release, and never use maven for building. That way I can fix crlf for the root level text files in the zip and lf for the tar. However, as this discussion didn't reach agreement, I shall have to keep quiet on this subject in future release votes. Stephen - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [all][POLL] what files to fixcrlf for windows distributions
Hi Stephen,
Just add something like this in the maven.xml (the test is pretty useless probably..):
j:if test=${os != 'Linux'}
ant:fixcrlf srcdir=. includes=${abatchfile} eol=dos/
/j:if
And of course you should add xmlns:ant=jelly:ant to have the ant namespace
available..
Don't know from the top of my head which target to hijack when using the dist
target (this was using a completely custom made dist target)
Mvgr,
Martin
Currently, I setup a custom ant script for each project I release, and
never use maven for building. That way I can fix crlf for the root level
text files in the zip and lf for the tar.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: [all][POLL] what files to fixcrlf for windows distributions
On 12/3/05, Martin van den Bemt [EMAIL PROTECTED] wrote:
Hi Stephen,
Just add something like this in the maven.xml (the test is pretty useless
probably..):
j:if test=${os != 'Linux'}
ant:fixcrlf srcdir=. includes=${abatchfile} eol=dos/
/j:if
And of course you should add xmlns:ant=jelly:ant to have the ant namespace
available..
Don't know from the top of my head which target to hijack when using the dist
target (this was using a completely custom made dist target)
Which is what you need to do to make that work. You can't pregoal any
of the subgoals of dist to make this work, unfortunately. Requires a
patch to the dist plugin itself, which has been applied to svn head.
What is in svn head now does not allow the filter to be configured. I
will submit another patch that allows both crlf and lf filters to be
configured. We can choose to ignore the lf conversion if that is the
consensus, but apply crlf to .txt and other files for zips.
Phil
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: [all][POLL] what files to fixcrlf for windows distributions
On 27/11/05, Phil Steitz [EMAIL PROTECTED] wrote:
snip
I am working on a patch to the maven dist plugin that makes the crlf
filter property configurable. Arnaud pointed out in another thread
that there should probably a similar filter applied to make sure lf
line endings are used in the tar.gz distros. My inclination is to
apply this to *all* files in the tar.gz distros by default. It occurs
Won't this cause corruption of .jar, .class, .jpg, etc files?
Doh! Good point. Would need to exclude at least these.
to me now that because of the eol=native svn props, RMs who cut
releases on Windows are likely making tars with crlf line endings
(since svn will convert files to crlf on checkout). Note that this
applies to ant as well as maven-built distros.
And RMs who use Unix will be creating zips with LF line endings...
Yes, IIUC, that is the consequence of using eol=native. Having the
file contents of distros different depending on the OS used to cut the
release does not seem like a good thing to me. Could be I have
Agreed. But we are talking about different contents for the zip and
tar.gz releases.
something wrong here. Maybe someone who has both unix and Windows
boxes set up can test and see if the the Windows-built distros (from
Windows checkouts) end up with CRLF everywhere (at least for files
with eol=native).
[Are there any who use Macs?]
I will make the unix filter configurable as well, but want feedback on
what the default should be, as well as what our policy should be in
commons on line endings in tar.gz as well as zip distros.
Ideally the default would depend on the svn:eol-type.
i.e. if svn:eol-type=native, then convert the file for the appropriate
native format, otherwise leave alone.
Dunno whether this can be made to work or not, though.
But this would still result (again, assuming I have it right) in
different line endings in distros depending on what OS was used to cut
the release - unless we all set up Windows boxes to cut the zip
releases and *nix boxes for the tarballs ;-)
Hopefully not - I assumed that the maven/ant task would change the
line-endings as needed, so as to create an archive that was the same
as if it had been created on an OS with a different native setting.
That's why it needs to know what SVN has done.
It's possible that a particular file type may have different endings
for different files.
For example on JMeter, the old JMX files were CRLF on Windows, but the
new JMX files are LF on all OSes.
Is this specified in the svn eol props for these files?
Yes (or it should be).
It now occurs to me that assuming my understanding of how eol=native
is correct, we should be fixing crlf for non-txt files in zip distros
I assume you mean files with an extension that is not .txt here ...
as well, since if these are cut using Windows, e.g. the java source
files will have CRLF line endings. So, in terms of properties, it
seems we need
maven.dist.lf.filter=**/*.java, **/*.txt, all other non-binary file
specs, i.e. all files that could have eol=native
maven.dist.crlf.filter=**/*.txt (plus whatever else we agree on)
and have the plugin do the following:
For tar.gz distributions, run fixcrlf with eol=lf and
includes=${maven.dist.lf.filter}
For zip distributions, run fixcrlf first with eol=lf and
includes=${maven.dist.lf.filter} and then again with eol=crlf and
includes=${maven.dist.crlf.filter}.
If one can use svn:eol-type, then this can all be automated, as the
eol-type will determine whether or not the line endings have been
changed by SVN.
I guess another alternative would be to agree to cut all distros on
unix, maybe even using minotaur to do the release builds (I know this
would have some practical problems associated with it now). Then all
we would need is the first property, assuming we want lf line endings
everywhere else (IIRC, in the cvs days, that was the standard).
Sorry if I have misunderstood the way the svn client works or if
others view this as a non-issue. I just want to patch the plugin so
that we can get consistent distros built using maven.
And Ant, ideally.
Phil
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: [all][POLL] what files to fixcrlf for windows distributions
On 27/11/05, Phil Steitz [EMAIL PROTECTED] wrote: On 11/15/05, Stephen Colebourne [EMAIL PROTECTED] wrote: Mario Ivankovits wrote: [X] .txt [ ] .java [ ] .properties [ ] .xml [ ] .css [ ] .mf My preference is for all ASCII files not in the src/java or src/text folders to be converted. Note that I doubt [collections] or [io] do this fully yet. Stephen I am working on a patch to the maven dist plugin that makes the crlf filter property configurable. Arnaud pointed out in another thread that there should probably a similar filter applied to make sure lf line endings are used in the tar.gz distros. My inclination is to apply this to *all* files in the tar.gz distros by default. It occurs Won't this cause corruption of .jar, .class, .jpg, etc files? to me now that because of the eol=native svn props, RMs who cut releases on Windows are likely making tars with crlf line endings (since svn will convert files to crlf on checkout). Note that this applies to ant as well as maven-built distros. And RMs who use Unix will be creating zips with LF line endings... [Are there any who use Macs?] I will make the unix filter configurable as well, but want feedback on what the default should be, as well as what our policy should be in commons on line endings in tar.gz as well as zip distros. Ideally the default would depend on the svn:eol-type. i.e. if svn:eol-type=native, then convert the file for the appropriate native format, otherwise leave alone. Dunno whether this can be made to work or not, though. It's possible that a particular file type may have different endings for different files. For example on JMeter, the old JMX files were CRLF on Windows, but the new JMX files are LF on all OSes. This all assumes that tar.gz is for Unix and .zip for Windows users - not unreasonable, but should probably be documented somewhere (perhaps it is ?). Phil - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [all][POLL] what files to fixcrlf for windows distributions
Mario Ivankovits wrote: Hi! It turns out that it is required to fix the crlf (EOL) thing when building distributions for windows distributions. A windows distribution is a distribution packed as zip file. Required by who and what is broken ? And I don't agree that zips are windows distributions. I tend to download one distribution (eg zip or tar.gz, depending on what I click) and use that on linux windows, instead of downloading it twice. So I currently don't see a need to change line endings at all for distributions. Mvgr, Martin - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [all][POLL] what files to fixcrlf for windows distributions
snip
I am working on a patch to the maven dist plugin that makes the crlf
filter property configurable. Arnaud pointed out in another thread
that there should probably a similar filter applied to make sure lf
line endings are used in the tar.gz distros. My inclination is to
apply this to *all* files in the tar.gz distros by default. It occurs
Won't this cause corruption of .jar, .class, .jpg, etc files?
Doh! Good point. Would need to exclude at least these.
to me now that because of the eol=native svn props, RMs who cut
releases on Windows are likely making tars with crlf line endings
(since svn will convert files to crlf on checkout). Note that this
applies to ant as well as maven-built distros.
And RMs who use Unix will be creating zips with LF line endings...
Yes, IIUC, that is the consequence of using eol=native. Having the
file contents of distros different depending on the OS used to cut the
release does not seem like a good thing to me. Could be I have
something wrong here. Maybe someone who has both unix and Windows
boxes set up can test and see if the the Windows-built distros (from
Windows checkouts) end up with CRLF everywhere (at least for files
with eol=native).
[Are there any who use Macs?]
I will make the unix filter configurable as well, but want feedback on
what the default should be, as well as what our policy should be in
commons on line endings in tar.gz as well as zip distros.
Ideally the default would depend on the svn:eol-type.
i.e. if svn:eol-type=native, then convert the file for the appropriate
native format, otherwise leave alone.
Dunno whether this can be made to work or not, though.
But this would still result (again, assuming I have it right) in
different line endings in distros depending on what OS was used to cut
the release - unless we all set up Windows boxes to cut the zip
releases and *nix boxes for the tarballs ;-)
It's possible that a particular file type may have different endings
for different files.
For example on JMeter, the old JMX files were CRLF on Windows, but the
new JMX files are LF on all OSes.
Is this specified in the svn eol props for these files?
It now occurs to me that assuming my understanding of how eol=native
is correct, we should be fixing crlf for non-txt files in zip distros
as well, since if these are cut using Windows, e.g. the java source
files will have CRLF line endings. So, in terms of properties, it
seems we need
maven.dist.lf.filter=**/*.java, **/*.txt, all other non-binary file
specs, i.e. all files that could have eol=native
maven.dist.crlf.filter=**/*.txt (plus whatever else we agree on)
and have the plugin do the following:
For tar.gz distributions, run fixcrlf with eol=lf and
includes=${maven.dist.lf.filter}
For zip distributions, run fixcrlf first with eol=lf and
includes=${maven.dist.lf.filter} and then again with eol=crlf and
includes=${maven.dist.crlf.filter}.
I guess another alternative would be to agree to cut all distros on
unix, maybe even using minotaur to do the release builds (I know this
would have some practical problems associated with it now). Then all
we would need is the first property, assuming we want lf line endings
everywhere else (IIRC, in the cvs days, that was the standard).
Sorry if I have misunderstood the way the svn client works or if
others view this as a non-issue. I just want to patch the plugin so
that we can get consistent distros built using maven.
Phil
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: [all][POLL] what files to fixcrlf for windows distributions
On 11/15/05, Stephen Colebourne [EMAIL PROTECTED] wrote: Mario Ivankovits wrote: [X] .txt [ ] .java [ ] .properties [ ] .xml [ ] .css [ ] .mf My preference is for all ASCII files not in the src/java or src/text folders to be converted. Note that I doubt [collections] or [io] do this fully yet. Stephen I am working on a patch to the maven dist plugin that makes the crlf filter property configurable. Arnaud pointed out in another thread that there should probably a similar filter applied to make sure lf line endings are used in the tar.gz distros. My inclination is to apply this to *all* files in the tar.gz distros by default. It occurs to me now that because of the eol=native svn props, RMs who cut releases on Windows are likely making tars with crlf line endings (since svn will convert files to crlf on checkout). Note that this applies to ant as well as maven-built distros. I will make the unix filter configurable as well, but want feedback on what the default should be, as well as what our policy should be in commons on line endings in tar.gz as well as zip distros. Phil - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [all][POLL] what files to fixcrlf for windows distributions
As someone who uses Windows on a daily basis, I've never had a pproblem with the LF line endings and hence would only filter *.txt in a binary distribution. On 11/15/05, Mario Ivankovits [EMAIL PROTECTED] wrote: Hi! It turns out that it is required to fix the crlf (EOL) thing when building distributions for windows distributions. A windows distribution is a distribution packed as zip file. Maven will add support to define a filter to setup the extension of the files to convert. Anyway I think we should find a standard how to setup this filter. The extension in question are: [ ] .txt [ ] .java [ ] .properties [ ] .xml [ ] .css .java, .properties, .xml, .css might be problematic as we might get diffs with the wrong line ending then - on the other hand for an developer its not needed as every IDE can handle LF only line endings and our source code format should define this (the LF ending) as required. So only .txt might be save as this is generally the release notes files and getting patches for it is very unlikely ;-) What do you think? My personal vote goes to .txt only. Thanks! Mario - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- http://www.multitask.com.au/people/dion/ You are going to let the fear of poverty govern your life and your reward will be that you will eat, but you will not live. - George Bernard Shaw - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [all][POLL] what files to fixcrlf for windows distributions
Hi, I opened these issue to add a property to allow us to select the filter : http://jira.codehaus.org/browse/MPDIST-28 Isn't it useful to ensure that the LF line ending is used in Unix Archives ? Arnaud -Message d'origine- De : Dion Gillard [mailto:[EMAIL PROTECTED] Envoyé : mardi 15 novembre 2005 10:02 À : Jakarta Commons Developers List Objet : Re: [all][POLL] what files to fixcrlf for windows distributions As someone who uses Windows on a daily basis, I've never had a pproblem with the LF line endings and hence would only filter *.txt in a binary distribution. On 11/15/05, Mario Ivankovits [EMAIL PROTECTED] wrote: Hi! It turns out that it is required to fix the crlf (EOL) thing when building distributions for windows distributions. A windows distribution is a distribution packed as zip file. Maven will add support to define a filter to setup the extension of the files to convert. Anyway I think we should find a standard how to setup this filter. The extension in question are: [ ] .txt [ ] .java [ ] .properties [ ] .xml [ ] .css .java, .properties, .xml, .css might be problematic as we might get diffs with the wrong line ending then - on the other hand for an developer its not needed as every IDE can handle LF only line endings and our source code format should define this (the LF ending) as required. So only .txt might be save as this is generally the release notes files and getting patches for it is very unlikely ;-) What do you think? My personal vote goes to .txt only. Thanks! Mario - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- http://www.multitask.com.au/people/dion/ You are going to let the fear of poverty govern your life and your reward will be that you will eat, but you will not live. - George Bernard Shaw - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [all][POLL] what files to fixcrlf for windows distributions
Mario Ivankovits wrote: [X] .txt [ ] .java [X] .properties [X] .xml [X] .css [X] .mf My preference is for all ASCII files not in the src/java or src/text folders to be converted. Note that I doubt [collections] or [io] do this fully yet. Stephen - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
