Re: DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[Antoine Levy-Lambert]

Peter, you are not the only one who sees failures on Windows for these
tests. Additionnally, on my machine a couple of other tests fail (Image,
Antlr, ...)
That is not so good. We need to work on these failing tests (to see if the
testcases or the classes being tested are wrong, and fix whatever needs to
be fixed).
On top of the 400+ open bugreps.
Antoine
> PS. I have had difficulty in getting the following unit tests to work on
> windows:
>CopyTest - trancoding test - result not as expected
>ZipTest  - last test - unable to delete a file
>XMLCatalogTest - most of the tests fail
>FileUnitTests - to/from uri and testRemoveLeadingPath
> Peter
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-23 15:31 ---
Patch applied.  Thanks.

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-22 23:40 ---
I hadn't considered Mac.  I will apply your patch tomorrow.  Thanks, once again!

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-22 23:35 ---
snap!
I was just about to submit the changes under windows.

I have added a new patch against current cvs to fix
the unit tests and to provide an alternative fix for stripjavacomments.

The patched files are:
src/etc/testcases/filters/head-tail.xml:
  add the targets for the filterreader tests
src/testcases/org/apache/tools/ant/filters/HeadTailTest.java
  fix the result and expected filenames, I had them mixed up
  before, and the test succeeded as two non-existant files were matched
src/main/org/apache/tools/ant/filters/StripJavaComments.java
  I initially had the same logic as your fix, but realized that
  the code would not work under mac line endings. Looking at the
  code, I released that \r, or \n where all that is needed to terminate
  the the \\ comment - tested this on windows.

Cheers,
  Peter.

PS. I have had difficulty in getting the following unit tests to work on
windows:
   CopyTest - trancoding test - result not as expected
   ZipTest  - last test - unable to delete a file
   XMLCatalogTest - most of the tests fail
   FileUnitTests - to/from uri and testRemoveLeadingPath
Peter

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-22 23:30 ---
Created an attachment (id=5966)
patches for headtail unit tests and stripjavacomments

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings

[EMAIL PROTECTED] changed:

   What|Removed |Added

 Status|NEW |RESOLVED
 Resolution||FIXED



--- Additional Comments From [EMAIL PROTECTED]  2003-04-22 18:30 ---
ok - first of all, thanks for the patch.  I have applied it.

I had to modify StripJavaComments.java to retain proper line endings.

I had to disable a couple of tests for now: testFilterReaderHeadLinesSkip and
testFilterReaderTailLinesSkip - reason is these targets have not been defined in
head-tail.xml.  I also do not have the two 'expected' files:
head-tail.filterReaderHeadLinesSkip.test
head-tail.filterReaderTailLinesSkip.test

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-22 00:07 ---
I have not seen that problem, but remember that I made the
patch on linux.
I have now tested on windows but have run out of time today to
make a patch on windows (needed to download cvsnt instead of cygwin
cvs as the latter treated the files as unix, and it takes a long
time to cvs ant on a modem).

On doing this I found another bug in the tests.
in testHeadTail, the expected outout is now incorrect (
it should be "Line  3\r\nLine  4\r\n").

All the tests thus passed, except the last test.

I could not understand why the unix tests passed so I rebooted
into linux and found the problem - the file names used in
the test where of a different case to the files on the filesystem
(head-tail.headtail.test as against head-tail.headTail.test). So
on unix the test was comparing two non-existent files!

If you could apply the patches to all the files except the head-tail.test
file, and add a new line using an editor to the head-tail.test, the
line ending problem (\n instead of \r\n) may be resolved.

Also I see that I included the wrong patched file (CommandlineJavaTest.java)
in patch 5850. The file should have been
src/etc/testcases/taskdefs/copy.filterset, and the change was to add a
new line to the end of the file. Sorry..., I will make a new patch tomorrow
from windows.

Peter

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-21 18:01 ---
After applying the patch:
HeadTailTest fails on Windows.  The result file contains Unix style line-
endings (\n) instead of \r\n and therefore the expected files and the resultant
files don't match.  Haven't dug intto the code yet...

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-18 21:57 ---
Created an attachment (id=5898)
tailfilter: remove unused member variable, comment "linelist"

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-18 21:51 ---
Minor errata on 5897:
I left in a unused member variable in tailfilter - bufferPos.
Peter

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-18 21:47 ---
I have made a patch to modify head and tail filter to use linetokenizer
from tokenfilter. LineTokenizer tokenizes files using  \r, \n or \r\n, so
it should work with dos, unix and mac for head and tail filter.
Make a change to linetokenizer to optionally include the lineending with
the line. Made a similar change to stringtokenizer.
Also when head and tail filters where used a filterreader
with paramaters, only the first parameter was used.

The head and tail tests now pass on linux, but I have not tested on
dos (or mac).

This patch is in addition to 5850 and 5851.

The changes are:
docs/manual/CoreTypes/filterchain.html:
   document new attribute to linetokenizer and to stringtokenizer
src/etc/testcases/filters/input/head-tail.test
   add a new line to end of file
src/main/org/apache/tools/ant/filters/HeadFilter.java
   use linetokenizer
   use all parameters
src/main/org/apache/tools/ant/filters/TailFilter.java
   use linetokenizer
   use all parameters
src/main/org/apache/tools/ant/filters/TokenFilter.java
   add includeDelims attribute to linetokenizer and to stringtokenizer
src/testcases/org/apache/tools/ant/filters/HeadTailTest.java
   add tests to test use of filterreader with head and tail filter

Peter

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-18 21:33 ---
Created an attachment (id=5897)
patches for head/tail fixes

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-16 00:32 ---
I meant to say for 2) above that
this is most likely an off-by-one problem
where the original testing was done on
dos (where new line is \r\n and the \r
needs to be removed with \n).
Peter

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-16 00:26 ---
Ok, got the unit test and patches to current tests
id 5850:
This contains patches against the following files:
src/etc/testcases/filters/build.xml
   add test target for no end of line adding for copy/filterchain
src/etc/testcases/filters/input/stripjavacomments.test
   add nl to end of this file so that it matches expected output
   (magic of cvs will pick the correct nl form for the platform)
src/testcases/org/apache/tools/ant/types/CommandlineJavaTest.java
   add nl to end of this file so that it matches expected output
src/main/org/apache/tools/ant/util/FileUtils.java
   reissue of patch

id 5851:
The unit test for checking if the copy/filterchain does
not add a new line (it uses testNoAddNewLine target in build.xml)

Without the change to fileutils all the enabled tests pass except nonewlinetest
With the change all the enabled tests pass

I have not been able to fix the head/tail unit tests.
However I have a number of observations (testing under linux, after patch)

1) the number of lines for tail is incorrect if the file ends
   in a new-line

input: "line 1\nline 2\nline 3\n"
tail l=2 -> just gives "line3\n"
input: "line 1\nline 2\nline 3"
tail l=2 -> gives "line 2\nline 3"

2) the skip attribute does not work for linux for tail
input: "line 1\nline 2\nline 3\n"
tail l=-1 s=1 -> gives "line 1\nline 2\nline "
input: "line 1\nline 2\nline 3"
tail l=-1 s=1 -> gives "line 1\nline "

3) the filter does not deal with mac files (the code only looks for \n).


Peter

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-15 23:41 ---
Created an attachment (id=5851)
src/testcases/org/apache/tools/ant/filters/NoNewLineTest.java

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-15 23:38 ---
Created an attachment (id=5850)
mods to unit tests, and patch to fileutils

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-15 00:30 ---
It appears like a bug to me even though the existing testcases don't consider
it so.  We can change the flawed testcases as well, can't we? :-)

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-15 00:03 ---
>>Please provide testcases - the tests should fail before the patch is applied
>>and must pass afterwards.

Magesh,

Ok, - but some of the current tests rely on the current behaviour.
(I think as well as testcases/org.apache.tools.ant.taskdefs#testFilterChain)

Peter

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-04-14 18:24 ---
Peter,
Please provide testcases - the tests should fail before the patch is applied
and must pass afterwards.

DO NOT REPLY [Bug 18476] - copy with filtering modifies lineendings

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=18476

copy with filtering modifies lineendings





--- Additional Comments From [EMAIL PROTECTED]  2003-03-28 18:17 ---
Created an attachment (id=5556)
Modify FileUtils#copyFile to retain line endings when used with filterchains