Re: Compile-time detection of EOL translation mode (CLISP)

[Aaron Brown]

Reini Urban wrote:


For me [defined(WIN32)] is true during clisp compilation.
I had to change the patch to fix the WIN32 section, not
the UNIX one. I'll check which optional header defines
WIN32.


I did actually test it to make sure I wasn't misremembering
it being false.  I haven't found any official Cygwin
documentation on WIN32, but a few mailing list posts also
say it should be false:

 http://www.cygwin.com/ml/cygwin/2005-03/msg00614.html
 http://www.cygwin.com/ml/cygwin-apps/2000-11/msg00043.html


There's a setter, but not easy to find.



[5]> (setq *terminal-encoding* (ext:make-encoding :line-terminator :unix))
#


This apparently doesn't work when stdout isn't a terminal.
E.g., running the following

 clisp.exe -i .clisprc test.lisp > test.txt

(where .clisprc does the above setq and test.lisp just
prints several lines) results in

 *** - SYSTEM::SET-STREAM-EXTERNAL-FORMAT on
#
  #>
   is illegal

If I have .clisprc do

 (setf (stream-external-format *standard-output* )
   (make-encoding :line-terminator :unix))

and stdout is not a terminal, then there's no error, but
test.txt is still in dos format.  (Of course output
redirections like this are exactly why I care about EOLs.)

If you think it's a good idea, we can take this discussion
to the CLISP list.

Thanks,

--
Aaron
Beginning Lua Programming: http://www.amazon.com/gp/product/0470069171/

--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

Re: Compile-time detection of EOL translation mode (CLISP)

[Reini Urban]

2007/1/30, Aaron Brown <[EMAIL PROTECTED]>:

Reini Urban (quoting Sam Steingold at the issue tracker
page [1]) wrote:

> The original problem is best solved by a
>   (setq *default-file-encoding* :unix)
> in ~/.clisprc.lisp

As mentioned in an earlier post, that doesn't work for
*standard-output*, presumably because it's already :dos
before .clisprc runs.  It would work if there were a way to
change the external-format of an already-open stream.  Is
there?


There's a setter, but not easy to find.

(setf (stream-external-format *standard-output* )

 (make-encoding :line-terminator :unix))
*** - SYSTEM::SET-STREAM-EXTERNAL-FORMAT on # is illegal

You cannot set it on the command-line via "clisp -Efile 1:1 -Eterminal 1:1"
only via the setter. Thanksfully derived streams get it from the parent.

$ clisp -Efile 1:1 -Eterminal 1:1 -q
[1]> *DEFAULT-FILE-ENCODING*
#
[2]> *TERMINAL-ENCODING*
#
[3]> (stream-external-format *standard-output*)
#
[4]> (setf (stream-external-format *standard-output*) (ext:make-encoding :line-
terminator :unix))
*** - SYSTEM::SET-STREAM-EXTERNAL-FORMAT on # is illegal
[5]> (setq *terminal-encoding* (ext:make-encoding :line-terminator :unix))
#
[6]> (stream-external-format *standard-output*)
#
[7]> (system::set-terminal-encoding (ext:make-encoding :line-terminator :unix))
#

The -E arg is only for the charset, not for the line ending. See
encoding_from_name.


Sam also wrote at the issue tracker page [1]:

> note that even the cygwin CLISP is expected to write files
> useful for other (non-cygwin) programs, so the fact that
> they really expect CRLF does matter to us.

Au contraire; speaking for myself, the reason I use Cygwin
is to avoid (as much as possible) dealing with Windows's
non-unixy things such as CRLFs.  If I wanted CLISP to act
like a Windows program rather than a unix program, I would
use a Windows (non-Cygwin) version of it.


I agree with you. For the reader it is better to detect optional CRLF,
but the stdout *standard-io* and stderr writer *debug-io* should
produce LF only.
Same for the *default-file-encoding*. Esp. annoying since you cannot
set it on the command-line.


By the way, the cited section of the CLISP implementation
notes [2] says that defined(WIN32) is true in Cygwin, but it
isn't.

[1] http://tinyurl.com/3b3yux
[2] http://clisp.podval.org/impnotes/encoding.html#line-term-default


For me it is true during clisp compilation.
I had to change the patch to fix the WIN32 section,
not the UNIX one. I'll check which optional header defines WIN32.
--
Reini Urban
http://phpwiki.org/  http://murbreak.at/
http://spacemovie.mur.at/   http://helsinki.at/

--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

Re: Compile-time detection of EOL translation mode (CLISP)

[Aaron Brown]

Reini Urban (quoting Sam Steingold at the issue tracker
page [1]) wrote:


The original problem is best solved by a
  (setq *default-file-encoding* :unix)
in ~/.clisprc.lisp


As mentioned in an earlier post, that doesn't work for
*standard-output*, presumably because it's already :dos
before .clisprc runs.  It would work if there were a way to
change the external-format of an already-open stream.  Is
there?

Sam also wrote at the issue tracker page [1]:


note that even the cygwin CLISP is expected to write files
useful for other (non-cygwin) programs, so the fact that
they really expect CRLF does matter to us.


Au contraire; speaking for myself, the reason I use Cygwin
is to avoid (as much as possible) dealing with Windows's
non-unixy things such as CRLFs.  If I wanted CLISP to act
like a Windows program rather than a unix program, I would
use a Windows (non-Cygwin) version of it.

By the way, the cited section of the CLISP implementation
notes [2] says that defined(WIN32) is true in Cygwin, but it
isn't.

[1] http://tinyurl.com/3b3yux
[2] http://clisp.podval.org/impnotes/encoding.html#line-term-default

Thanks,

--
Aaron

--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

Re: Compile-time detection of EOL translation mode (CLISP)

[Reini Urban]

Reini Urban schrieb:

Aaron Brown schrieb:

Reini Urban wrote:


Confirmed.
Bug patched at 
http://sourceforge.net/tracker/index.php?func=detail&aid=1633552&group_id=1355&atid=101355 



It's been moved to  with the
following comment from Sam Steingold:


This patch does not seem right.
on linux O_BINARY==0 and I see no reason to default line
termination to :DOS there.
I am rejecting it pending your convincing me that I am
wrong here.


To be honest, it didn't seem quite right to me either, but I
figured that was down to me not knowing the ins and outs of
how Cygwin handles text modes.

The original logic was:


#if defined(WIN32) || (defined(UNIX) && (O_BINARY != 0))
 pushSTACK(S(Kdos));   /* :line-terminator */
#else
 pushSTACK(S(Kunix));  /* :line-terminator */


which translates to "if O_BINARY is something other than 0,
then this unix must differentiate between text and binary
streams, so assume that text streams use dos EOLs".  That
makes sense, except that O_BINARY is nonzero even on (for
instance) my Cygwin installation, where I selected unix EOLs
as the default with setup.exe.

The patch changes the condition to:


#if defined(WIN32) || (defined(UNIX) && (O_BINARY == 0))


which, as Sam pointed out, would make most unixes (on which
O_BINARY is 0) use dos EOLs.


I tought the UNIX O_BINARY logic was only for CYGWIN and wondered what 
other UNIX that could be.

Ok, so we'll have to use
#if defined(WIN32) || \
  (defined(UNIX) && (O_BINARY != 0) && !defined(__CYGWIN__))

or make the call slower by doing a dynamic textmount check.
But I really don't want to do that.


This issue was solved here
https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1633552&group_id=1355
by adding a documentation chapter to clisp's next version:
  http://clisp.podval.org/impnotes/encoding.html#line-term-default

See also:
  http://clisp.cons.org/impnotes/clhs-newline.html
  http://www.unicode.org/reports/tr13/tr13-9.html

The original problem is best solved by a
  (setq *default-file-encoding* :unix)
in ~/.clisprc.lisp
--
Reini Urban
http://phpwiki.org/  http://murbreak.at/
http://helsinki.at/  http://spacemovie.mur.at/

--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

Re: Compile-time detection of EOL translation mode (CLISP)

[Reini Urban]

Aaron Brown schrieb:

Reini Urban wrote:


Confirmed.
Bug patched at 
http://sourceforge.net/tracker/index.php?func=detail&aid=1633552&group_id=1355&atid=101355 



It's been moved to  with the
following comment from Sam Steingold:


This patch does not seem right.
on linux O_BINARY==0 and I see no reason to default line
termination to :DOS there.
I am rejecting it pending your convincing me that I am
wrong here.


To be honest, it didn't seem quite right to me either, but I
figured that was down to me not knowing the ins and outs of
how Cygwin handles text modes.

The original logic was:


#if defined(WIN32) || (defined(UNIX) && (O_BINARY != 0))
 pushSTACK(S(Kdos));   /* :line-terminator */
#else
 pushSTACK(S(Kunix));  /* :line-terminator */


which translates to "if O_BINARY is something other than 0,
then this unix must differentiate between text and binary
streams, so assume that text streams use dos EOLs".  That
makes sense, except that O_BINARY is nonzero even on (for
instance) my Cygwin installation, where I selected unix EOLs
as the default with setup.exe.

The patch changes the condition to:


#if defined(WIN32) || (defined(UNIX) && (O_BINARY == 0))


which, as Sam pointed out, would make most unixes (on which
O_BINARY is 0) use dos EOLs.


I tought the UNIX O_BINARY logic was only for CYGWIN and wondered what 
other UNIX that could be.

Ok, so we'll have to use
#if defined(WIN32) || \
  (defined(UNIX) && (O_BINARY != 0) && !defined(__CYGWIN__))

or make the call slower by doing a dynamic textmount check.
But I really don't want to do that.
--
Reini Urban
http://phpwiki.org/  http://murbreak.at/
http://helsinki.at/  http://spacemovie.mur.at/

--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

Re: Compile-time detection of EOL translation mode (CLISP)

[Aaron Brown]

Reini Urban wrote:


Confirmed.
Bug patched at 
http://sourceforge.net/tracker/index.php?func=detail&aid=1633552&group_id=1355&atid=101355


It's been moved to  with the
following comment from Sam Steingold:


This patch does not seem right.
on linux O_BINARY==0 and I see no reason to default line
termination to :DOS there.
I am rejecting it pending your convincing me that I am
wrong here.


To be honest, it didn't seem quite right to me either, but I
figured that was down to me not knowing the ins and outs of
how Cygwin handles text modes.

The original logic was:


#if defined(WIN32) || (defined(UNIX) && (O_BINARY != 0))
 pushSTACK(S(Kdos));   /* :line-terminator */
#else
 pushSTACK(S(Kunix));  /* :line-terminator */


which translates to "if O_BINARY is something other than 0,
then this unix must differentiate between text and binary
streams, so assume that text streams use dos EOLs".  That
makes sense, except that O_BINARY is nonzero even on (for
instance) my Cygwin installation, where I selected unix EOLs
as the default with setup.exe.

The patch changes the condition to:


#if defined(WIN32) || (defined(UNIX) && (O_BINARY == 0))


which, as Sam pointed out, would make most unixes (on which
O_BINARY is 0) use dos EOLs.

Thanks,

--
Aaron
Beginning Lua Programming: http://www.amazon.com/gp/product/0470069171/

--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

Re: Compile-time detection of EOL translation mode (CLISP)

[Aaron Brown]

Reini Urban wrote:


Confirmed.
Bug patched at 
http://sourceforge.net/tracker/index.php?func=detail&aid=1633552&group_id=1355&atid=101355


Thanks!

--
Aaron
Beginning Lua Programming: http://www.amazon.com/gp/product/0470069171/

--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

Re: Compile-time detection of EOL translation mode (CLISP)

[Reini Urban]

2007/1/9, Reini Urban <[EMAIL PROTECTED]>:

defined(UNIX) && (O_BINARY != 0) seems to only target cygwin and looks
like an upstream bug to me. If binary then do binary and not DOS.

Thanks for the report!
I'll check and update it then.


Confirmed.
Bug patched at 
http://sourceforge.net/tracker/index.php?func=detail&aid=1633552&group_id=1355&atid=101355

Will be in the next update as well as the pending 8MB stack increase.
I also cannot compile KM (The Knowledge Machine) anymore because of
the too small stacksize.

--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

Re: Compile-time detection of EOL translation mode (CLISP)

[Reini Urban]

2007/1/4, Dave Korn <[EMAIL PROTECTED]>:

On 04 January 2007 18:58, Aaron Brown wrote:

> Dave Korn wrote:

>>  What is the /end/ result you're trying to achieve here?
>
> When CLISP does text output, it opens the file in binary
> mode and does EOL translation itself, according to the
> requested line terminator mode.  For instance, the following
> function writes a file with LF, CRLF, or CR EOLs depending
> on whether its argument is :unix, :dos, or :mac.

> That works exactly as intended.

  Good, and indeed, CLISP should probably not care either way about the mount
mode on which it writes these files, because that's the whole point of
textmounts: to provide a way for the user to instruct cygwin (the kernel,
effectively) to override the application's choice at a lower level when
needed.

> My problem is that if I
> don't specify a :line-terminator, :dos is used instead of
>> unix.  (This is controlled by *default-file-encoding*,
> which itself is set by line 2571 of encoding.d when CLISP is
> compiled.)

  Ok, that is simply the wrong default, and I think it is /that/ which needs
to be fixed.

> The reason I expect :unix to be the default
> rather than :dos is that when I ran the Cygwin setup.exe, I
> selected the "Unix / binary (RECOMMENDED)" radio button
> under "Default Text File Type".  Is this expectation
> unwarranted?

  Yes, it is really; unless CLISP used some compile-time autoconf test to
figure this out and adjust that default you mentioned, how else could it make
a difference?

> The one-sentence version of the problem: CLISP writes
> newlines as "\r\n" by default, when I want the default to be
> "\n".
>
> I could just specify :unix every time, but that's a hassle
> and is (as far as I can tell) impossible with standard
> output.

  This smacks of a build/packaging error to me, but perhaps it was a
deliberate decision; we need to hear from the CLISP maintainer.  Without
having looked into building CLISP, I don't know whether it would just need an
option when configuring or a #define patched into the source somewhere.


The maintainer is now back from holidays.
We really have to ask upstream Sam why he wanted to use global
DOS EOL for cygwin. Probably to support default textmounts.

But then
defined(UNIX) && (O_BINARY != 0) seems to only target cygwin and looks
like an upstream bug to me. If binary then do binary and not DOS.

Thanks for the report!
I'll check and update it then.
--
Reini Urban
http://phpwiki.org/  http://murbreak.at/
http://spacemovie.mur.at/   http://helsinki.at/

--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

RE: Compile-time detection of EOL translation mode (CLISP)

[Dave Korn]

On 04 January 2007 18:58, Aaron Brown wrote:

> Dave Korn wrote:

>>  What is the /end/ result you're trying to achieve here?
> 
> When CLISP does text output, it opens the file in binary
> mode and does EOL translation itself, according to the
> requested line terminator mode.  For instance, the following
> function writes a file with LF, CRLF, or CR EOLs depending
> on whether its argument is :unix, :dos, or :mac.

> That works exactly as intended.  

  Good, and indeed, CLISP should probably not care either way about the mount
mode on which it writes these files, because that's the whole point of
textmounts: to provide a way for the user to instruct cygwin (the kernel,
effectively) to override the application's choice at a lower level when
needed.

> My problem is that if I
> don't specify a :line-terminator, :dos is used instead of
>> unix.  (This is controlled by *default-file-encoding*,
> which itself is set by line 2571 of encoding.d when CLISP is
> compiled.)  

  Ok, that is simply the wrong default, and I think it is /that/ which needs
to be fixed.

> The reason I expect :unix to be the default
> rather than :dos is that when I ran the Cygwin setup.exe, I
> selected the "Unix / binary (RECOMMENDED)" radio button
> under "Default Text File Type".  Is this expectation
> unwarranted?

  Yes, it is really; unless CLISP used some compile-time autoconf test to
figure this out and adjust that default you mentioned, how else could it make
a difference?

> The one-sentence version of the problem: CLISP writes
> newlines as "\r\n" by default, when I want the default to be
> "\n".
> 
> I could just specify :unix every time, but that's a hassle
> and is (as far as I can tell) impossible with standard
> output.

  This smacks of a build/packaging error to me, but perhaps it was a
deliberate decision; we need to hear from the CLISP maintainer.  Without
having looked into building CLISP, I don't know whether it would just need an
option when configuring or a #define patched into the source somewhere.

cheers,
  DaveK
-- 
Can't think of a witty .sigline today


--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

Re: Compile-time detection of EOL translation mode (CLISP)

[Aaron Brown]

Dave Korn wrote:


From within the C preprocessor, how do I find out whether
a Cygwin installation is set to use dos or unix EOLs?


 You don't.  Whatever you're trying to do, this is the
 wrong way to go about it.



 What is the /end/ result you're trying to achieve here?


When CLISP does text output, it opens the file in binary
mode and does EOL translation itself, according to the
requested line terminator mode.  For instance, the following
function writes a file with LF, CRLF, or CR EOLs depending
on whether its argument is :unix, :dos, or :mac.

 ;; See http://clisp.cons.org/impnotes/encoding.html#newline
 (defun newlinetest (line-terminator)
   (with-open-file
 (out "eoltest.txt" :external-format
   (ext:make-encoding :line-terminator line-terminator)
   :direction :output
   :if-does-not-exist :create
   :if-exists :new-version)
 (format out "foo~%bar~%")))

That works exactly as intended.  My problem is that if I
don't specify a :line-terminator, :dos is used instead of
:unix.  (This is controlled by *default-file-encoding*,
which itself is set by line 2571 of encoding.d when CLISP is
compiled.)  The reason I expect :unix to be the default
rather than :dos is that when I ran the Cygwin setup.exe, I
selected the "Unix / binary (RECOMMENDED)" radio button
under "Default Text File Type".  Is this expectation
unwarranted?

The one-sentence version of the problem: CLISP writes
newlines as "\r\n" by default, when I want the default to be
"\n".

I could just specify :unix every time, but that's a hassle
and is (as far as I can tell) impossible with standard
output.

Thanks,

--
Aaron
http://www.amazon.com/gp/product/0470069171/


--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

RE: Compile-time detection of EOL translation mode (CLISP)

[Dave Korn]

On 04 January 2007 17:17, Aaron Brown wrote:

> From within the C preprocessor, how do I find out whether a
> Cygwin installation is set to use dos or unix EOLs?

  You don't.  Whatever you're trying to do, this is the wrong way to go about
it.

  For a start, the answer to the question "Is this in dos or unix mode" can
change from one directory to another.

  What is the /end/ result you're trying to achieve here?


cheers,
  DaveK
-- 
Can't think of a witty .sigline today


--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

Compile-time detection of EOL translation mode (CLISP)

[Aaron Brown]

From within the C preprocessor, how do I find out whether a

Cygwin installation is set to use dos or unix EOLs?

The reason I ask is this:  I'm using version 2.41 as
downloaded from a Cygwin mirror, and my
*default-file-encoding* uses dos-style line terminators

 [1]> *default-file-encoding*
 #

even though my Cygwin installation is set to use unix ones.
With help from comp.lang.lisp, I found that this is set in
src/encoding.d:

  #if defined(WIN32) || (defined(UNIX) && (O_BINARY != 0))
   pushSTACK(S(Kdos));   /* :line-terminator */
  #else
   pushSTACK(S(Kunix));  /* :line-terminator */
  #endif

Specifically, O_BINARY is 0x1 (even though it should
have no effect on my Cygwin installation).

I kluged around this by using "#if 0", but it really should
look at the Cygwin installation's EOL setting.  How can this
be done?  (Perhaps check the CYGWIN environment variable in
the makefile?)

(The reason CLISP doesn't just open files in text mode is
that it does its own EOL translation, controllable at
runtime.)

Thanks,

--
Aaron
http://www.amazon.com/gp/product/0470069171/


--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/