Re: lib/test/simple/t/create.t help with VMS issue needed.
Michael G Schwern wrote:
On Sat, Aug 13, 2005 at 10:33:45PM -0400, John E. Malmberg wrote:
Test 7 is failing because normally on VMS, unless you specify otherwise,
you get exclusive access to the file, so the second open is failing.
The logical name DECC$FILE_SHARING defined as "ENABLE" will change VMS
behavior to that of UNIX which will allow test 7 to pass.
I can probably come up with some code to have the script on VMS make
sure that that value is set and to clear it on exit.
Test 8 is more of a problem. The issue is that the buffers for the
other stream written by the new_tb->output("some_file") have not made it
to disk, so they can not yet be read by the new input stream to pass the
test.
There does not seem to be a method of explicitly closing or flushing the
output stream being written to by Test::Builder.
I was about to commit the test fix for this and then I realized that
Test::Builder unbuffers its newly created output filehandles. Everything
should be written to disk immediately. If not then there's a bug either
in Test::Builder's autoflush logic or in Perl.
Could you have another look at this? Test::Builder _new_fh() and _autoflush()
will be of some interest.
_autoflush is setting $| which should cause a flush.
The only call to fsync() that I can find that gets compiled in on the
VMS build is in vms/vms.c routine Perl_my_flush() which has a macro
alias of my_flush(), and a macro alias Fflush().
Setting a breakpoint on vms/Perl_my_flush() on the debugger shows that
it is not called, and this is the only way that the I/O will be written
to disk.
Putting a break on pp_hot/Perl_pp_print() shows that it recognizes that
the $| variable was set, and it calls PerlIO_flush.
PerlIO_flush looks up a function in a table and ends up calling
PerlIOBuf_flush().
PerlIOBuf_flush() writes the internal buffer to the C library internal
buffer.
It then calls PerlIO_flush again and this time the function lookup is to
a routine that does nothing by design.
Nothing ever called an fsync() call, so nothing is guaranteed to get to
the disk.
It seems to work ok flushing I/O to the terminal.
A scan of the source code shows that my_flush or Perl_my_flush is only
referenced in VMS.C
Fflush is only referenced in ext/IO/io.c, and an alias of PerlSIO_fflush.
It appears that that PerlSIO_fflush() is called by PerlIOStdio_flush() only.
So it appears the problem is that nothing is calling fsync(), but I am
not sure what needs to be changed to fix this. I also do not know if
this affects anything other than VMS.
-John
[EMAIL PROTECTED]
Personal Opinion Only
Re: lib/test/simple/t/create.t help with VMS issue needed.
Michael G Schwern wrote:
On Sat, Aug 13, 2005 at 10:33:45PM -0400, John E. Malmberg wrote:
Test 7 is failing because normally on VMS, unless you specify otherwise,
you get exclusive access to the file, so the second open is failing.
The logical name DECC$FILE_SHARING defined as "ENABLE" will change VMS
behavior to that of UNIX which will allow test 7 to pass.
I can probably come up with some code to have the script on VMS make
sure that that value is set and to clear it on exit.
Test 8 is more of a problem. The issue is that the buffers for the
other stream written by the new_tb->output("some_file") have not made it
to disk, so they can not yet be read by the new input stream to pass the
test.
There does not seem to be a method of explicitly closing or flushing the
output stream being written to by Test::Builder.
I was about to commit the test fix for this and then I realized that
Test::Builder unbuffers its newly created output filehandles. Everything
should be written to disk immediately. If not then there's a bug either
in Test::Builder's autoflush logic or in Perl.
Could you have another look at this? Test::Builder _new_fh() and _autoflush()
will be of some interest.
Sure, I can take a look. I also discovered while doing research on how
to add the methods to set/get the current VMS C Library behavior that
VMS was not configuring "d_fsync", and with out a fsync() call, C I/O is
not flushed down to the disk.
fsync() is being called in VMS.C in one place, but I do not have time to
look for a while.
-John
[EMAIL PROTECTED]
Personal Opinion Only
Re: lib/test/simple/t/create.t help with VMS issue needed.
On Sat, Aug 13, 2005 at 10:33:45PM -0400, John E. Malmberg wrote:
> Test 7 is failing because normally on VMS, unless you specify otherwise,
> you get exclusive access to the file, so the second open is failing.
>
> The logical name DECC$FILE_SHARING defined as "ENABLE" will change VMS
> behavior to that of UNIX which will allow test 7 to pass.
>
> I can probably come up with some code to have the script on VMS make
> sure that that value is set and to clear it on exit.
>
> Test 8 is more of a problem. The issue is that the buffers for the
> other stream written by the new_tb->output("some_file") have not made it
> to disk, so they can not yet be read by the new input stream to pass the
> test.
>
> There does not seem to be a method of explicitly closing or flushing the
> output stream being written to by Test::Builder.
I was about to commit the test fix for this and then I realized that
Test::Builder unbuffers its newly created output filehandles. Everything
should be written to disk immediately. If not then there's a bug either
in Test::Builder's autoflush logic or in Perl.
Could you have another look at this? Test::Builder _new_fh() and _autoflush()
will be of some interest.
--
Michael G Schwern [EMAIL PROTECTED] http://www.pobox.com/~schwern
Stabbing you in the face so you don't have to.
Re: lib/test/simple/t/create.t help with VMS issue needed.
On Fri, Sep 16, 2005 at 08:47:42PM -0400, John E. Malmberg wrote: > Anything happening with this for the 5.8.8 timeframe? I guess I can kick 0.61 out the door. -- Michael G Schwern [EMAIL PROTECTED] http://www.pobox.com/~schwern Ahh email, my old friend. Do you know that revenge is a dish that is best served cold? And it is very cold on the Internet!
Re: lib/test/simple/t/create.t help with VMS issue needed.
John E. Malmberg wrote: Michael G Schwern wrote: On Sat, Aug 13, 2005 at 10:33:45PM -0400, John E. Malmberg wrote: I can change the test to write to a tied filehandle or I can make sure the new Test::Builder object which outputs to some_file is destroyed before I read from some_file. The later has the nice side effect of testing that destroying a Test::Builder object closes any open filehandles. Try the attached patch and let me know. The patch seems to work fine on my system, and I do not even need to use the feature logical for sharing the file. EAGLE> mcr []Perl. "-I[-.lib]" [-.lib.test.simple.t]create.t 1..8 ok 1 - The object isa Test::Builder ok 2 - create does not interfere with ->builder ok 3 -does not interfere with ->new ok 4 - The object isa Test::Builder ok 5 - Test::Builder->create makes a new object ok 6 - Changing output() of new TB doesn't interfere with singleton ok 7 ok 8 Anything happening with this for the 5.8.8 timeframe? -John [EMAIL PROTECTED] Personal Opinion Only
Re: lib/test/simple/t/create.t help with VMS issue needed.
Michael G Schwern wrote: On Sat, Aug 13, 2005 at 10:33:45PM -0400, John E. Malmberg wrote: There does not seem to be a method of explicitly closing or flushing the output stream being written to by Test::Builder. Any ideas on how to resolve this? I can change the test to write to a tied filehandle or I can make sure the new Test::Builder object which outputs to some_file is destroyed before I read from some_file. The later has the nice side effect of testing that destroying a Test::Builder object closes any open filehandles. Try the attached patch and let me know. The patch seems to work fine on my system, and I do not even need to use the feature logical for sharing the file. EAGLE> mcr []Perl. "-I[-.lib]" [-.lib.test.simple.t]create.t 1..8 ok 1 - The object isa Test::Builder ok 2 - create does not interfere with ->builder ok 3 -does not interfere with ->new ok 4 - The object isa Test::Builder ok 5 - Test::Builder->create makes a new object ok 6 - Changing output() of new TB doesn't interfere with singleton ok 7 ok 8 -John [EMAIL PROTECTED] Personal Opinion Only
Re: lib/test/simple/t/create.t help with VMS issue needed.
On Sat, Aug 13, 2005 at 10:33:45PM -0400, John E. Malmberg wrote:
> There does not seem to be a method of explicitly closing or flushing the
> output stream being written to by Test::Builder.
>
> Any ideas on how to resolve this?
I can change the test to write to a tied filehandle or I can make sure the
new Test::Builder object which outputs to some_file is destroyed before I
read from some_file. The later has the nice side effect of testing that
destroying a Test::Builder object closes any open filehandles.
Try the attached patch and let me know.
--
Michael G Schwern [EMAIL PROTECTED] http://www.pobox.com/~schwern
Don't try the paranormal until you know what's normal.
-- "Lords and Ladies" by Terry Prachett
=== t/create.t
==
--- t/create.t (revision 2525)
+++ t/create.t (local)
@@ -16,22 +16,24 @@
use Test::Builder;
my $more_tb = Test::More->builder;
-my $new_tb = Test::Builder->create;
-
-isa_ok $new_tb, 'Test::Builder';
isa_ok $more_tb, 'Test::Builder';
-isnt $more_tb, $new_tb, 'Test::Builder->create makes a new object';
-
is $more_tb, Test::More->builder, 'create does not interfere with ->builder';
is $more_tb, Test::Builder->new, ' does not interfere with ->new';
-$new_tb->output("some_file");
-END { 1 while unlink "some_file" }
+{
+my $new_tb = Test::Builder->create;
-$new_tb->plan(tests => 1);
-$new_tb->ok(1);
+isa_ok $new_tb, 'Test::Builder';
+isnt $more_tb, $new_tb, 'Test::Builder->create makes a new object';
+$new_tb->output("some_file");
+END { 1 while unlink "some_file" }
+
+$new_tb->plan(tests => 1);
+$new_tb->ok(1);
+}
+
pass("Changing output() of new TB doesn't interfere with singleton");
ok open FILE, "some_file";
Re: lib/test/simple/t/create.t help with VMS issue needed.
On 16/08/05, John E. Malmberg <[EMAIL PROTECTED]> wrote: > Sebb wrote: > > On 15/08/05, Carl Friedberg <[EMAIL PROTECTED]> wrote: > > > >>sebb wrote: > >> > >>On VMS, the default behavior makes sense. VMS is a > >>multi-user system, and when you open a file, the > >>default access is exclusive. VMS has rich file > >>sharing semantics, so it is easy to change this > >>behavior; but a VMS user expects default access > >>to be exclusive. > > > > I use VMS and I don't ... > > > > For example, the VMS TYPE command allows one to look at the contents > > of batch LOG files, but Perl cannot _read_ the same file without the > > work-round above (or vmsopen() with suitable parameters). Actually it seems to depend on the log file - see below. > > You are only seeing that behavior because of that the log files for > batch jobs explicitly were opened in a way that allows other programs to > read them. Note that a large amount of data in the log file may not > have yet been written to disk, and is unavailable to the type command. Agreed. > If you write an application that opens a file with the default options > and then sleeps, you will find that TYPE and other non-privileged > applications will not be able to access the file at all. > > >>Changing this behavior to conform to perl usage on > >>other OSes could be a configuration option; but > >>the default "no surprise" behavior should be > >>exclusive access. > > > IMO, the surprise in this case is that Perl can't open a file > > that TYPEs OK ... > > Perl can access any file that TYPEs ok. That is not the issue here. With Perl 5.5A3 I get: $ perl -n -e "print" sys$manager:operator.log Can't open sys$manager:operator.log: file currently locked by another user With 5.6.1, I get: $ perl -n -e "print" sys$manager:operator.log Can't open sys$manager:operator.log: error in directory name. Both work OK if I define DECC$FILE_SHARING > Type has no more privileges or features than Perl does in this case. Indeed, but presumably it opens the file slightly differently. > If pause the lib/test/simple/t/create/t test in the debugger just before > Perl tries to open it again for read access, and then spawn to DCL or > use another terminal session, you will find that Type also has no access > to that file. > > If you define DECC$FILE_SHARING ENABLE before the running the test > again, you will find that both perl and Type will be able to access the > file, except that they both may display it as empty. > > A change in behavior in Perl by default would break any existing scripts > on VMS that depend on the traditional VMS behavior. > > A very high probability of data corruption could result. In which case the data corruption could also be caused by a non-Perl program, surely? Or indeed a Perl program run with file-sharing enabled. [...] > > On 16/08/05, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > >> > >>The performance impact of altering perl's open() to use the CRTL > >>> shr would be significant. Ordinary perl usage would slow to a crawl. > >>> Folks who want shr can load and use VMS::Stdio::vmsopen() and pay > >>> the performance penalty. > > > > I've not noticed vmsopen() causing any read performance problems, and > > I don't recall this being mentioned in regard to DECC$FILE_SHARING. > > > > Is this true with recent CRTLs? > > I am not aware of any performance penalty for turning off the file > locking. The vmsopen() probably allows even better tuning of the open > request to the use of the file than the Perl open() routine. > > If there is a performance penalty between the two calls, I would look > for a different cause. > > -John >
Re: lib/test/simple/t/create.t help with VMS issue needed.
Sebb wrote: > On 15/08/05, Carl Friedberg <[EMAIL PROTECTED]> wrote: > >>sebb wrote: >> >>On VMS, the default behavior makes sense. VMS is a >>multi-user system, and when you open a file, the >>default access is exclusive. VMS has rich file >>sharing semantics, so it is easy to change this >>behavior; but a VMS user expects default access >>to be exclusive. > > I use VMS and I don't ... > > For example, the VMS TYPE command allows one to look at the contents > of batch LOG files, but Perl cannot _read_ the same file without the > work-round above (or vmsopen() with suitable parameters). You are only seeing that behavior because of that the log files for batch jobs explicitly were opened in a way that allows other programs to read them. Note that a large amount of data in the log file may not have yet been written to disk, and is unavailable to the type command. If you write an application that opens a file with the default options and then sleeps, you will find that TYPE and other non-privileged applications will not be able to access the file at all. >>Changing this behavior to conform to perl usage on >>other OSes could be a configuration option; but >>the default "no surprise" behavior should be >>exclusive access. > IMO, the surprise in this case is that Perl can't open a file > that TYPEs OK ... Perl can access any file that TYPEs ok. That is not the issue here. Type has no more privileges or features than Perl does in this case. If pause the lib/test/simple/t/create/t test in the debugger just before Perl tries to open it again for read access, and then spawn to DCL or use another terminal session, you will find that Type also has no access to that file. If you define DECC$FILE_SHARING ENABLE before the running the test again, you will find that both perl and Type will be able to access the file, except that they both may display it as empty. A change in behavior in Perl by default would break any existing scripts on VMS that depend on the traditional VMS behavior. A very high probability of data corruption could result. John Malmberg wrote: : The issue that I can not work around is the one that you did not quote, : and that is that data from the test module has not yet been written to : the disk because the file is not closed, or an fsync() call has not been : made for it. Sounds as though the test may perhaps not be valid... I.e. is it sensible for the test to assume that the file will have been > flushed? If it is assuming that the platform it is running on is compliant with the X/Open Single Unix Specification for such I/O, it should not assume that the file has been flushed unless the file is closed or fsync() has been called. Usually when I have encountered a program that does not work on VMS because the program is expecting a different behavior than documented by X/Open or other official UNIX/ANSI/ISO standards, with in a year, I see of report of a "real" UNIX implementation where that program does not work for the same reason. > On 16/08/05, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: >> The performance impact of altering perl's open() to use the CRTL shr would be significant. Ordinary perl usage would slow to a crawl. Folks who want shr can load and use VMS::Stdio::vmsopen() and pay the performance penalty. I've not noticed vmsopen() causing any read performance problems, and I don't recall this being mentioned in regard to DECC$FILE_SHARING. Is this true with recent CRTLs? I am not aware of any performance penalty for turning off the file locking. The vmsopen() probably allows even better tuning of the open request to the use of the file than the Perl open() routine. If there is a performance penalty between the two calls, I would look for a different cause. -John
Re: lib/test/simple/t/create.t help with VMS issue needed.
On 14/08/05, John E. Malmberg <[EMAIL PROTECTED]> wrote: > Tests 7 and 8 for this script are failing on VMS. > > Test 7 is failing because normally on VMS, unless you specify otherwise, > you get exclusive access to the file, so the second open is failing. > > The logical name DECC$FILE_SHARING defined as "ENABLE" will change VMS > behavior to that of UNIX which will allow test 7 to pass. > > I can probably come up with some code to have the script on VMS make > sure that that value is set and to clear it on exit. Would it not be better to fix the VMS Perl open() call so it works the same as on other OSes? S.
Re: lib/test/simple/t/create.t help with VMS issue needed.
On 15/08/05, Carl Friedberg <[EMAIL PROTECTED]> wrote: > sebb wrote: > > JEM wrote: > > > Test 7 is failing because normally on VMS, unless you > > specify otherwise, > > > you get exclusive access to the file, so the second open is failing. > > > > > > The logical name DECC$FILE_SHARING defined as "ENABLE" will > > change VMS > > > behavior to that of UNIX which will allow test 7 to pass. > > > > > > I can probably come up with some code to have the script on VMS make > > > sure that that value is set and to clear it on exit. > > > > Would it not be better to fix the VMS Perl open() call so it works the > > same as on other OSes? I meant for READ access only here. > > I'll jump in here with my own 2-cents worth. > > On VMS, the default behavior makes sense. VMS is a > multi-user system, and when you open a file, the > default access is exclusive. VMS has rich file > sharing semantics, so it is easy to change this > behavior; but a VMS user expects default access > to be exclusive. I use VMS and I don't ... For example, the VMS TYPE command allows one to look at the contents of batch LOG files, but Perl cannot _read_ the same file without the work-round above (or vmsopen() with suitable parameters). > Changing this behavior to conform to perl usage on > other OSes could be a configuration option; but > the default "no surprise" behavior should be > exclusive access. IMO, the surprise in this case is that Perl can't open a file that TYPEs OK ... > Most of the files I use within Perl are flat text > files where file sharing would not be feasible. Not even _read_ access? > If a VMS programmer is working on a VMS > "RMS indexed" or "RMS relative" file organization, > then a different sharing option might be > appropriate, but that needs to be dealt with in > VMS-specfic .xs files that manipulate those files. > > Again, just my 2 cents. > > Carl Friedberg > [EMAIL PROTECTED] > +1.212.233.5470 > www.comets.com > > >
Re: lib/test/simple/t/create.t help with VMS issue needed.
On 16/08/05, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > > sebb <[EMAIL PROTECTED]> wrote on 08/15/2005 08:34:31 PM: > > > > > Would it not be better to fix the VMS Perl open() call so it works > the > > > > same as on other OSes? > > > > I meant for READ access only here. > > The performance impact of altering perl's open() to use the CRTL > shr would be significant. Ordinary perl usage would slow to a crawl. > Folks who want shr can load and use VMS::Stdio::vmsopen() and pay > the performance penalty. I've not noticed vmsopen() causing any read performance problems, and I don't recall this being mentioned in regard to DECC$FILE_SHARING. Is this true with recent CRTLs?
Re: lib/test/simple/t/create.t help with VMS issue needed.
On 16/08/05, John E. Malmberg <[EMAIL PROTECTED]> wrote: > sebb wrote: > > On 14/08/05, John E. Malmberg <[EMAIL PROTECTED]> wrote: > > > >>Tests 7 and 8 for this script are failing on VMS. > >> > >>Test 7 is failing because normally on VMS, unless you specify otherwise, > >>you get exclusive access to the file, so the second open is failing. > >> > >>The logical name DECC$FILE_SHARING defined as "ENABLE" will change VMS > >>behavior to that of UNIX which will allow test 7 to pass. > >> > >>I can probably come up with some code to have the script on VMS make > >>sure that that value is set and to clear it on exit. > > > > Would it not be better to fix the VMS Perl open() call so it works the > > same as on other OSes? > > Carl Friedberg explained about the VMS default behavior. > > The logical name DECC$FILE_SHARING will locally change the behavior of > applications to be follow it, and this can be turned on and off locally > inside of a Perl script on VMS now. It turns off all file locking by > applications that open files through the C runtime support. Yes, I just think this should not be necessary. > The issue that I can not work around is the one that you did not quote, > and that is that data from the test module has not yet been written to > the disk because the file is not closed, or an fsync() call has not been > made for it. Sounds as though the test may perhaps not be valid... I.e. is it sensible for the test to assume that the file will have been flushed? > -John > [EMAIL PROTECTED] > Personal Opinion Only >
RE: lib/test/simple/t/create.t help with VMS issue needed.
sebb wrote: > JEM wrote: > > Test 7 is failing because normally on VMS, unless you > specify otherwise, > > you get exclusive access to the file, so the second open is failing. > > > > The logical name DECC$FILE_SHARING defined as "ENABLE" will > change VMS > > behavior to that of UNIX which will allow test 7 to pass. > > > > I can probably come up with some code to have the script on VMS make > > sure that that value is set and to clear it on exit. > > Would it not be better to fix the VMS Perl open() call so it works the > same as on other OSes? I'll jump in here with my own 2-cents worth. On VMS, the default behavior makes sense. VMS is a multi-user system, and when you open a file, the default access is exclusive. VMS has rich file sharing semantics, so it is easy to change this behavior; but a VMS user expects default access to be exclusive. Changing this behavior to conform to perl usage on other OSes could be a configuration option; but the default "no surprise" behavior should be exclusive access. Most of the files I use within Perl are flat text files where file sharing would not be feasible. If a VMS programmer is working on a VMS "RMS indexed" or "RMS relative" file organization, then a different sharing option might be appropriate, but that needs to be dealt with in VMS-specfic .xs files that manipulate those files. Again, just my 2 cents. Carl Friedberg [EMAIL PROTECTED] +1.212.233.5470 www.comets.com
Re: lib/test/simple/t/create.t help with VMS issue needed.
sebb <[EMAIL PROTECTED]> wrote on 08/15/2005 08:34:31 PM: > > > Would it not be better to fix the VMS Perl open() call so it works the > > > same as on other OSes? > > I meant for READ access only here. The performance impact of altering perl's open() to use the CRTL shr would be significant. Ordinary perl usage would slow to a crawl. Folks who want shr can load and use VMS::Stdio::vmsopen() and pay the performance penalty. Peter Prymmer
Re: lib/test/simple/t/create.t help with VMS issue needed.
sebb wrote: On 14/08/05, John E. Malmberg <[EMAIL PROTECTED]> wrote: Tests 7 and 8 for this script are failing on VMS. Test 7 is failing because normally on VMS, unless you specify otherwise, you get exclusive access to the file, so the second open is failing. The logical name DECC$FILE_SHARING defined as "ENABLE" will change VMS behavior to that of UNIX which will allow test 7 to pass. I can probably come up with some code to have the script on VMS make sure that that value is set and to clear it on exit. Would it not be better to fix the VMS Perl open() call so it works the same as on other OSes? Carl Friedberg explained about the VMS default behavior. The logical name DECC$FILE_SHARING will locally change the behavior of applications to be follow it, and this can be turned on and off locally inside of a Perl script on VMS now. It turns off all file locking by applications that open files through the C runtime support. The issue that I can not work around is the one that you did not quote, and that is that data from the test module has not yet been written to the disk because the file is not closed, or an fsync() call has not been made for it. -John [EMAIL PROTECTED] Personal Opinion Only
lib/test/simple/t/create.t help with VMS issue needed.
Tests 7 and 8 for this script are failing on VMS.
Test 7 is failing because normally on VMS, unless you specify otherwise,
you get exclusive access to the file, so the second open is failing.
The logical name DECC$FILE_SHARING defined as "ENABLE" will change VMS
behavior to that of UNIX which will allow test 7 to pass.
I can probably come up with some code to have the script on VMS make
sure that that value is set and to clear it on exit.
Test 8 is more of a problem. The issue is that the buffers for the
other stream written by the new_tb->output("some_file") have not made it
to disk, so they can not yet be read by the new input stream to pass the
test.
There does not seem to be a method of explicitly closing or flushing the
output stream being written to by Test::Builder.
Any ideas on how to resolve this?
-John
[EMAIL PROTECTED]
Personal Opinion Only
