Idea for a smoke setup: read-only home directory

[David Golden]

So I keep noticing "litter" in my home directory while running the
smoker.  It looks like either temp files or default locations for
things.  I consider that bad behavior for tests.

I'm considering making my home directory read-only and then smoking
all of CPAN again to see what fails its tests.

Do people think this would be useful or am I just getting overly
annoyed without good reason?

David

Idea for a smoke setup: read-only home directory

[Serguei Trouchelle]

David Golden wrote:


So I keep noticing "litter" in my home directory while running the
smoker.  It looks like either temp files or default locations for
things.  I consider that bad behavior for tests.

I'm considering making my home directory read-only and then smoking
all of CPAN again to see what fails its tests.

Do people think this would be useful or am I just getting overly
annoyed without good reason?


When smoking MSWin32 perl, I have some files written to /, /tmp/, or 
even /userdata/smueller/ (and have to clean up all this stuff).


So yes, it's a good idea.

--
Serguei Trouchelle

Re: Idea for a smoke setup: read-only home directory

[Andy Armstrong]

On 18 Jun 2008, at 02:22, David Golden wrote:

So I keep noticing "litter" in my home directory while running the
smoker.  It looks like either temp files or default locations for
things.  I consider that bad behavior for tests.

I'm considering making my home directory read-only and then smoking
all of CPAN again to see what fails its tests.

Do people think this would be useful or am I just getting overly
annoyed without good reason?



How about a user who's home directory is a unionfs mount with the  
basic file structure R/O and a disposable R/W layer on top of it?


--
Andy Armstrong, Hexten

Re: Idea for a smoke setup: read-only home directory

[David Golden]

On Tue, Jun 17, 2008 at 9:26 PM, Andy Armstrong <[EMAIL PROTECTED]> wrote:
> How about a user who's home directory is a unionfs mount with the basic file
> structure R/O and a disposable R/W layer on top of it?

It's not the clutter that concerns me -- it's that I don't think
testers should be writing to $ENV{HOME} during testing.  So setting it
RO would probably cause some FAIL reports and flush out the
responsible parties.

David

Re: Idea for a smoke setup: read-only home directory

[Andy Armstrong]

On 18 Jun 2008, at 02:28, David Golden wrote:
On Tue, Jun 17, 2008 at 9:26 PM, Andy Armstrong <[EMAIL PROTECTED]>  
wrote:
How about a user who's home directory is a unionfs mount with the  
basic file

structure R/O and a disposable R/W layer on top of it?


s/who's/whose/


It's not the clutter that concerns me -- it's that I don't think
testers should be writing to $ENV{HOME} during testing.  So setting it
RO would probably cause some FAIL reports and flush out the
responsible parties.


Ah yes - in that case I agree. Out of interest what kind of stuff are
people dumping there?

--
Andy Armstrong, Hexten

Re: Idea for a smoke setup: read-only home directory

[David Golden]

On Tue, Jun 17, 2008 at 9:32 PM, Andy Armstrong <[EMAIL PROTECTED]> wrote:
> Ah yes - in that case I agree. Out of interest what kind of stuff are
> people dumping there?

* "mail-audit.log" -- that appears to be from Mail::Audit.

* "abc" -- containing the string "hello"

* "cookies.txt" -- containing the string "#LWP-Cookies-1.0"

* "cgi-bin" -- directory

There also seem to be some things in ~/tmp/

That's on linux.  It's worse on Win32.  Lots of stuff winds up dumped
in c:\.  In one case, I think there were several thousand tiny
directories.

David

Re: Idea for a smoke setup: read-only home directory

[Elliot Shank]

David Golden wrote:

So I keep noticing "litter" in my home directory while running the
smoker.  It looks like either temp files or default locations for
things.  I consider that bad behavior for tests.

I'm considering making my home directory read-only and then smoking
all of CPAN again to see what fails its tests.

Do people think this would be useful or am I just getting overly
annoyed without good reason?


Please do.  Forget "mere" testing, I don't want that happening with modules I 
install for actual use.

Re: Idea for a smoke setup: read-only home directory

[David Golden]

On Tue, Jun 17, 2008 at 10:32 PM, Elliot Shank <[EMAIL PROTECTED]> wrote:
> Please do.  Forget "mere" testing, I don't want that happening with modules
> I install for actual use.

Well, some default to that.  Or at least to creating .foo config files
or directories.  But still, I don't think that should be what happens
at test time.

Though I will say that from experience, mocking a home directory isn't
the easiest to do and always get right.

David

Re: Idea for a smoke setup: read-only home directory

[Andreas J. Koenig]

> On Tue, 17 Jun 2008 23:23:00 -0400, "David Golden" <[EMAIL PROTECTED]> 
> said:

  > Though I will say that from experience, mocking a home directory isn't
  > the easiest to do and always get right.

adduser ?

-- 
andreas

Re: Idea for a smoke setup: read-only home directory

[David Golden]

On Wed, Jun 18, 2008 at 1:01 AM, Andreas J. Koenig
<[EMAIL PROTECTED]> wrote:
>> On Tue, 17 Jun 2008 23:23:00 -0400, "David Golden" <[EMAIL PROTECTED]> 
>> said:
>
>  > Though I will say that from experience, mocking a home directory isn't
>  > the easiest to do and always get right.
>
> adduser ?

Funny.  No, I'd *really* prefer if test scripts didn't try to add new
user directories.  :-)

My "standard" approach is to have a test setup routine that (a)
creates a temp directory and (b) mocks File::HomeDir to make sure that
all it's methods return the temp directory.  The only real "trick" is
making sure that $INC{File/Homedir.pm} gets set before anything might
possibly load File::HomeDir.

I should probably just wrap it up into a Test::MockHomeDir module.

David

Re: Idea for a smoke setup: read-only home directory

[Barbie]

On Wed, Jun 18, 2008 at 12:08:46PM +0300, Serguei Trouchelle wrote:
> David Golden wrote:
> 
> >So I keep noticing "litter" in my home directory while running the
> >smoker.  It looks like either temp files or default locations for
> >things.  I consider that bad behavior for tests.
> >
> >I'm considering making my home directory read-only and then smoking
> >all of CPAN again to see what fails its tests.
> >
> >Do people think this would be useful or am I just getting overly
> >annoyed without good reason?
> 
> When smoking MSWin32 perl, I have some files written to /, /tmp/, or 
> even /userdata/smueller/ (and have to clean up all this stuff).

I used to get tons of this sort of thing. It was annoying when modules
would unwrap into the .cpanplus/build directory, as opposed to their
unquiely named build directory below that. Though I would say the home
directory would be easier to spot. 

I can forgive /tmp somewhat, but again it doesn't always work on other
OSs, and assuming you're working on a Unix style system did make
distributions fail a few times when I was testing.

> So yes, it's a good idea.

Agreed. CPANTS checks for this in the unwrapping, but we currently don't
have any suggestions for the actual testing.

Cheers,
Barbie.
-- 
Birmingham Perl Mongers 
Memoirs Of A Roadie 

Re: Idea for a smoke setup: read-only home directory

[David Cantrell]

On Tue, Jun 17, 2008 at 09:40:02PM -0400, David Golden wrote:

> There also seem to be some things in ~/tmp/
> That's on linux.  It's worse on Win32.  Lots of stuff winds up dumped
> in c:\.  In one case, I think there were several thousand tiny
> directories.

On *nix I periodically clean out several hundred files from /tmp.
I assume that they're created by File::Temp.

-- 
David Cantrell | Godless Liberal Elitist

Us Germans take our humour very seriously
  -- German cultural attache talking to the Today Programme,
 about the German supposed lack of a sense of humour, 29 Aug 2001

Re: Idea for a smoke setup: read-only home directory

[David Golden]

On Wed, Jun 18, 2008 at 9:11 AM, David Cantrell <[EMAIL PROTECTED]> wrote:
> On *nix I periodically clean out several hundred files from /tmp.
> I assume that they're created by File::Temp.

Or just created by hand and ignored.

David

Re: Idea for a smoke setup: read-only home directory

[David Cantrell]

On Wed, Jun 18, 2008 at 06:53:30AM -0400, David Golden wrote:

> My "standard" approach is to have a test setup routine that (a)
> creates a temp directory and (b) mocks File::HomeDir to make sure that
> all it's methods return the temp directory.  The only real "trick" is
> making sure that $INC{File/Homedir.pm} gets set before anything might
> possibly load File::HomeDir.
> 
> I should probably just wrap it up into a Test::MockHomeDir module.

I wouldn't bother, cos there's so much stuff out there that will look
directly at ~ or $ENV{HOME}, or read /etc/passwd by hand, or ...

FWIW, instead of mocking File::HomeDir, my approach would be to wrap all
uses of it in a tiny function:

  sub _gethomedir { return File::HomeDir->my_home(); }

and then when testing my module that uses it, do this in the test suite:

  use File::Temp;
  my $fakehome = File::Temp->newdir();

  *Module::_gethomedir = sub { return $fakehome };

-- 
David Cantrell | Enforcer, South London Linguistic Massive

  engineer: n. one who, regardless of how much effort he puts in
to a job, will never satisfy either the suits or the scientists

Re: Idea for a smoke setup: read-only home directory

[David Golden]

On Wed, Jun 18, 2008 at 9:34 AM, David Cantrell <[EMAIL PROTECTED]> wrote:
> FWIW, instead of mocking File::HomeDir, my approach would be to wrap all
> uses of it in a tiny function:
>
>  sub _gethomedir { return File::HomeDir->my_home(); }
>
> and then when testing my module that uses it, do this in the test suite:

That works if it's all in my control.  But if I'm depending on other
modules and *they* look at File::HomeDir, then I need to mock it.

David

Re: Idea for a smoke setup: read-only home directory

[imacat]

On Tue, 17 Jun 2008 21:22:50 -0400
"David Golden" <[EMAIL PROTECTED]> wrote:
> I'm considering making my home directory read-only and then smoking
> all of CPAN again to see what fails its tests.

In my test script I create a temporarily directory as the home
directory and set the HOME environemnt variable to it.  After the tests
are finished that directory is removed with File::Path::rmtree() and the
HOME environment variable is restored.

This work for me in most cases.

--
Best regards,
imacat ^_*' <[EMAIL PROTECTED]>
PGP Key: http://www.imacat.idv.tw/me/pgpkey.asc

<> News: http://www.wov.idv.tw/
Tavern IMACAT's: http://www.imacat.idv.tw/
TLUG List Manager: http://lists.linux.org.tw/cgi-bin/mailman/listinfo/tlug


pgpSGwIWbnlHx.pgp
Description: PGP signature

Re: Idea for a smoke setup: read-only home directory

[James E Keenan]

David Cantrell wrote:

On Tue, Jun 17, 2008 at 09:40:02PM -0400, David Golden wrote:


There also seem to be some things in ~/tmp/
That's on linux.  It's worse on Win32.  Lots of stuff winds up dumped
in c:\.  In one case, I think there were several thousand tiny
directories.


On *nix I periodically clean out several hundred files from /tmp.
I assume that they're created by File::Temp.



In the course of my testing of Parrot's configuration and build tools, I 
have had *many* occasions to create temporary files and directories.  At 
one point last year I got complaints.  We found that code like this did 
the trick:


use File::Temp qw( tempdir tempfile );
my $tdir = tempdir( CLEANUP => 1 );
my ( $tmpfile, $fname ) = tempfile( UNLINK => 1 );

Since these are functions from a core module, there is no excuse for any 
CPAN author not to use them with these cleanup options.  (I'd say that 
proper cleanup of temp files created during testing should be a 
'kwalitee' metric -- but I don't want to get into that argument!)


Jim Keenan

Re: Idea for a smoke setup: read-only home directory

[James E Keenan]

David Golden wrote:

So I keep noticing "litter" in my home directory while running the
smoker.  It looks like either temp files or default locations for
things.  I consider that bad behavior for tests.



I have been guilty of that *in the past*.

During my rewrite of ExtUtils::ModuleMaker in July-Sept 2005, I wanted 
to add functionality for a .modulemakerrc file which would enable an 
experienced user to do short-cuts around the prompts in the 
'modulemaker' command-line utility.  So the question, "Can I save a file 
under a user's home directory?" became important.  I know that at one 
point I was unintentionally leaving files on tester's servers. 
Eventually I deduced what was happening by examining FAIL reports I was 
getting from imacat.


I corrected the problem and, at the suggestion of Michael Graham, made 
that solution generally available on CPAN:  File::Save::Home.



I'm considering making my home directory read-only and then smoking
all of CPAN again to see what fails its tests.


I haven't had to think much of File::Save::Home recently (except insofar 
as an alternative to File::HomeDir -- but that's a separate thread), so 
I don't know how such a change in your policies would affect it.  But 
since people are probably not aware of the problem, I would hope that 
you would first try to encourage good practices (which may or may not 
include File::Save::Home) and only make your directory read-only as a 
last resort.


Jim Keenan

Re: Idea for a smoke setup: read-only home directory

[James E Keenan]

imacat wrote:


In my test script I create a temporarily directory as the home
directory and set the HOME environemnt variable to it.  After the tests
are finished that directory is removed with File::Path::rmtree() and the
HOME environment variable is restored.



A very good suggestion -- and one which contributes to the pristine 
environment you keep for testing, which has helped me uncover subtle 
bugs in my modules in the past.


jimk

Re: Idea for a smoke setup: read-only home directory

[Barbie]

On Fri, Jul 04, 2008 at 10:02:24PM -0400, James E Keenan wrote:
> David Cantrell wrote:
> >On Tue, Jun 17, 2008 at 09:40:02PM -0400, David Golden wrote:
> >
> >>There also seem to be some things in ~/tmp/
> >>That's on linux.  It's worse on Win32.  Lots of stuff winds up dumped
> >>in c:\.  In one case, I think there were several thousand tiny
> >>directories.
> >
> >On *nix I periodically clean out several hundred files from /tmp.
> >I assume that they're created by File::Temp.
> >
> 
> In the course of my testing of Parrot's configuration and build tools, I 
> have had *many* occasions to create temporary files and directories.  At 
> one point last year I got complaints.  We found that code like this did 
> the trick:
> 
> use File::Temp qw( tempdir tempfile );
> my $tdir = tempdir( CLEANUP => 1 );
> my ( $tmpfile, $fname ) = tempfile( UNLINK => 1 );
> 
> Since these are functions from a core module, there is no excuse for any 
> CPAN author not to use them with these cleanup options.  (I'd say that 
> proper cleanup of temp files created during testing should be a 
> 'kwalitee' metric -- but I don't want to get into that argument!)

Funny you should mention that. I've been slowly working through the
rewrite of YACSmoke. One of the aspects that I've been thinking about is
having CPAN Testers run the often disputed "pod.t" and "podcover.t" type
tests.

I haven't had a chance to think how it could be done, but there must be
some way to easily inject tests, that are marked in someway so that the
author knows that they where they appeared from, that can automatically
add this kind of testing. This would allow authors to not include them
with their distribution, and CPANTS draw information from the test
reports.

However, the major problem I can see with that is if a distribution is
well respected, but fails these tests. I can see a huge backlash. The
best alternative I had was for there to be two sets of reports, the
first as per the author package, then a second with the extra tests,
which is sent to a different server for informational purposes only.
This would then feed into some kwalitee metric of CPANTS.

We could then build on this to check for leaving behind temp files,
writing to directories they shouldn't and all manner of other things.

Not practical in the current setups, but something to consider for the
future.

Cheers,
Barbie.
-- 
Birmingham Perl Mongers 
Memoirs Of A Roadie 

Re: Idea for a smoke setup: read-only home directory

[James E Keenan]

On Jul 5, 2008, at 7:58 AM, Barbie wrote:


On Fri, Jul 04, 2008 at 10:02:24PM -0400, James E Keenan wrote:

David Cantrell wrote:

On Tue, Jun 17, 2008 at 09:40:02PM -0400, David Golden wrote:


There also seem to be some things in ~/tmp/
That's on linux.  It's worse on Win32.  Lots of stuff winds up  
dumped

in c:\.  In one case, I think there were several thousand tiny
directories.


On *nix I periodically clean out several hundred files from /tmp.
I assume that they're created by File::Temp.



In the course of my testing of Parrot's configuration and build  
tools, I
have had *many* occasions to create temporary files and  
directories.  At
one point last year I got complaints.  We found that code like  
this did

the trick:

use File::Temp qw( tempdir tempfile );
my $tdir = tempdir( CLEANUP => 1 );
my ( $tmpfile, $fname ) = tempfile( UNLINK => 1 );

Since these are functions from a core module, there is no excuse  
for any
CPAN author not to use them with these cleanup options.  (I'd say  
that

proper cleanup of temp files created during testing should be a
'kwalitee' metric -- but I don't want to get into that argument!)


Funny you should mention that. I've been slowly working through the
rewrite of YACSmoke. One of the aspects that I've been thinking  
about is
having CPAN Testers run the often disputed "pod.t" and "podcover.t"  
type

tests.


Yecch!  I particularly dislike the pod.t and podcover.t tests.  I  
specifically do not include them as default selections in  
ExtUtils::ModuleMaker.



[snip]




However, the major problem I can see with that is if a distribution is
well respected, but fails these tests. I can see a huge backlash.


Starting with me.

The CPAN distro of which I am most proud and for which I get the most  
positive feedback is List-Compare.  But for years this distro got red  
marks in the CPANTS columns for those two tests.  Why?  Because  
instead of organizing the POD with the subroutine names as "=head" or  
"=item" elements, I organized it by the way humans would pose a  
question, e.g., "How do I get the intersection of more than two  
sets?"  Those tests -- more likely podcover.t -- refused to recognize  
that I had written documentation which, if printed, would run to more  
than twenty pages.


Apart from these objections, to include pod.t and podcover.t in CPAN  
Testers would constitute scope creep.  As I see it, automated CPAN  
testing exists first and foremost to provide module authors with  
immediate feedback on these questions:


*  Were we able to download your distro from the CPAN and prepare it  
to run?
*  Were we able to build it successfully with either  
ExtUtils::MakeMaker or Module::Build?

*  Which tests, if any, did it fail?

Now, as the point of this thread is to prevent garbage from getting  
on the machines of CPAN testers, you could argue that a fourth  
question should be:


*  If the distro needs to clean up after itself, does it do so?

But once you start adding "kwalitee" features, you diverge from your  
core mission in a misguided attempt to be all things to all people.


Please don't do it!

Thank you very much.

Jim Keenan

Re: Idea for a smoke setup: read-only home directory

[Barbie]

On Sat, Jul 05, 2008 at 08:31:07AM -0400, James E Keenan wrote:
> 
> >Funny you should mention that. I've been slowly working through the
> >rewrite of YACSmoke. One of the aspects that I've been thinking  
> >about is
> >having CPAN Testers run the often disputed "pod.t" and "podcover.t"  
> >type
> >tests.
> 
> Yecch!  I particularly dislike the pod.t and podcover.t tests.  I  
> specifically do not include them as default selections in  
> ExtUtils::ModuleMaker.

I can appreciate that podcover.t might not be appropriate, but the pod.t
would be. Having well formed POD as documentation is standard good
practice, or at least it should be.

> >However, the major problem I can see with that is if a distribution is
> >well respected, but fails these tests. I can see a huge backlash.
> 
> Starting with me.

Hence why I suggested having a separate server that collated these
additional tests that were not part of any report sent to the CPAN
Testers server. Provided that tests were appropriate, it would be good
to having something that required running the distribution that fed into
CPANTS. However, what is appropriate, and who decides that, are two
questions for another time :)

As stated, I only mentioned these as something to perhaps think about
for the future, as CPAN Testers are in the postion to be actually
running the tests anyway. CPANTS isn't.

> But once you start adding "kwalitee" features, you diverge from your  
> core mission in a misguided attempt to be all things to all people.

I don't believe we are. The core mission is to give someone planning to
install a distribution a heads up as to whether there might be any
issues. The side effect of informing the author has always been just
that, a side effect :)

If the author isn't testing whether their distribution is broken, then
the CPAN Testers could potentially do that. However, I do think these
are perhaps metrics that are better fed to something like CPANTS, or
possibly a service that can be requested by the author, once we have the
admin facility for CPAN Testers.

Cheers,
Barbie.
-- 
Birmingham Perl Mongers 
Memoirs Of A Roadie 

Re: Idea for a smoke setup: read-only home directory

[David Cantrell]

Barbie wrote:

On Sat, Jul 05, 2008 at 08:31:07AM -0400, James E Keenan wrote:

Funny you should mention that. I've been slowly working through the
rewrite of YACSmoke. One of the aspects that I've been thinking  
about is
having CPAN Testers run the often disputed "pod.t" and "podcover.t"  
type

tests.
Yecch!  I particularly dislike the pod.t and podcover.t tests.  I  
specifically do not include them as default selections in  
ExtUtils::ModuleMaker.

I can appreciate that podcover.t might not be appropriate, but the pod.t
would be. Having well formed POD as documentation is standard good
practice, or at least it should be.


The Number-Phone distribution has well-formed POD AFAIK.  But just you 
try to test that using the usual tools.  The several lines of binary 
gibberish beginning with = in Number::Phone::UK::Data's __DATA__ section 
break them badly.


I've looked at fixing the tools, but got bogged down in figuring out 
whether /^__DATA__/ really was __DATA__ or just someone being awkward 
with quoted text.


--
David Cantrell | Official London Perl Mongers Bad Influence

Anyone willing to give up a little fun for tolerance deserves neither

Re: [SPAM] Re: Idea for a smoke setup: read-only home directory

[Serguei Trouchelle]

David Cantrell wrote:


There also seem to be some things in ~/tmp/
That's on linux.  It's worse on Win32.  Lots of stuff winds up dumped
in c:\.  In one case, I think there were several thousand tiny
directories.



On *nix I periodically clean out several hundred files from /tmp.
I assume that they're created by File::Temp.


File::Temp::tempdir() makes a directory in $ENV{'TEMP'}, unless 
explicitly specified, so it should be a module author's fault, not 
File::Temp's.


--
Serguei Trouchelle