Re: [OpenAFS] buildbot and packages

[Derrick Brashear]

> That's not a big deal. What's a big deal is I'll spend about 10 or 15 more
> hours arguing on the mailing list or on gerrit for a very simple change to
> make sure the default builds ensure I can always send you a reasonable
> stack trace.

You clearly have a script you're using to make these builds. Edit it
and save yourself
this huge pile of time you're using to post.

What you wish to do is your business. If other people think it's a
great idea, fine. So far no
reviewer (me included) does. I also think your definition of "fix"
deserves the quotes I
include here.

-- 
Derrick
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

On Tue, Sep 18, 2012 at 01:12:33PM -0400, Derrick Brashear wrote:
> On Tue, Sep 18, 2012 at 12:54 PM, Troy Benjegerdes  wrote:
> > [snip]
> 
> >> > If there's a perceived performance impact to having debug on in a release
> >> > build, then I want to see a full QA test and benchmark results showing 
> >> > that
> >> > it's actually slowing things down.
> >>
> >> Well, as soon as you finish it, feel free to share the results.
> >>
> >> We're waiting with bated breath.
> >
> > Done. http://gerrit.openafs.org/#change,8137
> >
> > My rate to prove that the perfomance impact of this change is negligible
> > for most all use cases is $125/hour. If I am contracted to perform a full
> > QA test and benchmark run for this change, I will refund half of my fee to
> > the first organization that can demonstrate that they are one of the edge
> > cases that  does actually see a performance degredation from default debug.
> 
> You said you wanted to see it. When you make enough money harvesting those
> soybeans to pay yourself, let us know what you find.

I have about half of them sold at a good price. But I need a working ipv6 and
rxgk/rxk5 capable to be able to store my yield data and notes about reverse
engineering the wiring diagram on the header control height. I'm going to hack
on those things in my own fork.

But if I see something broken in master, and propose a simple fix, I'm going 
to try and send it back upstream.

Now, the problem is that **YOU** asked for me to rebuild with --enable-debug
and I spent a couple of billable hours finding out it's a heisenbug that goes
away when I enable debugging.

That's not a big deal. What's a big deal is I'll spend about 10 or 15 more 
hours arguing on the mailing list or on gerrit for a very simple change to
make sure the default builds ensure I can always send you a reasonable
stack trace.

So if there's a better alternative to http://gerrit.openafs.org/#change,8137
please show me the code. I'd be perfectly happy if you had some nightly (or
weekly) builds that I can just run through my own test suite on a VM.

It's busted. Now, please, pick one of the following:

1) accept my fix
2) come up with something better for free
3) pay me to come up with something better, and prove it's better
4) find a client to pay you to come up with something better


(FYI, the fact that I'm still having this argument in the first place
is a good sign that I've tried everything else, and OpenAFS is the one
filesystem that I have any level of confidence I'll be able to get something
back out of it in 30 years from now)
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Derrick Brashear]

On Tue, Sep 18, 2012 at 12:54 PM, Troy Benjegerdes  wrote:
> [snip]

>> > If there's a perceived performance impact to having debug on in a release
>> > build, then I want to see a full QA test and benchmark results showing that
>> > it's actually slowing things down.
>>
>> Well, as soon as you finish it, feel free to share the results.
>>
>> We're waiting with bated breath.
>
> Done. http://gerrit.openafs.org/#change,8137
>
> My rate to prove that the perfomance impact of this change is negligible
> for most all use cases is $125/hour. If I am contracted to perform a full
> QA test and benchmark run for this change, I will refund half of my fee to
> the first organization that can demonstrate that they are one of the edge
> cases that  does actually see a performance degredation from default debug.

You said you wanted to see it. When you make enough money harvesting those
soybeans to pay yourself, let us know what you find.
-- 
Derrick
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

[snip]
> >> None of those steps knows about another, nor should they. If you want
> >> to enable debugging, just do it.
> >> If you want to provide a script which does debug builds, do it.
> >> Anything else is pointless complexity.
> >
> > Debug symbols are pointless complexity ;)
> >
> > If they are something you are going to ask a bug reporter for, then my
> > argument is ./configure (no arguments) should 'do the right thing' so
> > you can get all the information you need in a bug report with no extra
> > retests required.
> 
> If you know enough to use configure (not a frontend script, but configure) and
> end up with an AFS you install, I assume you have a small amount of clue and
> can deal. If you want to use a frontend script, fix that script.
> 
> > If there's a perceived performance impact to having debug on in a release
> > build, then I want to see a full QA test and benchmark results showing that
> > it's actually slowing things down.
> 
> Well, as soon as you finish it, feel free to share the results.
> 
> We're waiting with bated breath.

Done. http://gerrit.openafs.org/#change,8137

My rate to prove that the perfomance impact of this change is negligible
for most all use cases is $125/hour. If I am contracted to perform a full
QA test and benchmark run for this change, I will refund half of my fee to
the first organization that can demonstrate that they are one of the edge
cases that  does actually see a performance degredation from default debug.

I think I just saved the OpenAFS project at least $25,000 if we skip the
testing and accept the change.

If you want some justification about why I'm qualified for such a rate,
have a look at

http://www.scl.ameslab.gov/Publications/Brett/storage_challenge_sc06_presentation.pdf

You also might be amused to know that work was done with PVFS servers
running using OpenAFS as the root filesystem.

In the meantime, I have a combine I have to get ready to go pick soybeans.
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[chas williams - CONTRACTOR]

On Tue, 18 Sep 2012 10:39:45 -0400
Derrick Brashear  wrote:


> > What documentation on libtool/autoconf/etc/whatever should I be looking at
> > to make '--enable-checking' and '--enable-debug' be the default when I do
> > './regen && ./configure && make check' so I can submit a patch for master.
> 
> Frankly, I'd patch either the human or the script which runs "'./regen
> && ./configure && make check'" as it's
> gonna be less work.

cute.  updating http://wiki.openafs.org/AFSLore/GitDevelopers/ is
probably a good idea.  people should make check before they git submit
anyway.
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Derrick Brashear]

On Tue, Sep 18, 2012 at 10:47 AM, Troy Benjegerdes  wrote:
> On Tue, Sep 18, 2012 at 10:39:45AM -0400, Derrick Brashear wrote:
>> On Tue, Sep 18, 2012 at 10:31 AM, Troy Benjegerdes  wrote:
>> > On Mon, Sep 17, 2012 at 08:06:39PM +0100, Simon Wilkinson wrote:
>> >>
>> >> On 17 Sep 2012, at 19:54, Troy Benjegerdes wrote:
>> >> > If 'rebuild with debug' symbols is the answer to find the segfault, 
>> >> > then why
>> >> > don't we change './regen && ./configure && make check' to turn on debug 
>> >> > symbols
>> >> > by default (at least in master.. we can turn it back off in a release)
>> >>
>> >> If you are developing, then you should be running configure with at least 
>> >> --enable-checking and --enable-debug
>> >
>> > What documentation on libtool/autoconf/etc/whatever should I be looking at
>> > to make '--enable-checking' and '--enable-debug' be the default when I do
>> > './regen && ./configure && make check' so I can submit a patch for master.
>>
>> Frankly, I'd patch either the human or the script which runs "'./regen
>> && ./configure && make check'" as it's
>> gonna be less work.
>>
>> None of those steps knows about another, nor should they. If you want
>> to enable debugging, just do it.
>> If you want to provide a script which does debug builds, do it.
>> Anything else is pointless complexity.
>
> Debug symbols are pointless complexity ;)
>
> If they are something you are going to ask a bug reporter for, then my
> argument is ./configure (no arguments) should 'do the right thing' so
> you can get all the information you need in a bug report with no extra
> retests required.

If you know enough to use configure (not a frontend script, but configure) and
end up with an AFS you install, I assume you have a small amount of clue and
can deal. If you want to use a frontend script, fix that script.

> If there's a perceived performance impact to having debug on in a release
> build, then I want to see a full QA test and benchmark results showing that
> it's actually slowing things down.

Well, as soon as you finish it, feel free to share the results.

We're waiting with bated breath.



-- 
Derrick
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

On Tue, Sep 18, 2012 at 10:39:45AM -0400, Derrick Brashear wrote:
> On Tue, Sep 18, 2012 at 10:31 AM, Troy Benjegerdes  wrote:
> > On Mon, Sep 17, 2012 at 08:06:39PM +0100, Simon Wilkinson wrote:
> >>
> >> On 17 Sep 2012, at 19:54, Troy Benjegerdes wrote:
> >> > If 'rebuild with debug' symbols is the answer to find the segfault, then 
> >> > why
> >> > don't we change './regen && ./configure && make check' to turn on debug 
> >> > symbols
> >> > by default (at least in master.. we can turn it back off in a release)
> >>
> >> If you are developing, then you should be running configure with at least 
> >> --enable-checking and --enable-debug
> >
> > What documentation on libtool/autoconf/etc/whatever should I be looking at
> > to make '--enable-checking' and '--enable-debug' be the default when I do
> > './regen && ./configure && make check' so I can submit a patch for master.
> 
> Frankly, I'd patch either the human or the script which runs "'./regen
> && ./configure && make check'" as it's
> gonna be less work.
> 
> None of those steps knows about another, nor should they. If you want
> to enable debugging, just do it.
> If you want to provide a script which does debug builds, do it.
> Anything else is pointless complexity.

Debug symbols are pointless complexity ;)

If they are something you are going to ask a bug reporter for, then my 
argument is ./configure (no arguments) should 'do the right thing' so 
you can get all the information you need in a bug report with no extra
retests required.

If there's a perceived performance impact to having debug on in a release
build, then I want to see a full QA test and benchmark results showing that
it's actually slowing things down.
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Derrick Brashear]

On Tue, Sep 18, 2012 at 10:31 AM, Troy Benjegerdes  wrote:
> On Mon, Sep 17, 2012 at 08:06:39PM +0100, Simon Wilkinson wrote:
>>
>> On 17 Sep 2012, at 19:54, Troy Benjegerdes wrote:
>> > If 'rebuild with debug' symbols is the answer to find the segfault, then 
>> > why
>> > don't we change './regen && ./configure && make check' to turn on debug 
>> > symbols
>> > by default (at least in master.. we can turn it back off in a release)
>>
>> If you are developing, then you should be running configure with at least 
>> --enable-checking and --enable-debug
>
> What documentation on libtool/autoconf/etc/whatever should I be looking at
> to make '--enable-checking' and '--enable-debug' be the default when I do
> './regen && ./configure && make check' so I can submit a patch for master.

Frankly, I'd patch either the human or the script which runs "'./regen
&& ./configure && make check'" as it's
gonna be less work.

None of those steps knows about another, nor should they. If you want
to enable debugging, just do it.
If you want to provide a script which does debug builds, do it.
Anything else is pointless complexity.

> When we branch for release this should get turned off, but *only* after
> someone has complete QA and benchmark testing showing exactly what the
> performance impact of the debugging is.



-- 
Derrick
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

On Mon, Sep 17, 2012 at 08:06:39PM +0100, Simon Wilkinson wrote:
> 
> On 17 Sep 2012, at 19:54, Troy Benjegerdes wrote:
> > If 'rebuild with debug' symbols is the answer to find the segfault, then why
> > don't we change './regen && ./configure && make check' to turn on debug 
> > symbols
> > by default (at least in master.. we can turn it back off in a release)
> 
> If you are developing, then you should be running configure with at least 
> --enable-checking and --enable-debug

What documentation on libtool/autoconf/etc/whatever should I be looking at
to make '--enable-checking' and '--enable-debug' be the default when I do
'./regen && ./configure && make check' so I can submit a patch for master.

When we branch for release this should get turned off, but *only* after 
someone has complete QA and benchmark testing showing exactly what the
performance impact of the debugging is.
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

If a pre-stable -> master merge to trunk happens reliably every 3 
months, it might be an obnoxious merge, but it can't be any worse
than merging rxk5 (for gory details, see 
https://bitbucket.org/dahozer/tfs/changeset/10a38e703483fd99b3a41e99cba74f203524f731
)

The artificial version approach you mention also seems like it work
well if we want to keep a more centralized repository approach, and
treat Git like CVS.

But the great thing about Git (and to some extent Gerrit) is the fully
decentralized nature. While we're in the week leading up to a stable branch
point, everyone can just keep working in their own local repositories (or 
even push to Gerrit)

The only thing that stops is approving development changesets to master
on Gerrit for a week while someone does the pre-branch-and-mergeback.

Heck, if I can figure out how to get paid for 4 weeks a year to be the
release-and-merge nazi, it won't require anyone else to do any extra
work... And then you can ask me again after I've done it twice if I 
still think it's a good idea. 

On Mon, Sep 17, 2012 at 03:55:02PM -0700, Russ Allbery wrote:
> Simon Wilkinson  writes:
> 
> > We're not consistent about whether we release from trunk, or release
> > from a branch. This means that on some occasions the trunk has the tag,
> > and on others the branch. In a traditional git world, we would have
> > branched for 1.6.1, committed the changes necessary for 1.6.1 on that
> > branch, and then merged that branch back into the trunk. This final
> > merge step has the effect of making the 1.6.1 tag visible from both
> > branch and trunk, and so would cause both to git describe as
> > expected.
> 
> I'm very dubious about that merge back to trunk.  I'm not sure that
> development model really makes sense.  For better or worse, the trunk code
> and the stable branch code tends to diverge quickly and comprehensively,
> and we tend to apply separate fixes to trunk and to stable that are not
> equivalent.  Unless we make a reliable, regular habit of commiting -s ours
> merges in those cases, that merge from stable back to trunk can be a
> nightmare.
> 
> I also don't think it's necessary, in that I don't think that the 1.6.1
> tag needs to be exposed on the master branch.  What I do think is a
> serious problem is that it's not exposed on the stable branch, and there I
> don't really agree with the decision to create a separate branch off of
> stable to do 1.6 release stuff.  I sort of see how we got there, but I
> don't think it's wise.  (Of course, I'm not a gatekeeper now, so I can go
> on about how I would fix things without having to do any of the work)
> 
> I think it makes sense to have stable branches, but approaching a stable
> point release I think the only things that should go into that branch are
> things that are going into that release, and I would not make any more
> branches.  When the release goes out, it's with that stuff.  If one
> absolutely has to create a sub-branch for some reason (such as a purely
> platform-specific release), then *that* sub-branch I would merge back into
> the stable branch to make the tag visible there.
> 
> On master, I would do something different: tag master with some sort of
> artifical version at the point that the stable branch splits off.  So, for
> example, when we split off a stable branch for 1.6, we could have tagged
> master at that point with something else.  There are a couple of possible
> strategies for what "something else" is:
> 
> * devel/1.7.0 or something similar.  This would mean that all packages
>   built from master would be 1.7.0+something versions (or 1.7.1 if one
>   ever incremented it, but I suspect that we just wouldn't).  This would
>   mean that the Windows-only release would have been 1.8, and when we
>   split it off we would tag master as devel/1.9.0, and so forth.
>   Basically, reserve the odd numbers for the master branch and as soon as
>   one branches for release, increment (via tag) the versions on master to
>   the next odd number.
> 
> * devel/1.6.99.  This avoids the problem of reserving odd version numbers
>   for packages off of master, while creating a weird artificial version
>   number that might be somewhat confusing.  But the same semantics would
>   apply; we would have tagged it devel/1.7.99 when we split off the 1.7
>   branch and so forth.
> 
> -- 
> Russ Allbery (r...@stanford.edu) 
> ___
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Simon Wilkinson]

> A dearth of release managers, for example.  Among the other things that
> dearth is a bigger problem than is the absence of nightly builds.  :)

This is also the reason that 1.4.x has stagnated - it's been 16 months since 
1.4.14.1, and getting on for two years since 1.4.14. Andrew has been doing a 
diligent job of pulling things up to the 1.4 branch, we should at least try and 
get what's there wrapped up and shipped as a final 1.4 series release.

S.

___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Derrick Brashear]

the tests are not ready to be run on the buildslaves. it's been
working toward that point, but not yet.

On Tue, Sep 18, 2012 at 12:08 AM, Troy Benjegerdes  wrote:
> Nope, Debian x86-64
>
> Any chance the buildbots can be easily modified to run make check/make tests?
>
> I'm really curious what debian ppc32/ppc64 will do. I have an arm build, but
> no fuse kernel module (debian on an sdcard on an android tablet).
>
> On Mon, Sep 17, 2012 at 11:39:55PM -0400, Derrick Brashear wrote:
>> So. Were you perchance using it on a Mac? Probably a 64 bit Intel mac?
>>
>> http://gerrit.openafs.org/#change,8132
>>
>> As nearly as I can tell, this is a very specific problem. The code is fine. 
>> The
>> circumstances of building afsd.fuse meant it was collateral damage when we
>> started using roken, but only on MacOS, and probably only for non-32
>> bit pointers,
>> because MacOS does something odd with dirent.h
>>
>> On Mon, Sep 17, 2012 at 1:20 PM, Derrick Brashear  wrote:
>> > On Mon, Sep 17, 2012 at 1:15 PM, Troy Benjegerdes  wrote:
>> >> I'm looking to get all the low-hanging fruit with unskilled testing.
>> >> Particularly with regressions like this:
>> >>
>> >> hozer@six:~/src/openafs-fuse-git/tests/fuse$ 
>> >> /home/hozer/src/openafs-fuse-git/tests/fuse/../../src/afsd/afsd.fuse 
>> >> -dynroot -fakestat -d -confdir 
>> >> /home/hozer/src/openafs-fuse-git/tests/fuse/conf -cachedir 
>> >> /home/hozer/src/openafs-fuse-git/tests/fuse/vcache -mountdir 
>> >> /home/hozer/src/openafs-fuse-git/tests/fuse/mntdir
>> >> FUSE library version: 2.8.6
>> >> nullpath_ok: 0
>> >> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
>> >> INIT: 7.17
>> >> flags=0x047b
>> >> max_readahead=0x0002
>> >> Starting AFS cache scan...found 0 non-empty cache files (0%).
>> >> afsd: All AFS daemons started.
>> >> Segmentation fault
>> >>
>> >>
>> >> I am pretty sure this is related to the work Simon is doing on Libtool,
>> >> and there's a 90% probability it's a 30-second 'aha', followed by a two
>> >> line fix, and we're back to working again.
>> >>
>> >
>> > I'd bet not. However
>> >
>> >> The code is so complicated it will take me half a day to track down what
>> >> that two line fix is, or work in my own isolated fork and not get updates
>> >> as quickly. That unskilled smoke testing and/or automated runs gets a LOT
>> >> of mileage.
>> >
>> > Not really. Build with debugging and get a real backtrace. That said,
>> > since fuse is not *required*
>> > functionality in a build, yes, it's undertested. This is why we've
>> > generally avoided code which doesn't
>> > always build. Or, at least tried to.
>> >
>> >> It also gives people who want to learn about the codebase something simple
>> >> and meaningful they can do, instead of waiting around for someone else to
>> >> come up with a test plan.
>> >
>> >>
>> >> On Mon, Sep 17, 2012 at 11:25:36AM -0500, David Boyes wrote:
>> >>> > How about an effort to get nightly builds of master available on as 
>> >>> > many
>> >>> > platforms as possible, and getting thousands of bored college students 
>> >>> > to
>> >>> > download, install, and test them?
>> >>>
>> >>> I think that's still overly optimistic. There's a lot of moving parts 
>> >>> here; you just can't just install a package and have it do something 
>> >>> useful. You need to have a lot of surrounding infrastructure that 
>> >>> involves real control of a fair amount of stuff that random college 
>> >>> students won't have.  'make check' on a single machine will never give 
>> >>> you useful testing results other than to find packaging or "smoke test" 
>> >>> errors, which aren't all that helpful overall.
>> >>>
>> >>> > Wouldn't that massive crowsourced testing effort be worth the time of a
>> >>> > single developer to make sure *some* sort of package, even if it's 
>> >>> > half-
>> >>> > assed, gets distributed? I can't think of much of anything else that 
>> >>> > has a
>> >>> > bigger resource multiplation factor than a 'one click install', along 
>> >>> > with some
>> >>> > defaults to use a 'test.openafs.org' cell.
>> >>>
>> >>> As others have commented, unskilled testing performed without a detailed 
>> >>> test plan on software systems this complex is probably less helpful than 
>> >>> might otherwise appear. GIGO applies here. A uncoordinated test process 
>> >>> is unlikely to produce anything useful in that there have to be a 
>> >>> sequence of coordinated tests, replacing one component at a time in a 
>> >>> known order. I can't see how crowdsourcing would help here.
>> >>> ___
>> >>> OpenAFS-info mailing list
>> >>> OpenAFS-info@openafs.org
>> >>> https://lists.openafs.org/mailman/listinfo/openafs-info
>> >> ___
>> >> OpenAFS-info mailing list
>> >> OpenAFS-info@openafs.org
>> >> https://lists.openafs.org/mailman/listinfo/openafs-info
>> >>
>> >
>> >
>> >
>> > --
>> > Derrick
>>
>>
>>
>> --
>> Derrick
>> ___

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

Nope, Debian x86-64

Any chance the buildbots can be easily modified to run make check/make tests?

I'm really curious what debian ppc32/ppc64 will do. I have an arm build, but
no fuse kernel module (debian on an sdcard on an android tablet).

On Mon, Sep 17, 2012 at 11:39:55PM -0400, Derrick Brashear wrote:
> So. Were you perchance using it on a Mac? Probably a 64 bit Intel mac?
> 
> http://gerrit.openafs.org/#change,8132
> 
> As nearly as I can tell, this is a very specific problem. The code is fine. 
> The
> circumstances of building afsd.fuse meant it was collateral damage when we
> started using roken, but only on MacOS, and probably only for non-32
> bit pointers,
> because MacOS does something odd with dirent.h
> 
> On Mon, Sep 17, 2012 at 1:20 PM, Derrick Brashear  wrote:
> > On Mon, Sep 17, 2012 at 1:15 PM, Troy Benjegerdes  wrote:
> >> I'm looking to get all the low-hanging fruit with unskilled testing.
> >> Particularly with regressions like this:
> >>
> >> hozer@six:~/src/openafs-fuse-git/tests/fuse$ 
> >> /home/hozer/src/openafs-fuse-git/tests/fuse/../../src/afsd/afsd.fuse 
> >> -dynroot -fakestat -d -confdir 
> >> /home/hozer/src/openafs-fuse-git/tests/fuse/conf -cachedir 
> >> /home/hozer/src/openafs-fuse-git/tests/fuse/vcache -mountdir 
> >> /home/hozer/src/openafs-fuse-git/tests/fuse/mntdir
> >> FUSE library version: 2.8.6
> >> nullpath_ok: 0
> >> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
> >> INIT: 7.17
> >> flags=0x047b
> >> max_readahead=0x0002
> >> Starting AFS cache scan...found 0 non-empty cache files (0%).
> >> afsd: All AFS daemons started.
> >> Segmentation fault
> >>
> >>
> >> I am pretty sure this is related to the work Simon is doing on Libtool,
> >> and there's a 90% probability it's a 30-second 'aha', followed by a two
> >> line fix, and we're back to working again.
> >>
> >
> > I'd bet not. However
> >
> >> The code is so complicated it will take me half a day to track down what
> >> that two line fix is, or work in my own isolated fork and not get updates
> >> as quickly. That unskilled smoke testing and/or automated runs gets a LOT
> >> of mileage.
> >
> > Not really. Build with debugging and get a real backtrace. That said,
> > since fuse is not *required*
> > functionality in a build, yes, it's undertested. This is why we've
> > generally avoided code which doesn't
> > always build. Or, at least tried to.
> >
> >> It also gives people who want to learn about the codebase something simple
> >> and meaningful they can do, instead of waiting around for someone else to
> >> come up with a test plan.
> >
> >>
> >> On Mon, Sep 17, 2012 at 11:25:36AM -0500, David Boyes wrote:
> >>> > How about an effort to get nightly builds of master available on as many
> >>> > platforms as possible, and getting thousands of bored college students 
> >>> > to
> >>> > download, install, and test them?
> >>>
> >>> I think that's still overly optimistic. There's a lot of moving parts 
> >>> here; you just can't just install a package and have it do something 
> >>> useful. You need to have a lot of surrounding infrastructure that 
> >>> involves real control of a fair amount of stuff that random college 
> >>> students won't have.  'make check' on a single machine will never give 
> >>> you useful testing results other than to find packaging or "smoke test" 
> >>> errors, which aren't all that helpful overall.
> >>>
> >>> > Wouldn't that massive crowsourced testing effort be worth the time of a
> >>> > single developer to make sure *some* sort of package, even if it's half-
> >>> > assed, gets distributed? I can't think of much of anything else that 
> >>> > has a
> >>> > bigger resource multiplation factor than a 'one click install', along 
> >>> > with some
> >>> > defaults to use a 'test.openafs.org' cell.
> >>>
> >>> As others have commented, unskilled testing performed without a detailed 
> >>> test plan on software systems this complex is probably less helpful than 
> >>> might otherwise appear. GIGO applies here. A uncoordinated test process 
> >>> is unlikely to produce anything useful in that there have to be a 
> >>> sequence of coordinated tests, replacing one component at a time in a 
> >>> known order. I can't see how crowdsourcing would help here.
> >>> ___
> >>> OpenAFS-info mailing list
> >>> OpenAFS-info@openafs.org
> >>> https://lists.openafs.org/mailman/listinfo/openafs-info
> >> ___
> >> OpenAFS-info mailing list
> >> OpenAFS-info@openafs.org
> >> https://lists.openafs.org/mailman/listinfo/openafs-info
> >>
> >
> >
> >
> > --
> > Derrick
> 
> 
> 
> -- 
> Derrick
> ___
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openaf

Re: [OpenAFS] buildbot and packages

[Derrick Brashear]

So. Were you perchance using it on a Mac? Probably a 64 bit Intel mac?

http://gerrit.openafs.org/#change,8132

As nearly as I can tell, this is a very specific problem. The code is fine. The
circumstances of building afsd.fuse meant it was collateral damage when we
started using roken, but only on MacOS, and probably only for non-32
bit pointers,
because MacOS does something odd with dirent.h

On Mon, Sep 17, 2012 at 1:20 PM, Derrick Brashear  wrote:
> On Mon, Sep 17, 2012 at 1:15 PM, Troy Benjegerdes  wrote:
>> I'm looking to get all the low-hanging fruit with unskilled testing.
>> Particularly with regressions like this:
>>
>> hozer@six:~/src/openafs-fuse-git/tests/fuse$ 
>> /home/hozer/src/openafs-fuse-git/tests/fuse/../../src/afsd/afsd.fuse 
>> -dynroot -fakestat -d -confdir 
>> /home/hozer/src/openafs-fuse-git/tests/fuse/conf -cachedir 
>> /home/hozer/src/openafs-fuse-git/tests/fuse/vcache -mountdir 
>> /home/hozer/src/openafs-fuse-git/tests/fuse/mntdir
>> FUSE library version: 2.8.6
>> nullpath_ok: 0
>> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
>> INIT: 7.17
>> flags=0x047b
>> max_readahead=0x0002
>> Starting AFS cache scan...found 0 non-empty cache files (0%).
>> afsd: All AFS daemons started.
>> Segmentation fault
>>
>>
>> I am pretty sure this is related to the work Simon is doing on Libtool,
>> and there's a 90% probability it's a 30-second 'aha', followed by a two
>> line fix, and we're back to working again.
>>
>
> I'd bet not. However
>
>> The code is so complicated it will take me half a day to track down what
>> that two line fix is, or work in my own isolated fork and not get updates
>> as quickly. That unskilled smoke testing and/or automated runs gets a LOT
>> of mileage.
>
> Not really. Build with debugging and get a real backtrace. That said,
> since fuse is not *required*
> functionality in a build, yes, it's undertested. This is why we've
> generally avoided code which doesn't
> always build. Or, at least tried to.
>
>> It also gives people who want to learn about the codebase something simple
>> and meaningful they can do, instead of waiting around for someone else to
>> come up with a test plan.
>
>>
>> On Mon, Sep 17, 2012 at 11:25:36AM -0500, David Boyes wrote:
>>> > How about an effort to get nightly builds of master available on as many
>>> > platforms as possible, and getting thousands of bored college students to
>>> > download, install, and test them?
>>>
>>> I think that's still overly optimistic. There's a lot of moving parts here; 
>>> you just can't just install a package and have it do something useful. You 
>>> need to have a lot of surrounding infrastructure that involves real control 
>>> of a fair amount of stuff that random college students won't have.  'make 
>>> check' on a single machine will never give you useful testing results other 
>>> than to find packaging or "smoke test" errors, which aren't all that 
>>> helpful overall.
>>>
>>> > Wouldn't that massive crowsourced testing effort be worth the time of a
>>> > single developer to make sure *some* sort of package, even if it's half-
>>> > assed, gets distributed? I can't think of much of anything else that has a
>>> > bigger resource multiplation factor than a 'one click install', along 
>>> > with some
>>> > defaults to use a 'test.openafs.org' cell.
>>>
>>> As others have commented, unskilled testing performed without a detailed 
>>> test plan on software systems this complex is probably less helpful than 
>>> might otherwise appear. GIGO applies here. A uncoordinated test process is 
>>> unlikely to produce anything useful in that there have to be a sequence of 
>>> coordinated tests, replacing one component at a time in a known order. I 
>>> can't see how crowdsourcing would help here.
>>> ___
>>> OpenAFS-info mailing list
>>> OpenAFS-info@openafs.org
>>> https://lists.openafs.org/mailman/listinfo/openafs-info
>> ___
>> OpenAFS-info mailing list
>> OpenAFS-info@openafs.org
>> https://lists.openafs.org/mailman/listinfo/openafs-info
>>
>
>
>
> --
> Derrick



-- 
Derrick
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Russ Allbery]

Simon Wilkinson  writes:

> We're not consistent about whether we release from trunk, or release
> from a branch. This means that on some occasions the trunk has the tag,
> and on others the branch. In a traditional git world, we would have
> branched for 1.6.1, committed the changes necessary for 1.6.1 on that
> branch, and then merged that branch back into the trunk. This final
> merge step has the effect of making the 1.6.1 tag visible from both
> branch and trunk, and so would cause both to git describe as
> expected.

I'm very dubious about that merge back to trunk.  I'm not sure that
development model really makes sense.  For better or worse, the trunk code
and the stable branch code tends to diverge quickly and comprehensively,
and we tend to apply separate fixes to trunk and to stable that are not
equivalent.  Unless we make a reliable, regular habit of commiting -s ours
merges in those cases, that merge from stable back to trunk can be a
nightmare.

I also don't think it's necessary, in that I don't think that the 1.6.1
tag needs to be exposed on the master branch.  What I do think is a
serious problem is that it's not exposed on the stable branch, and there I
don't really agree with the decision to create a separate branch off of
stable to do 1.6 release stuff.  I sort of see how we got there, but I
don't think it's wise.  (Of course, I'm not a gatekeeper now, so I can go
on about how I would fix things without having to do any of the work)

I think it makes sense to have stable branches, but approaching a stable
point release I think the only things that should go into that branch are
things that are going into that release, and I would not make any more
branches.  When the release goes out, it's with that stuff.  If one
absolutely has to create a sub-branch for some reason (such as a purely
platform-specific release), then *that* sub-branch I would merge back into
the stable branch to make the tag visible there.

On master, I would do something different: tag master with some sort of
artifical version at the point that the stable branch splits off.  So, for
example, when we split off a stable branch for 1.6, we could have tagged
master at that point with something else.  There are a couple of possible
strategies for what "something else" is:

* devel/1.7.0 or something similar.  This would mean that all packages
  built from master would be 1.7.0+something versions (or 1.7.1 if one
  ever incremented it, but I suspect that we just wouldn't).  This would
  mean that the Windows-only release would have been 1.8, and when we
  split it off we would tag master as devel/1.9.0, and so forth.
  Basically, reserve the odd numbers for the master branch and as soon as
  one branches for release, increment (via tag) the versions on master to
  the next odd number.

* devel/1.6.99.  This avoids the problem of reserving odd version numbers
  for packages off of master, while creating a weird artificial version
  number that might be somewhat confusing.  But the same semantics would
  apply; we would have tagged it devel/1.7.99 when we split off the 1.7
  branch and so forth.

-- 
Russ Allbery (r...@stanford.edu) 
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Simon Wilkinson]

On 17 Sep 2012, at 23:18, Andrew Deason wrote:
> 
> I don't think you can make that say something based on 1.6.1, since the
> head of the 1.6.x branch right now is a different branch than 1.6.1. I
> mean, if git-version said something like "this is 1.6.1 plus N patches",
> that would be incorrect. Since, the head of 1.6.x is actually currently
> "1.6.1pre2 plus N patches" (specifically 163, apparently), which is what
> that says.

So, this is more of a problem with the way that we're using git, than a problem 
with tools that produce a version number.

We're not consistent about whether we release from trunk, or release from a 
branch. This means that on some occasions the trunk has the tag, and on others 
the branch. In a traditional git world, we would have branched for 1.6.1, 
committed the changes necessary for 1.6.1 on that branch, and then merged that 
branch back into the trunk. This final merge step has the effect of making the 
1.6.1 tag visible from both branch and trunk, and so would cause both to git 
describe as expected. (In the OpenAFS model, we commit first to master, then 
cherry pick to the 1.6 trunk, then cherry pick to the 1.6.1 branch). One way of 
fixing this immediate problem would be to commit to master, then cherry pick to 
the 1.6.1 version branch, then merge back into trunk.

We should probably talk about how to fix this before we do the 1.6.2 release. 
Oh, and master should probably get tagged with something sensible.

Cheers,

Simon.

___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

RE: [OpenAFS] buildbot and packages

[David Boyes]

> -Original Message-
> From: Simon Wilkinson [mailto:s...@your-file-system.com]

> On 17 Sep 2012, at 17:25, David Boyes wrote:
> >>  'make check' on a single machine will never give you useful testing
> results other than to find packaging or "smoke test" errors, which aren't all
> that helpful overall.
> 
> I agree with you with regards to crowd sourced testing, but I just wanted to
> call out this statement, as I think make check can actually be hugely helpful.
> [snip - bunch of good points about pre-build and commit-time sanity checking]

Good point.  I was considering more of the post-build QA portion of the 
problem, but addressing the issue of build failure at the input end of the 
problem makes sense as well. 

> Applying 'make check' as a commit requirement, and improving its coverage
> will hugely help with this problem. I see it more as a developer tool than a
> general QA solution.

Agreed. I think it'd be even more effective if the build sequence was less 
convoluted, but it's certainly better than nothing.

> What its not going to do is replace large scale test harnesses and detailed
> test plans. We still need the ability to generate large numbers of clients
> hammering a server to expose particular problems. 

Amen. I have a couple ideas on how that could be done, but all require a bit 
more baking before tossing them into public view. 



___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Russ Allbery]

"Matt W. Benjamin"  writes:

> Well, we used the fuse rather extensively for locking and dirformat
> testing.  It's experimental, but science experiment might be a little
> strong.

Ah, okay, it's gotten more attention than I thought.

-- 
Russ Allbery (r...@stanford.edu) 
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Russ Allbery]

Derrick Brashear  writes:
> On Mon, Sep 17, 2012 at 2:28 PM, Russ Allbery  wrote:

>> The fuse code currently in the tree was primarily a science experiment
>> by one developer and is not something that's really ready for
>> production use.  That's not to say this isn't a regression, and of
>> course it would be nice to fix, but I'm completely unsurprised that it
>> has issues.  So far as I know, no one is currently actively using the
>> fuse code.

> I don't think maintaining and improving it would be at all a bad thing
> as it's certainly valuable to have, tho.

Oh, yeah, absolutely.  I'm all in favor!  It's just not something I would
currently expect to be very robust.

-- 
Russ Allbery (r...@stanford.edu) 
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Simon Wilkinson]

On 17 Sep 2012, at 17:25, David Boyes wrote:
>>  'make check' on a single machine will never give you useful testing results 
>> other than to find packaging or "smoke test" errors, which aren't all that 
>> helpful overall. 

I agree with you with regards to crowd sourced testing, but I just wanted to 
call out this statement, as I think make check can actually be hugely helpful.

In the past, we used to have the problem that changes would get committed to 
master that would cause the build to fail. So, if you were working on new 
stuff, and had just pulled in a recent upstream to work on, you often ended up 
having to spend lots of time just getting the tree to work again on your chosen 
platform. Buildbot has pretty much eradicated this problem - I can be sure that 
git fetch will give me a tree that works.

So now, the new problem is that changes in master may well be unstable. Instead 
of working on building your shiny new feature, developers end up having to 
track down why a particular binary segmentation faults and so on. Applying 
'make check' as a commit requirement, and improving its coverage will hugely 
help with this problem. I see it more as a developer tool than a general QA 
solution.

What its not going to do is replace large scale test harnesses and detailed 
test plans. We still need the ability to generate large numbers of clients 
hammering a server to expose particular problems. Over the 1.6 cycle, the 
majority of this work was done for OpenAFS by volunteers at European HPC sites, 
who used down time on their clusters to generate particular types of load 
against our fileservers. This led to a number of long standing bugs in both the 
demand attach fileserver, and the RX stack itself, being identified and fixed.

Cheers,

Simon.

___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Simon Wilkinson]

On 17 Sep 2012, at 19:54, Troy Benjegerdes wrote:
> If 'rebuild with debug' symbols is the answer to find the segfault, then why
> don't we change './regen && ./configure && make check' to turn on debug 
> symbols
> by default (at least in master.. we can turn it back off in a release)

If you are developing, then you should be running configure with at least 
--enable-checking and --enable-debug

S.


___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Derrick Brashear]

On Mon, Sep 17, 2012 at 2:54 PM, Troy Benjegerdes  wrote:
>> >> FUSE library version: 2.8.6
>> >> nullpath_ok: 0
>> >> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
>> >> INIT: 7.17
>> >> flags=0x047b
>> >> max_readahead=0x0002
>> >> Starting AFS cache scan...found 0 non-empty cache files (0%).
>> >> afsd: All AFS daemons started.
>> >> Segmentation fault
>> >
>> > The fuse code currently in the tree was primarily a science experiment by
>> > one developer and is not something that's really ready for production use.
>> > That's not to say this isn't a regression, and of course it would be nice
>> > to fix, but I'm completely unsurprised that it has issues.  So far as I
>> > know, no one is currently actively using the fuse code.
>>
>> I don't think maintaining and improving it would be at all a bad thing
>> as it's certainly
>> valuable to have, tho.
>
> afsd-fuse is an awfully convenient smoke test...
>
> https://bitbucket.org/dahozer/tfs/changeset/c29b1275d8472cf85bf17873220390c01d05f023
>
> Something is different between 'tfs' bitbucket checkout on my laptop and the 
> git
> checkout, and I'm not sure what.
>
> If 'rebuild with debug' symbols is the answer to find the segfault, then why
> don't we change './regen && ./configure && make check' to turn on debug 
> symbols
> by default (at least in master.. we can turn it back off in a release)

"./regen && ./configure --enable-debug && make check"

done.

-- 
Derrick
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

> >> FUSE library version: 2.8.6
> >> nullpath_ok: 0
> >> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
> >> INIT: 7.17
> >> flags=0x047b
> >> max_readahead=0x0002
> >> Starting AFS cache scan...found 0 non-empty cache files (0%).
> >> afsd: All AFS daemons started.
> >> Segmentation fault
> >
> > The fuse code currently in the tree was primarily a science experiment by
> > one developer and is not something that's really ready for production use.
> > That's not to say this isn't a regression, and of course it would be nice
> > to fix, but I'm completely unsurprised that it has issues.  So far as I
> > know, no one is currently actively using the fuse code.
> 
> I don't think maintaining and improving it would be at all a bad thing
> as it's certainly
> valuable to have, tho.

afsd-fuse is an awfully convenient smoke test...

https://bitbucket.org/dahozer/tfs/changeset/c29b1275d8472cf85bf17873220390c01d05f023

Something is different between 'tfs' bitbucket checkout on my laptop and the git
checkout, and I'm not sure what.

If 'rebuild with debug' symbols is the answer to find the segfault, then why
don't we change './regen && ./configure && make check' to turn on debug symbols
by default (at least in master.. we can turn it back off in a release)
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Matt W. Benjamin]

Hi,

Well, we used the fuse rather extensively for locking and dirformat testing.  
It's experimental, but science experiment might be a little strong.

Matt

- "Russ Allbery"  wrote:

> Troy Benjegerdes  writes:
> 
> > I'm looking to get all the low-hanging fruit with unskilled
> testing.
> > Particularly with regressions like this:
> 
> > hozer@six:~/src/openafs-fuse-git/tests/fuse$
> /home/hozer/src/openafs-fuse-git/tests/fuse/../../src/afsd/afsd.fuse
> -dynroot -fakestat -d -confdir
> /home/hozer/src/openafs-fuse-git/tests/fuse/conf -cachedir
> /home/hozer/src/openafs-fuse-git/tests/fuse/vcache -mountdir
> /home/hozer/src/openafs-fuse-git/tests/fuse/mntdir
> > FUSE library version: 2.8.6
> > nullpath_ok: 0
> > unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
> > INIT: 7.17
> > flags=0x047b
> > max_readahead=0x0002
> > Starting AFS cache scan...found 0 non-empty cache files (0%).
> > afsd: All AFS daemons started.
> > Segmentation fault
> 
> The fuse code currently in the tree was primarily a science experiment
> by
> one developer and is not something that's really ready for production
> use.
> That's not to say this isn't a regression, and of course it would be
> nice
> to fix, but I'm completely unsurprised that it has issues.  So far as
> I
> know, no one is currently actively using the fuse code.
> 
> -- 
> Russ Allbery (r...@stanford.edu)
> 
> ___
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Derrick Brashear]

On Mon, Sep 17, 2012 at 2:28 PM, Russ Allbery  wrote:
> Troy Benjegerdes  writes:
>
>> I'm looking to get all the low-hanging fruit with unskilled testing.
>> Particularly with regressions like this:
>
>> hozer@six:~/src/openafs-fuse-git/tests/fuse$ 
>> /home/hozer/src/openafs-fuse-git/tests/fuse/../../src/afsd/afsd.fuse 
>> -dynroot -fakestat -d -confdir 
>> /home/hozer/src/openafs-fuse-git/tests/fuse/conf -cachedir 
>> /home/hozer/src/openafs-fuse-git/tests/fuse/vcache -mountdir 
>> /home/hozer/src/openafs-fuse-git/tests/fuse/mntdir
>> FUSE library version: 2.8.6
>> nullpath_ok: 0
>> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
>> INIT: 7.17
>> flags=0x047b
>> max_readahead=0x0002
>> Starting AFS cache scan...found 0 non-empty cache files (0%).
>> afsd: All AFS daemons started.
>> Segmentation fault
>
> The fuse code currently in the tree was primarily a science experiment by
> one developer and is not something that's really ready for production use.
> That's not to say this isn't a regression, and of course it would be nice
> to fix, but I'm completely unsurprised that it has issues.  So far as I
> know, no one is currently actively using the fuse code.

I don't think maintaining and improving it would be at all a bad thing
as it's certainly
valuable to have, tho.


-- 
Derrick
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Russ Allbery]

Troy Benjegerdes  writes:

> I'm looking to get all the low-hanging fruit with unskilled testing.
> Particularly with regressions like this:

> hozer@six:~/src/openafs-fuse-git/tests/fuse$ 
> /home/hozer/src/openafs-fuse-git/tests/fuse/../../src/afsd/afsd.fuse -dynroot 
> -fakestat -d -confdir /home/hozer/src/openafs-fuse-git/tests/fuse/conf 
> -cachedir /home/hozer/src/openafs-fuse-git/tests/fuse/vcache -mountdir 
> /home/hozer/src/openafs-fuse-git/tests/fuse/mntdir
> FUSE library version: 2.8.6
> nullpath_ok: 0
> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
> INIT: 7.17
> flags=0x047b
> max_readahead=0x0002
> Starting AFS cache scan...found 0 non-empty cache files (0%).
> afsd: All AFS daemons started.
> Segmentation fault

The fuse code currently in the tree was primarily a science experiment by
one developer and is not something that's really ready for production use.
That's not to say this isn't a regression, and of course it would be nice
to fix, but I'm completely unsurprised that it has issues.  So far as I
know, no one is currently actively using the fuse code.

-- 
Russ Allbery (r...@stanford.edu) 
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Derrick Brashear]

On Mon, Sep 17, 2012 at 1:15 PM, Troy Benjegerdes  wrote:
> I'm looking to get all the low-hanging fruit with unskilled testing.
> Particularly with regressions like this:
>
> hozer@six:~/src/openafs-fuse-git/tests/fuse$ 
> /home/hozer/src/openafs-fuse-git/tests/fuse/../../src/afsd/afsd.fuse -dynroot 
> -fakestat -d -confdir /home/hozer/src/openafs-fuse-git/tests/fuse/conf 
> -cachedir /home/hozer/src/openafs-fuse-git/tests/fuse/vcache -mountdir 
> /home/hozer/src/openafs-fuse-git/tests/fuse/mntdir
> FUSE library version: 2.8.6
> nullpath_ok: 0
> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
> INIT: 7.17
> flags=0x047b
> max_readahead=0x0002
> Starting AFS cache scan...found 0 non-empty cache files (0%).
> afsd: All AFS daemons started.
> Segmentation fault
>
>
> I am pretty sure this is related to the work Simon is doing on Libtool,
> and there's a 90% probability it's a 30-second 'aha', followed by a two
> line fix, and we're back to working again.
>

I'd bet not. However

> The code is so complicated it will take me half a day to track down what
> that two line fix is, or work in my own isolated fork and not get updates
> as quickly. That unskilled smoke testing and/or automated runs gets a LOT
> of mileage.

Not really. Build with debugging and get a real backtrace. That said,
since fuse is not *required*
functionality in a build, yes, it's undertested. This is why we've
generally avoided code which doesn't
always build. Or, at least tried to.

> It also gives people who want to learn about the codebase something simple
> and meaningful they can do, instead of waiting around for someone else to
> come up with a test plan.

>
> On Mon, Sep 17, 2012 at 11:25:36AM -0500, David Boyes wrote:
>> > How about an effort to get nightly builds of master available on as many
>> > platforms as possible, and getting thousands of bored college students to
>> > download, install, and test them?
>>
>> I think that's still overly optimistic. There's a lot of moving parts here; 
>> you just can't just install a package and have it do something useful. You 
>> need to have a lot of surrounding infrastructure that involves real control 
>> of a fair amount of stuff that random college students won't have.  'make 
>> check' on a single machine will never give you useful testing results other 
>> than to find packaging or "smoke test" errors, which aren't all that helpful 
>> overall.
>>
>> > Wouldn't that massive crowsourced testing effort be worth the time of a
>> > single developer to make sure *some* sort of package, even if it's half-
>> > assed, gets distributed? I can't think of much of anything else that has a
>> > bigger resource multiplation factor than a 'one click install', along with 
>> > some
>> > defaults to use a 'test.openafs.org' cell.
>>
>> As others have commented, unskilled testing performed without a detailed 
>> test plan on software systems this complex is probably less helpful than 
>> might otherwise appear. GIGO applies here. A uncoordinated test process is 
>> unlikely to produce anything useful in that there have to be a sequence of 
>> coordinated tests, replacing one component at a time in a known order. I 
>> can't see how crowdsourcing would help here.
>> ___
>> OpenAFS-info mailing list
>> OpenAFS-info@openafs.org
>> https://lists.openafs.org/mailman/listinfo/openafs-info
> ___
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>



-- 
Derrick
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

I'm looking to get all the low-hanging fruit with unskilled testing.
Particularly with regressions like this:

hozer@six:~/src/openafs-fuse-git/tests/fuse$ 
/home/hozer/src/openafs-fuse-git/tests/fuse/../../src/afsd/afsd.fuse -dynroot 
-fakestat -d -confdir /home/hozer/src/openafs-fuse-git/tests/fuse/conf 
-cachedir /home/hozer/src/openafs-fuse-git/tests/fuse/vcache -mountdir 
/home/hozer/src/openafs-fuse-git/tests/fuse/mntdir
FUSE library version: 2.8.6
nullpath_ok: 0
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.17
flags=0x047b
max_readahead=0x0002
Starting AFS cache scan...found 0 non-empty cache files (0%).
afsd: All AFS daemons started.
Segmentation fault


I am pretty sure this is related to the work Simon is doing on Libtool,
and there's a 90% probability it's a 30-second 'aha', followed by a two
line fix, and we're back to working again.


The code is so complicated it will take me half a day to track down what
that two line fix is, or work in my own isolated fork and not get updates
as quickly. That unskilled smoke testing and/or automated runs gets a LOT
of mileage.

It also gives people who want to learn about the codebase something simple
and meaningful they can do, instead of waiting around for someone else to
come up with a test plan.


On Mon, Sep 17, 2012 at 11:25:36AM -0500, David Boyes wrote:
> > How about an effort to get nightly builds of master available on as many
> > platforms as possible, and getting thousands of bored college students to
> > download, install, and test them?
> 
> I think that's still overly optimistic. There's a lot of moving parts here; 
> you just can't just install a package and have it do something useful. You 
> need to have a lot of surrounding infrastructure that involves real control 
> of a fair amount of stuff that random college students won't have.  'make 
> check' on a single machine will never give you useful testing results other 
> than to find packaging or "smoke test" errors, which aren't all that helpful 
> overall. 
> 
> > Wouldn't that massive crowsourced testing effort be worth the time of a
> > single developer to make sure *some* sort of package, even if it's half-
> > assed, gets distributed? I can't think of much of anything else that has a
> > bigger resource multiplation factor than a 'one click install', along with 
> > some
> > defaults to use a 'test.openafs.org' cell.
> 
> As others have commented, unskilled testing performed without a detailed test 
> plan on software systems this complex is probably less helpful than might 
> otherwise appear. GIGO applies here. A uncoordinated test process is unlikely 
> to produce anything useful in that there have to be a sequence of coordinated 
> tests, replacing one component at a time in a known order. I can't see how 
> crowdsourcing would help here. 
> ___
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

RE: [OpenAFS] buildbot and packages

[David Boyes]

> How about an effort to get nightly builds of master available on as many
> platforms as possible, and getting thousands of bored college students to
> download, install, and test them?

I think that's still overly optimistic. There's a lot of moving parts here; you 
just can't just install a package and have it do something useful. You need to 
have a lot of surrounding infrastructure that involves real control of a fair 
amount of stuff that random college students won't have.  'make check' on a 
single machine will never give you useful testing results other than to find 
packaging or "smoke test" errors, which aren't all that helpful overall. 

> Wouldn't that massive crowsourced testing effort be worth the time of a
> single developer to make sure *some* sort of package, even if it's half-
> assed, gets distributed? I can't think of much of anything else that has a
> bigger resource multiplation factor than a 'one click install', along with 
> some
> defaults to use a 'test.openafs.org' cell.

As others have commented, unskilled testing performed without a detailed test 
plan on software systems this complex is probably less helpful than might 
otherwise appear. GIGO applies here. A uncoordinated test process is unlikely 
to produce anything useful in that there have to be a sequence of coordinated 
tests, replacing one component at a time in a known order. I can't see how 
crowdsourcing would help here. 
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Ken Dreyer]

Thanks everyone who replied. There was a lot of email on this thread,
so I'm going to write a quick summary.

There are concerns that this would increase the support load on
organizational helpdesks that support AFS internally, and on the
OpenAFS community overall.

There are concerns that generating the packages (at least on Windows)
will dramatically increase the build time on the buildbot slaves.

Windows binaries go through a manual signing and registration process,
so the additional developer dependencies there may make it infeasible
to do fully automated packages for Windows at all right now.

It's important to identify these snapshot builds as "not official".
This must be clear to the user before they install it, and clear to
the support staff after the software has been installed. The web
interface for downloads must have big warning signs, and from a
support perspective, we should use unique version strings accessible
by "rxdebug -v" or "fs version" etc.

There are concerns about how to do version numbers for these
snapshots. I did some research and it looks like we were setting
version numbers of "9.9.9" back when we had nightly CVS snapshots.
Those were just source tarballs though, so they weren't as accessible
to the casual user as what we're talking about. And this wouldn't
really work well for nightly builds since the date value wouldn't be
encoded in the binaries.

Ben pointed out that whatever version numbering scheme we use should
be cross-platform so it can align with package managers where
possible.

Simon pointed out it would be nice to extend the build testing to
packaging so that package maintainers could still catch the necessary
packaging changes (eg. when new files appear). This would probably
need to be balanced against the other factors.

-

I think a good next step would be to decide how we should handle
version numbers in snapshot packages. For example: should configure.ac
in the 1_6_x branch contain something like "1.6.2" or "1.6.2git" right
now, and a build script could tack on a date or git hash to that
number every night? I'm open to writing patches to the
"build-tools/git-version" script, or otherwise.

- Ken
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Doug Hirsch]

Troy,

I like where this is going.  Here are the immediate hurdles.  I am
indeed just another ISP customer behind NAT layers.  I've been
interested in setting up dynamic DNS, but have not yet done it.  My
travel schedule for the week changed today, which will prevent me from
working on the debian machine until next week.  The time tracking
tools search is a good idea for this week from the Macbook, thanks.

Doug
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

Here's my thought:

Spend your 5 start-up hours either re-installing or upgrading your Debian
system to squeeze (the current debian stable). Try doing 'apt-get install
openafs-fileserver', and then, if it works, please edit
https://bitbucket.org/dahozer/tfs/wiki/Home saying so, or if it does not,
create an issue: https://bitbucket.org/dahozer/tfs/issues/new

Also, spend a few minutes looking at time tracking tools (maybe one from
http://lifehacker.com/5362829/five-best-time+tracking-applications or try
http://rescuetime.com ), and let's see what you actually spend. 

If you can spend 2-4 hours a week installing a weekly build on both a debian
linux server, and a client on a MacOS X laptop, and then occasionally try
accessing your server and using disconnected operation, my personal opinion is
that's a huge benefit.

(Yes, I'm hand-waving over some things like needing multiple DB servers and
kerberos right now.. we'll get to that next week, or I'll just create a
principal for you on my HOZED.ORG realm)


I'd also like to ask the openafs-info list if it would be appropriate
to create a 'openafs-test' list so we can have a more focused discussion.


Some more technical in the weeds stuff:

Can you leave your debian box powered on, with a real ip, or is it behind a
NAT? My server has a real IP that might change, so I have some VPN tunnels to a
'cloud' virtual private server with a static IP.  I can probably explain how to
set up OpenVPN, but I got irritated enough setting that up that I'd rather
spend time setting up IPv6 tunnels and hacking on v6 support for AFS than
dealing with certificate creation again.


I think the biggest reason I'm leading a crusade for IPv6 support is that I
have a use case a lot like Doug, and if I can get v6 support with fully krb5
authenticated/encrypted transports, then I can forget about ever having to 
utter the word 'vpn' or create another damn openssl cert ever again. If,
for some reason, I *do* have to mess around with ssl, I'll have the directions
and documentation and all my certs in my globally accessible (and protected)
AFS directory, so I can find it.





On Sat, Sep 15, 2012 at 11:21:33PM -0700, Doug Hirsch wrote:
> Troy,
> 
> I'm unclear I've offered you anything you can actually use.  Mostly,
> I'm offering you the reality check of a non-programmer, a Macbook with
> me on the road and a stale Debian box powered down back at home.
> You'll have to steer me through downloading, installing and using
> anything that's not on a stock Mac running Mac OS 10.6.8, or bringing
> the Linux box up to whatever environment you want once I get home.
> Most of my other machines run Windows, although I have a couple of G4
> Mac mini's hanging around for fun.  Your average college student will
> not have much more to offer you, so I'm offering a chance for you to
> define what you could actually accomplish harnessing thousands of us
> "amicable zombies" with limited time, experience and resources.  If it
> will help, I'm willing to install some virtualization package on the
> Macbook, but will need guidance.  I also need to keep a lid on my time
> commitment, so assume no more than 5 hours a week from me, with an
> extra 5 hours this week to start up.  If you can make use of that, let
> me know and I'll wander over to bitbucket.org.  What I want is someone
> to talk me through getting OpenAFS going in my personal environment.
> I'm unclear how much value you'll get out of me on just five hours a
> week and 1.5 machines.  I've written proposals and defended engineers
> building test environments, among other things, but I haven't gotten
> my hands into code for many years, so I'm sure you'll be surprised by
> what cultural assumptions you discover I don't know.  I see and
> appreciate your energy and optimism, while I think you're
> underestimating what you're asking.  But if you can make something
> work with limited commitments from others, I'm happy to go along to
> see what we can contribute to the community together.
> 
> Doug
> 
> On Sat, Sep 15, 2012 at 10:34 PM, Troy Benjegerdes  wrote:
> > I'll buy that for a few emails.
> >
> > Let's start by having you take a look at:
> >
> > https://bitbucket.org/dahozer/tfs
> >
> > There are tabs for issues & wikis, so sign up for a bitbucket account and
> > ask some questions there, so we don't spam the -devel list with lots of
> > 'how do I xyz' questions
> >
> > For the openafs-devel list, please let the list know what resources/
> > platforms you have for testing, and I'd like to hear from the list what
> > could I write some tests for that could utilize those resources.
> >
> >
> > On Sat, Sep 15, 2012 at 09:44:07PM -0700, Doug Hirsch wrote:
> >> Troy,
> >>
> >> If you set this up, I'm willing to be your guinea pig.  It'll cost you
> >> enough support and/or documentation to get me over initial learning
> >> curve.
> >>
> >> Doug
> >>
> >> On 9/15/12, Troy Benjegerdes  wrote:
> >> > Sometimes I think we get hung up on 'good testing' 

Re: [OpenAFS] buildbot and packages

[Doug Hirsch]

Troy,

I'm unclear I've offered you anything you can actually use.  Mostly,
I'm offering you the reality check of a non-programmer, a Macbook with
me on the road and a stale Debian box powered down back at home.
You'll have to steer me through downloading, installing and using
anything that's not on a stock Mac running Mac OS 10.6.8, or bringing
the Linux box up to whatever environment you want once I get home.
Most of my other machines run Windows, although I have a couple of G4
Mac mini's hanging around for fun.  Your average college student will
not have much more to offer you, so I'm offering a chance for you to
define what you could actually accomplish harnessing thousands of us
"amicable zombies" with limited time, experience and resources.  If it
will help, I'm willing to install some virtualization package on the
Macbook, but will need guidance.  I also need to keep a lid on my time
commitment, so assume no more than 5 hours a week from me, with an
extra 5 hours this week to start up.  If you can make use of that, let
me know and I'll wander over to bitbucket.org.  What I want is someone
to talk me through getting OpenAFS going in my personal environment.
I'm unclear how much value you'll get out of me on just five hours a
week and 1.5 machines.  I've written proposals and defended engineers
building test environments, among other things, but I haven't gotten
my hands into code for many years, so I'm sure you'll be surprised by
what cultural assumptions you discover I don't know.  I see and
appreciate your energy and optimism, while I think you're
underestimating what you're asking.  But if you can make something
work with limited commitments from others, I'm happy to go along to
see what we can contribute to the community together.

Doug

On Sat, Sep 15, 2012 at 10:34 PM, Troy Benjegerdes  wrote:
> I'll buy that for a few emails.
>
> Let's start by having you take a look at:
>
> https://bitbucket.org/dahozer/tfs
>
> There are tabs for issues & wikis, so sign up for a bitbucket account and
> ask some questions there, so we don't spam the -devel list with lots of
> 'how do I xyz' questions
>
> For the openafs-devel list, please let the list know what resources/
> platforms you have for testing, and I'd like to hear from the list what
> could I write some tests for that could utilize those resources.
>
>
> On Sat, Sep 15, 2012 at 09:44:07PM -0700, Doug Hirsch wrote:
>> Troy,
>>
>> If you set this up, I'm willing to be your guinea pig.  It'll cost you
>> enough support and/or documentation to get me over initial learning
>> curve.
>>
>> Doug
>>
>> On 9/15/12, Troy Benjegerdes  wrote:
>> > Sometimes I think we get hung up on 'good testing' vs having *something*.
>> >
>> > The last time I worked for someone else, it was writing test code for
>> > Cray's
>> > supercomputer systems. You don't get much more complex than a machine
>> > with 30,000 cores in which 'acceptable' performance is defined as 'pushing
>> > the system to the point right before it collapses into an unusable heap',
>> > and it's got to run a workload of hundreds of thousands of the world's most
>> > complex and numerically sensitive computational codes.
>> >
>> > And I'd hazard a guess that 3/4 of the system problems were with the
>> > filesystem
>> > (Lustre most often). I've also heard a pretty good argument that the reason
>> >
>> > Cray went bankrupt a couple of times is they over-tested. If you did get a
>> > machine back in the YMP days, it was very well tested, but the price showed
>> >
>> > it, and clusters ate their market.
>> >
>> >
>> > Maybe we don't have money.. But how many users of AFS are there. I'm not
>> > talking
>> > companies, I'm talking people.. specifically, bored college students. How
>> > many
>> > people have used AFS at a major university, and might help us out doing
>> > manual
>> > testing if we give them a framework?
>> >
>> > To paraphrase the .. well.. chief cat herder .. of the most widely deployed
>> > operating system ever (Linux),
>> > "With enough QA testers, all bugs are shallow"
>> >
>> > On Fri, Sep 14, 2012 at 04:42:37PM -0500, David Boyes wrote:
>> >> > In this case I think you are low-balling the estimate.  To do it right
>> >> > it isn't
>> >> > sufficient to test one build against itself.  You need to test new
>> >> > clients
>> >> > against a range of old servers and vice versa in a constrained
>> >> > environment.
>> >> > It is necessary to be able to identify when a change has an adverse
>> >> > performance impact as well as accuracy.  There is a need to be able to
>> >> > introduce intentional errors at various points in the protocol.  Just
>> >> > the
>> >> > hardware costs are mid 5 digits and the software development is
>> >> > significantly more than that.
>> >>
>> >>  I agree --  if you were starting from scratch, you're probably right.
>> >>
>> >> But, a) I wasn't starting from scratch, so the additional equipment for
>> >> adding the AFS framework stuff was about what I quoted, a

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

I'll buy that for a few emails.

Let's start by having you take a look at:

https://bitbucket.org/dahozer/tfs

There are tabs for issues & wikis, so sign up for a bitbucket account and
ask some questions there, so we don't spam the -devel list with lots of 
'how do I xyz' questions

For the openafs-devel list, please let the list know what resources/
platforms you have for testing, and I'd like to hear from the list what
could I write some tests for that could utilize those resources.


On Sat, Sep 15, 2012 at 09:44:07PM -0700, Doug Hirsch wrote:
> Troy,
> 
> If you set this up, I'm willing to be your guinea pig.  It'll cost you
> enough support and/or documentation to get me over initial learning
> curve.
> 
> Doug
> 
> On 9/15/12, Troy Benjegerdes  wrote:
> > Sometimes I think we get hung up on 'good testing' vs having *something*.
> >
> > The last time I worked for someone else, it was writing test code for
> > Cray's
> > supercomputer systems. You don't get much more complex than a machine
> > with 30,000 cores in which 'acceptable' performance is defined as 'pushing
> > the system to the point right before it collapses into an unusable heap',
> > and it's got to run a workload of hundreds of thousands of the world's most
> > complex and numerically sensitive computational codes.
> >
> > And I'd hazard a guess that 3/4 of the system problems were with the
> > filesystem
> > (Lustre most often). I've also heard a pretty good argument that the reason
> >
> > Cray went bankrupt a couple of times is they over-tested. If you did get a
> > machine back in the YMP days, it was very well tested, but the price showed
> >
> > it, and clusters ate their market.
> >
> >
> > Maybe we don't have money.. But how many users of AFS are there. I'm not
> > talking
> > companies, I'm talking people.. specifically, bored college students. How
> > many
> > people have used AFS at a major university, and might help us out doing
> > manual
> > testing if we give them a framework?
> >
> > To paraphrase the .. well.. chief cat herder .. of the most widely deployed
> > operating system ever (Linux),
> > "With enough QA testers, all bugs are shallow"
> >
> > On Fri, Sep 14, 2012 at 04:42:37PM -0500, David Boyes wrote:
> >> > In this case I think you are low-balling the estimate.  To do it right
> >> > it isn't
> >> > sufficient to test one build against itself.  You need to test new
> >> > clients
> >> > against a range of old servers and vice versa in a constrained
> >> > environment.
> >> > It is necessary to be able to identify when a change has an adverse
> >> > performance impact as well as accuracy.  There is a need to be able to
> >> > introduce intentional errors at various points in the protocol.  Just
> >> > the
> >> > hardware costs are mid 5 digits and the software development is
> >> > significantly more than that.
> >>
> >>  I agree --  if you were starting from scratch, you're probably right.
> >>
> >> But, a) I wasn't starting from scratch, so the additional equipment for
> >> adding the AFS framework stuff was about what I quoted, and b) I was
> >> discussing our tooling and test setup, not the general case.
> >> We reused existing tooling in a number of places, and layered the AFS
> >> component onto that. We do this kind of thing for other software, so we
> >> had a decent baseline to start from.
> >>
> >> Solid QA infrastructure -- especially for complex systems -- isn't simple
> >> or cheap; there we agree wholeheartedly.
> >>
> >>
> >>
> >> :??
> > ___
> > OpenAFS-info mailing list
> > OpenAFS-info@openafs.org
> > https://lists.openafs.org/mailman/listinfo/openafs-info
> >
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Doug Hirsch]

Troy,

If you set this up, I'm willing to be your guinea pig.  It'll cost you
enough support and/or documentation to get me over initial learning
curve.

Doug

On 9/15/12, Troy Benjegerdes  wrote:
> Sometimes I think we get hung up on 'good testing' vs having *something*.
>
> The last time I worked for someone else, it was writing test code for
> Cray's
> supercomputer systems. You don't get much more complex than a machine
> with 30,000 cores in which 'acceptable' performance is defined as 'pushing
> the system to the point right before it collapses into an unusable heap',
> and it's got to run a workload of hundreds of thousands of the world's most
> complex and numerically sensitive computational codes.
>
> And I'd hazard a guess that 3/4 of the system problems were with the
> filesystem
> (Lustre most often). I've also heard a pretty good argument that the reason
>
> Cray went bankrupt a couple of times is they over-tested. If you did get a
> machine back in the YMP days, it was very well tested, but the price showed
>
> it, and clusters ate their market.
>
>
> Maybe we don't have money.. But how many users of AFS are there. I'm not
> talking
> companies, I'm talking people.. specifically, bored college students. How
> many
> people have used AFS at a major university, and might help us out doing
> manual
> testing if we give them a framework?
>
> To paraphrase the .. well.. chief cat herder .. of the most widely deployed
> operating system ever (Linux),
> "With enough QA testers, all bugs are shallow"
>
> On Fri, Sep 14, 2012 at 04:42:37PM -0500, David Boyes wrote:
>> > In this case I think you are low-balling the estimate.  To do it right
>> > it isn't
>> > sufficient to test one build against itself.  You need to test new
>> > clients
>> > against a range of old servers and vice versa in a constrained
>> > environment.
>> > It is necessary to be able to identify when a change has an adverse
>> > performance impact as well as accuracy.  There is a need to be able to
>> > introduce intentional errors at various points in the protocol.  Just
>> > the
>> > hardware costs are mid 5 digits and the software development is
>> > significantly more than that.
>>
>>  I agree --  if you were starting from scratch, you're probably right.
>>
>> But, a) I wasn't starting from scratch, so the additional equipment for
>> adding the AFS framework stuff was about what I quoted, and b) I was
>> discussing our tooling and test setup, not the general case.
>> We reused existing tooling in a number of places, and layered the AFS
>> component onto that. We do this kind of thing for other software, so we
>> had a decent baseline to start from.
>>
>> Solid QA infrastructure -- especially for complex systems -- isn't simple
>> or cheap; there we agree wholeheartedly.
>>
>>
>>
>> :??
> ___
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

Sometimes I think we get hung up on 'good testing' vs having *something*.

The last time I worked for someone else, it was writing test code for Cray's
supercomputer systems. You don't get much more complex than a machine
with 30,000 cores in which 'acceptable' performance is defined as 'pushing
the system to the point right before it collapses into an unusable heap',
and it's got to run a workload of hundreds of thousands of the world's most
complex and numerically sensitive computational codes.

And I'd hazard a guess that 3/4 of the system problems were with the filesystem
(Lustre most often). I've also heard a pretty good argument that the reason 
Cray went bankrupt a couple of times is they over-tested. If you did get a
machine back in the YMP days, it was very well tested, but the price showed 
it, and clusters ate their market.


Maybe we don't have money.. But how many users of AFS are there. I'm not talking
companies, I'm talking people.. specifically, bored college students. How many
people have used AFS at a major university, and might help us out doing manual
testing if we give them a framework?

To paraphrase the .. well.. chief cat herder .. of the most widely deployed
operating system ever (Linux), 
"With enough QA testers, all bugs are shallow"

On Fri, Sep 14, 2012 at 04:42:37PM -0500, David Boyes wrote:
> > In this case I think you are low-balling the estimate.  To do it right it 
> > isn't
> > sufficient to test one build against itself.  You need to test new clients
> > against a range of old servers and vice versa in a constrained environment.
> > It is necessary to be able to identify when a change has an adverse
> > performance impact as well as accuracy.  There is a need to be able to
> > introduce intentional errors at various points in the protocol.  Just the
> > hardware costs are mid 5 digits and the software development is
> > significantly more than that.
> 
>  I agree --  if you were starting from scratch, you're probably right. 
> 
> But, a) I wasn't starting from scratch, so the additional equipment for 
> adding the AFS framework stuff was about what I quoted, and b) I was 
> discussing our tooling and test setup, not the general case. 
> We reused existing tooling in a number of places, and layered the AFS 
> component onto that. We do this kind of thing for other software, so we had a 
> decent baseline to start from. 
> 
> Solid QA infrastructure -- especially for complex systems -- isn't simple or 
> cheap; there we agree wholeheartedly. 
> 
> 
> 
> :??
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

On Fri, Sep 14, 2012 at 04:06:12PM -0500, David Boyes wrote:
> > Just to say explicitly, while OpenAFS developers are certainly welcome to
> > use whatever techniques make sense to them, I am completely uninterested
> > in doing anything at all with any of those half-assed meta-build systems and
> > will not assist in using them on Debian.  I believe they're irredeemably
> > broken as designed and are hopeless for generating packages that actually
> > work properly and integrate properly with the rest of the system, and have
> > better things to do with the time I have available to work on Debian
> > packages for OpenAFS.  Other people's mileage obviously may vary.
> 
> Opinion noted. Still, *something* has to drive the process, and if that 
> something can do more than one package format without having to write - and 
> maintain - a lot of custom scripting, then there's at least something worth 
> discussing there, given the recent project resource availability discussion 
> here and elsewhere. I can't see how burning developer time creating a 
> packaging tool is a smart use of resources when there are so many other 
> things that need doing far worse. 
> 

So automated testing costs a lot, and thus may not be practical.

How about an effort to get nightly builds of master available on 
as many platforms as possible, and getting thousands of bored college
students to download, install, and test them?

Wouldn't that massive crowsourced testing effort be worth the time of
a single developer to make sure *some* sort of package, even if it's
half-assed, gets distributed? I can't think of much of anything else 
that has a bigger resource multiplation factor than a 'one click install',
along with some defaults to use a 'test.openafs.org' cell.
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Jeffrey Altman]

On 9/15/2012 12:18 AM, Troy Benjegerdes wrote:
> 
> 
> Here's some code.
> 
> http://gerrit.openafs.org/#change,6844

The test itself is probably fine but the custom licensing is not.
OpenAFS accepts IPL1.0, MIT, and Simple BSD.  Pick one.

> Quick question: How many of these 1130 patchsets result in 'make check'
> completing successfully? 

Possibly none.  OpenAFS cannot prioritize the efforts of contributors.
Nor can it require that all code, documentation and tests written for
or derived from OpenAFS are contributed upstream.

> How about instead of long rants on the mailing list, we all spend 15 
> minutes thinking about a simple test that could go in 'make check'?

Please think about the audience you are targeting the request to because
it matters.  If you are asking individuals then your target audience is
not the Top-10 contributor list as their time is paid for by others.  If
you want individuals to contribute, then you will need to figure out how
to encourage those that do not regularly do so today to get involved.

If you are asking end user organizations, then requests on this mailing
list are unlikely to have much impact.  I do not believe that many of
the decision makers that set organizational priorities read it.

If you are asking support contract organizations, their priorities and
staffing are driven by the support customers.  If the support customers
ask that tests be written, then support engineer time will be spent on
writing tests.  I have never received a support request for writing
tests or documentation or protocol standards.

If you are asking product companies, then the person responsible for
managing development timelines and budget are the target audience.   The
determining factors I would evaluate are a cost-benefit analysis of the
work, the impact of performing the work on delivery dates, and
competitive advantage.

If you come to me with a request to fund unit test development I want to
see a work breakdown structure describing all of the tests to be
implemented, time/cost estimates for each test, and a benefit analysis
for each test.  At the moment, no such proposal exists but when asked,
the back of the envelope feedback that is received is similar to Russ'
from yesterday (http://tinyurl.com/8msb8dv).  The costs are very high,
there are benefits but not enough to prioritize writing tests above
other work.

Here is what I would suggest if you want to change my mind and those of
others that fund development resources.  Develop a work breakdown
structure for the testing that you want to see implemented.  Determine
estimates for time, cost and benefit.  Submit the proposal to this list
or privately.  If it is deemed that there is sufficient value to all or
part of the work, it might be possible to obtain funding or development
resources to implement it.

Jeffrey Altman




signature.asc
Description: OpenPGP digital signature

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

Here's some code.

http://gerrit.openafs.org/#change,6844

> As Tom Keiser wrote to you a few days ago.   Start contributing code
> that is useable to OpenAFS today.  If you want to write tests, people
> will jump up and down with joy.  However, please do not stomp your feet
> and scream that no one is doing anything when Your File System, Inc. has
> contributed more than 900 patchsets and Sine Nomine Associates more than
> 230 patchsets in just the last year.  Neither of these organizations
> have any obligation to contribute anything and yet both want to see
> OpenAFS survive.

Quick question: How many of these 1130 patchsets result in 'make check'
completing successfully? 

How about instead of long rants on the mailing list, we all spend 15 
minutes thinking about a simple test that could go in 'make check'?
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Jeffrey Altman]

On 9/14/2012 9:12 AM, Troy Benjegerdes wrote:
>>> However, this requires having a much greater availability of release
>>> management and testing resources.
>>
>> And perhaps an argument for automated tests that could prove out a release?
>> If you mean manual testing resources, given the scope of platform support 
>> and myriad branches for OpenAFS I doubt 'enough' will ever be enough :)  If 
>> we could bend those resources to creating and maintaining functional tests 
>> then that might be a better use of time.  Definitely a challenge though.
> 
> All this talk about 'reliable code for our users' is total BS
> until 'make check' actually does some realistic functionality tests.
> 
> If you can't write an automated test for a feature, then I would
> request we consider disabling that feature.

With all due respect, what group of individuals, companies or
organizations is covered by the above "you"?  Based upon your recent
e-mails I assume you mean the OpenAFS Gatekeepers, Sine Nomine
Associates and Your File System Inc.

I believe that everyone on this list will agree that unit tests,
distributed load testing, VFS and IFS kernel interface testing, packet
loss testing, performance testing, error path testing, etc. is important
BUT lets try to put some perspective on things.

The OpenAFS source code repository contains over one million lines of
code from more than 250 developers submitted over more than 25 years
supporting dozens if not hundreds of operating system versions,
processor architectures and distributions.

Of those 250+ contributors only 36 have contributed in the last 12
months.  Of those, only eight have contributed at least ten patches.
Here they are with their affiliations and patch counts to the master branch.
  12 month  All Time

  Jeffrey Altman [Your File System] 477  2980
  Simon Wilkinson [Your File System]230  1153
  Andrew Deason [Sine Nomine Associates]162   758
  Derrick Brashear [Your File System]   107  1865
  Mike Meffie [Sine Nomine Associates]   65   138
  Marc Dionne [Your File System / Individual]49   311
  Garrett Wollman [MIT CSAIL]4075
  Peter Scott [Your File System /Kernel Drivers] 3636
  Other  33  2965

We all know that commits to the master branch are not a direct
correlation to total contribution to the community.  There are other
indicators such as mailing list responses, IRC and Jabber responses,
pullups to 1.6 and 1.4 branches, Gerrit reviews, web site updates, etc.
which are not included.  However, the commit counts are easily obtained
from Ohloh (http://tinyurl.com/8lvxjph) and identify the point:

  There are very few organizations or individuals that
  contribute to OpenAFS in significant way.

In the last twelve months there have been 1248 patchsets submitted to
Gerrit and committed into the master branch.  The prior twelve months
had 1434 patchsets.  That is an awful lot of code to review and much of
that code included unit tests.  For a community that is effectively
running on $0 a year, that is exceptional.

Continuing to provide some perspective.  The Kermit Project at Columbia
University developed Kermit and generated nearly 700 binaries for each
release of C-Kermit.  http://www.columbia.edu/kermit/ckbinaries.html  At
its peak the Kermit Project, http://www.kermitproject.org/, had four FTE
responsible for coding, documentation, testing, building, and
distribution.  C-Kermit has less than half the amount of code and is
significantly less complex than OpenAFS.

More perspective.  The code IBM released as OpenAFS via the
DeveloperWorks site was pretty close to the source code that IBM used to
build the production releases of IBM AFS 3.6 in 2000.  I don't know what
the staffing was at IBM Pittsburgh Labs dedicated to AFS at the time but
it was certainly greater than the amount of dedicated staff at the
disposal of OpenAFS.

You have suggested removing all functionality that doesn't have
sufficient test suites.  As Mike Meffie points out, most of the
interesting testing is not functional unit testing but load testing,
deadlock analysis, multi-platform interoperability, etc.  But more
importantly, code that is included in the OpenAFS distribution is there
because someone probably uses it.  Since there is no method to determine
which features are used by whom, OpenAFS does not remove functionality
without a really strong reason.   The Gatekeepers do not want to make
upgrading to newer releases contingent upon reworking tooling that
relies upon some feature.

No insult intended to those that came before OpenAFS but the code
quality today is so much better today than it was two years ago, let
alone when 1.4 and 1.2 or the IBM DeveloperWorks releases were issued.
Significant strides have been made in removing warnings from the code
tree thanks to Simon, Marc, Garre

RE: [OpenAFS] buildbot and packages

[David Boyes]

> In this case I think you are low-balling the estimate.  To do it right it 
> isn't
> sufficient to test one build against itself.  You need to test new clients
> against a range of old servers and vice versa in a constrained environment.
> It is necessary to be able to identify when a change has an adverse
> performance impact as well as accuracy.  There is a need to be able to
> introduce intentional errors at various points in the protocol.  Just the
> hardware costs are mid 5 digits and the software development is
> significantly more than that.

 I agree --  if you were starting from scratch, you're probably right. 

But, a) I wasn't starting from scratch, so the additional equipment for adding 
the AFS framework stuff was about what I quoted, and b) I was discussing our 
tooling and test setup, not the general case. 
We reused existing tooling in a number of places, and layered the AFS component 
onto that. We do this kind of thing for other software, so we had a decent 
baseline to start from. 

Solid QA infrastructure -- especially for complex systems -- isn't simple or 
cheap; there we agree wholeheartedly. 



:��T���&j)b�   b�өzpJ)ߢ�^��좸!��l��b��(���~�+Y���b�ا~�~ȧ~

Re: [OpenAFS] buildbot and packages

[Russ Allbery]

Troy Benjegerdes  writes:

> All this talk about 'reliable code for our users' is total BS until
> 'make check' actually does some realisitic functionality tests.

Speaking as someone who has worked on adding a test framework to OpenAFS
and who is obsessive about test-driven development for his own code, I
agree with you.  However, again, you have to be sensitive to how much work
this is.

We unfortunately have a code base right now that is almost devoid of
testing.  Even putting aside the importance of integration and performance
testing for AFS and focusing just on unit testing, which isn't as useful
but which is much, much easier (and which *would* be useful for a lot of
components; for example, I'd love to see a thorough unit test for the
demand-attach state model), retrofitting this sort of test suite onto
existing software is hard.

This is particularly true given that we have a code base in C, not Java or
another language with strong testing support.  That means we don't have
tools like Mockito, or patterns like dependency injection and inversion of
control, available to us to make testing easier.  In C, you effectively
have to write all that stuff yourself, including mock components so that
you can write effective tests without requiring a vast infrastructure be
in place for any test to be effective.

To give you an example, I maintain a couple of PAM modules, and I decided
that I was sick and tired of doing development without a net and
retrofitted test suites onto both of them.  I now have an infrastructure
that I'm relatively happy with, one package entirely converted, and one
package partly converted, and I have test coverage (of the one that's
entirely converted) of about 60% of the option space.  That effort took
about 40 hours of dedicated programming effort solely on the test suite.

I've been writing C for 20 years; I'm not particularly slow at it.  But
among the things that I had to write:

* A mock PAM library so that I could feed configuration into my modules
  and inspect state to do proper testing.

* Infrastructure to allow user configuration of test accounts and
  principals in a Kerberos realm that could be used for testing for the
  Kerberos-related components.

* A scripting language for writing tests, combined with the parser and
  implementation of that language, so that tests could be data-driven.

This is *typical* of the kind of infrastructure that has to be developed
to retrofit test suites to large-scale C projects.

Furthermore, one of the reasons why test-driven development is so powerful
is that test-driven development changes how you write code.  You write
self-contained modules with clear interfaces because otherwise they're
untestable.  You insert annotation points and code injection points so
that you can instrument code and inspect state and ensure that it is
behaving properly.  You have to be much more careful about how
configuration and implicit system state knowledge is handled so that you
can configure things in a test mode.  All that makes your code much
better.

But the flip-side of that is that a huge code base that *wasn't* written
using test-driven development techniques doesn't do any of those things.
The code is frequently almost untestable.  There are no clear functional
boundaries where you can inject test machinery.  There is no little or
isolation to make independently-testable modules.  The configuration
support is frequently insufficient to properly configure the code to run
in a testing mode.  You have to do *huge* refactorings of the code in
order to add all of that.

And, from the perspective of an external client, all that work is
overhead.  It doesn't add any new features, it rarely fixes any
significant bugs while you're doing it, and it takes a ton of time.  No
one, and I mean no one, pays you to do that sort of work.  I was able to
do this for my PAM modules because Stanford gives me huge amounts of
discretion in how I do my job and is used to me periodically taking some
time away from "real work" to do these weird things that only Russ cares
about.  And even there, there's a limit; those 40 hours of development had
to be spread across two calendar years, where I could steal some time here
and some time there.

So yes, preach the testing religion, brother!  But you simply have to be
realistic about the prospects of comprehensive testing of a huge existing
code base that was never designed to be testable.

-- 
Russ Allbery (r...@stanford.edu) 
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Jeffrey Altman]

On 9/14/2012 12:12 PM, David Boyes wrote:
>>> All this talk about 'reliable code for our users' is total BS until
>>> 'make check' actually does some realisitic functionality tests.
>>> If you can't write an automated test for a feature, they I would
>>> request we consider disabling that feature.
> 
> I'm not sure this is a realistic goal in a single machine environment. For a 
> realistic testing environment, you need at least 10 system images (and the 
> ability to create a lot more would be very desirable for some of the subtler 
> bugs), and the ability to control time and replay multiple streams of events 
> in a repeatable way. It involves a separate Kerberos infrastructure, and a 
> lot of other moving parts, plus a lot of scripting to build the environment , 
> run the test, and then reset the environment for the next run. You also need 
> different types of systems, different OS levels, etc which complicate the 
> test even further.
>  
> It's not impossible, but I can say that it cost a fair amount (in the 
> mid-to-high 5 digit range) to build that environment here. 

David,

In this case I think you are low-balling the estimate.  To do it right
it isn't sufficient to test one build against itself.  You need to test
new clients against a range of old servers and vice versa in a
constrained environment.  It is necessary to be able to identify when a
change has an adverse performance impact as well as accuracy.  There is
a need to be able to introduce intentional errors at various points in
the protocol.  Just the hardware costs are mid 5 digits and the software
development is significantly more than that.

Jeffrey Altman




signature.asc
Description: OpenPGP digital signature

RE: [OpenAFS] buildbot and packages

[David Boyes]

> Just to say explicitly, while OpenAFS developers are certainly welcome to
> use whatever techniques make sense to them, I am completely uninterested
> in doing anything at all with any of those half-assed meta-build systems and
> will not assist in using them on Debian.  I believe they're irredeemably
> broken as designed and are hopeless for generating packages that actually
> work properly and integrate properly with the rest of the system, and have
> better things to do with the time I have available to work on Debian
> packages for OpenAFS.  Other people's mileage obviously may vary.

Opinion noted. Still, *something* has to drive the process, and if that 
something can do more than one package format without having to write - and 
maintain - a lot of custom scripting, then there's at least something worth 
discussing there, given the recent project resource availability discussion 
here and elsewhere. I can't see how burning developer time creating a packaging 
tool is a smart use of resources when there are so many other things that need 
doing far worse. 






___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Russ Allbery]

David Boyes  writes:

> There have been several discussions in the past on using a meta-build
> system like cmake or similar to try to address this, or at least using
> the packaging components. Some reorganization of the build process would
> probably be desirable to really take advantage of that, but I can't say
> that that would be unwelcome. The current setup is awfully clunky, and
> the meta-build system would allow generation of different types of
> packages from the same source (at least with cpack, you can generate
> RPM, .deb, Solaris, AIX and Windows packages fairly easily from the same
> packaging description file).

Just to say explicitly, while OpenAFS developers are certainly welcome to
use whatever techniques make sense to them, I am completely uninterested
in doing anything at all with any of those half-assed meta-build systems
and will not assist in using them on Debian.  I believe they're
irredeemably broken as designed and are hopeless for generating packages
that actually work properly and integrate properly with the rest of the
system, and have better things to do with the time I have available to
work on Debian packages for OpenAFS.  Other people's mileage obviously may
vary.

Proper Debian packaging for master is something that I've been thinking
about for a while, but I was waiting for all the libtool stuff to land
since that's going to drastically change how OpenAFS should be packaged.
At this point, I don't have a lot of resources in the immediate future to
work on that packaging, but most of the important changes are now in so
that we can start looking at what those changes should be.  I'm hoping to
have somewhat more time for this towards the end of the year.

There's no obvious reason why the Debian packaging files in the tree
should need to remain stale; it's just a matter of someone importing the
changes from the Debian packaging.  The more pressing problem at the
moment is that master has never been packaged for Debian and is
substantially different from the stable branch.

-- 
Russ Allbery (r...@stanford.edu) 
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Michael Meffie]

On Fri, 14 Sep 2012 08:12:42 -0500
Troy Benjegerdes  wrote:

> > > However, this requires having a much greater availability of release
> > > management and testing resources.
> > 
> > And perhaps an argument for automated tests that could prove out a release?
> > If you mean manual testing resources, given the scope of platform support 
> > and myriad branches for OpenAFS I doubt 'enough' will ever be enough :)  If 
> > we could bend those resources to creating and maintaining functional tests 
> > then that might be a better use of time.  Definitely a challenge though.
> 
> All this talk about 'reliable code for our users' is total BS
> until 'make check' actually does some realisitic functionality tests.
> 
> If you can't write an automated test for a feature, they I would
> request we consider disabling that feature.

I appreciate your zeal for automated functional testing, and there has been,
and continues to be improvement in the unit test code.  Everyone agrees,
functional unit testing it a good use of time, and helps improve the quality
and maintainability of the code.

Functional testing is mandated for changes where it makes sense, such as
changes to libraries, administrative commands, and so forth.  However, it is
important to keep in mind, OpenAFS is a complex, large, distributed
filesystem, supported on a large number, and ever changing number of platforms.
Much of the really interesting and complex interactions are not realistically
covered by a simple functional test of one one platform.

I'm not trying to suggest testing is not important, just keep in mind, there
are different types of testing, with different cost/benefit models.

Just my 2 cents

-- 
Michael Meffie 
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

RE: [OpenAFS] buildbot and packages

[David Boyes]

> > All this talk about 'reliable code for our users' is total BS until
> > 'make check' actually does some realisitic functionality tests.
> > If you can't write an automated test for a feature, they I would
> > request we consider disabling that feature.

I'm not sure this is a realistic goal in a single machine environment. For a 
realistic testing environment, you need at least 10 system images (and the 
ability to create a lot more would be very desirable for some of the subtler 
bugs), and the ability to control time and replay multiple streams of events in 
a repeatable way. It involves a separate Kerberos infrastructure, and a lot of 
other moving parts, plus a lot of scripting to build the environment , run the 
test, and then reset the environment for the next run. You also need different 
types of systems, different OS levels, etc which complicate the test even 
further.
 
It's not impossible, but I can say that it cost a fair amount (in the 
mid-to-high 5 digit range) to build that environment here. 
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

RE: [OpenAFS] buildbot and packages

[David Boyes]

> Most distro/OSes have their own packaging system, and it would seem that
> life would be easier for such potential testers if they could install a 
> snapshot
> of openafs within their packaging system.

I would argue that there's no value at all in doing this if this isn't the 
case. Installing things that don't play nice with the packaging system is a 
guaranteed recipe for disaster, or at least messing up a system in really 
unhelpful ways, and if the goal is improving testing, there's going to be a lot 
of install/uninstall cycles. Working with the packaging system is the best way 
to do that. 

>  However, the packaging files in
> our tree are frequently stale (since we are not the canonical source for
> them), so this could place a new burden on our distro maintainers if we
> choose to take this route.

There have been several discussions in the past on using a meta-build system 
like cmake or similar to try to address this, or at least using the packaging 
components. Some reorganization of the build process would probably be 
desirable to really take advantage of that, but I can't say that that would be 
unwelcome. The current setup is awfully clunky, and the meta-build system would 
allow generation of different types of packages from the same source (at least 
with cpack, you can generate RPM, .deb, Solaris, AIX and Windows packages 
fairly easily from the same packaging description file). 

You could also do this fairly easily without converting the whole package to 
cmake (you just need a comprehensive list of what binaries/scripts are needed 
and where they need to go) by calling the existing build system inside cmake 
and adding the packaging info to create packages from the output. We already do 
something similar here for our own internal builds, so maybe some of that 
tooling could be pushed upstream. 
 
> Also, there is a question of what version number to put on snapshots so that
> they will sort properly between "real" releases.

Suggestion: use a different package name to clearly distinguish between stable 
and beta releases (ie, something like openafs-beta instead of openafs). You 
could then use the same release/versioning information plus a build date or 
something to identify the package version. It would also clearly indicate to 
users that they are NOT getting the "stable" release and they shouldn't expect 
everything to work perfectly (not that that will stop the "must have bleeding 
edge" nutcases, but at least there's a clear "I told you so" warning). 

At least on Linux, that wouldn't be too hard to do (just a different spec file 
for RPM, and similar for Debian). The cmake overlay I mentioned above could 
also easily support this (the control files are plain text, and a bit of sed-fu 
would be sufficient to be able to generate packages with a clearly different 
name and version without a lot of rework). 


___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Benjamin Kaduk]

On Fri, 14 Sep 2012, Ken Dreyer wrote:


On Fri, Sep 14, 2012 at 9:04 AM, Benjamin Kaduk  wrote:

On Thu, 13 Sep 2012, Chaz Chandler wrote:
Also, there is a question of what version number to put on snapshots so that
they will sort properly between "real" releases.


Ordinarily "git describe" would be an easy way to come up with this,
but when I was looking at this a few weeks ago I hit a roadbump. For
1_6_1 we branched 1_6_x-stable to a separate 1_6_1 branch, and then
tagged version 1_6_1 on that branch. So it's not as simple as "git
describe" on the main 1_6_x branch. Maybe there will be another git
incantation we can use that would work in a similar way.


Not to mention that 'git describe' on master has been quite silly for a 
long time...


But the point I was really trying to make is that different packaging 
systems have different sorting behavior around special characters [-.~_] 
and the like, and some even forbid certain characters or the appearance of 
multi-letter substrings.  So custom work would probably be needed in some 
places, if not everywhere.


-Ben
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Derrick Brashear]

On Fri, Sep 14, 2012 at 9:12 AM, Troy Benjegerdes  wrote:
>> > However, this requires having a much greater availability of release
>> > management and testing resources.
>>
>> And perhaps an argument for automated tests that could prove out a release?
>> If you mean manual testing resources, given the scope of platform support 
>> and myriad branches for OpenAFS I doubt 'enough' will ever be enough :)  If 
>> we could bend those resources to creating and maintaining functional tests 
>> then that might be a better use of time.  Definitely a challenge though.
>
> All this talk about 'reliable code for our users' is total BS
> until 'make check' actually does some realisitic functionality tests.
>
> If you can't write an automated test for a feature, they I would
> request we consider disabling that feature.

You have TFS forked. I suggest trying that approach and seeing what
kind of filesystem you are left with.

It will probably not involve kernel extensions.

-- 
Derrick
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Ken Dreyer]

On Fri, Sep 14, 2012 at 9:04 AM, Benjamin Kaduk  wrote:
> On Thu, 13 Sep 2012, Chaz Chandler wrote:
> Also, there is a question of what version number to put on snapshots so that
> they will sort properly between "real" releases.

Ordinarily "git describe" would be an easy way to come up with this,
but when I was looking at this a few weeks ago I hit a roadbump. For
1_6_1 we branched 1_6_x-stable to a separate 1_6_1 branch, and then
tagged version 1_6_1 on that branch. So it's not as simple as "git
describe" on the main 1_6_x branch. Maybe there will be another git
incantation we can use that would work in a similar way.

- Ken
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Benjamin Kaduk]

On Thu, 13 Sep 2012, Chaz Chandler wrote:

no objection here, esp. if there's anyone out there with the spare time 
for and interest in testing them.


Most distro/OSes have their own packaging system, and it would seem that 
life would be easier for such potential testers if they could install a 
snapshot of openafs within their packaging system.  However, the packaging 
files in our tree are frequently stale (since we are not the canonical 
source for them), so this could place a new burden on our distro 
maintainers if we choose to take this route.


Also, there is a question of what version number to put on snapshots so 
that they will sort properly between "real" releases.


-Ben
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

> > However, this requires having a much greater availability of release
> > management and testing resources.
> 
> And perhaps an argument for automated tests that could prove out a release?
> If you mean manual testing resources, given the scope of platform support and 
> myriad branches for OpenAFS I doubt 'enough' will ever be enough :)  If we 
> could bend those resources to creating and maintaining functional tests then 
> that might be a better use of time.  Definitely a challenge though.

All this talk about 'reliable code for our users' is total BS
until 'make check' actually does some realisitic functionality tests.

If you can't write an automated test for a feature, they I would
request we consider disabling that feature.
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Troy Benjegerdes]

Don't think of this as a nightmare, think of this as an opportunity for
support contract upsales.

nightly installable builds and enthusiastic users that install the latest
one every day will make for a much more reliable product, and catch 
problems before they show up and cause trouble for bigger customers.

On Fri, Sep 14, 2012 at 02:54:20PM +0200, Harald Barth wrote:
> 
> > My big concern is that nightly installable builds will be a support
> > nightmare.
> 
> > There are a large number of users that will always take the latest
> > no matter what.
> 
> I know. Been in support. However, when "X does not work" it helps a
> lot if some $USER - even if he can't spell g i t or . / c o n f i g u
> r e ; m a k e can tell us that 2012-01-17 "it" worked and 2012-01-18
> it did not any more. The binaries that come form this type of build
> should however clearly tell so in the rxdebug output.
> 
> > to move to a biweekly release cycle. 
> 
> Nice if we would be there. Nice if it would be per month (which I find
> more realistic). Still above holds.
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Harald Barth]

> My big concern is that nightly installable builds will be a support
> nightmare.

> There are a large number of users that will always take the latest
> no matter what.

I know. Been in support. However, when "X does not work" it helps a
lot if some $USER - even if he can't spell g i t or . / c o n f i g u
r e ; m a k e can tell us that 2012-01-17 "it" worked and 2012-01-18
it did not any more. The binaries that come form this type of build
should however clearly tell so in the rxdebug output.

> to move to a biweekly release cycle. 

Nice if we would be there. Nice if it would be per month (which I find
more realistic). Still above holds.

Harald.
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Derrick Brashear]

if we do that, nightly (not per patch) builds would be better

Derrick


On Sep 14, 2012, at 0:59, Chaz Chandler  wrote:

> no objection here, esp. if there's anyone out there with the spare time for 
> and interest in testing them.
> 
>> -Original Message-
>> From: ja...@rampaginggeek.com
>> Sent: Thu, 13 Sep 2012 20:21:08 -0400
>> To: sha...@gmail.com
>> Subject: Re: [OpenAFS] buildbot and packages
>> 
>> Are there any objections to doing this for non-windows platforms? It
>> could be a nightly build.
>> 
>> On 09/13/2012 12:35 PM, Derrick Brashear wrote:
>>> we talked about doing it for releases; this would be a generalized case
>>> of it
>>> 
>>> possibly, but i can see it becoming a support nightmare. you're running
>>> what??
>>> 
>>> 
>>> On Thu, Sep 13, 2012 at 12:21 PM, Ken Dreyer 
>>> wrote:
>>>> Would it be feasible or desirable to have Buildbot actually provide
>>>> install-able packages (eg. RPMs on Linux, MSIs on Windows)? It could
>>>> help new users confirm "yes, this change fixes my bug".
>>>> 
>> ___
>> OpenAFS-info mailing list
>> OpenAFS-info@openafs.org
>> https://lists.openafs.org/mailman/listinfo/openafs-info
> 
> 
> FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop!
> Check it out at http://www.inbox.com/earth
> 
> 
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Chaz Chandler]

> My big concern is that nightly installable builds will be a support
> nightmare.
> There are a large number of users that will always take the latest no
> matter what.
> 
> I realize there is an argument to be made for users being free to do
> hang themselves.  But I question whether that is what organizational
> help desks are prepared to support.
> 
> The real issue that needs to be addressed is how to produce supportable
> releases on a more frequent basis.  In the past the consensus among the
> gatekeepers has been to move to a biweekly release cycle.  Whatever is
> ready for a given release date gets pulled up and anything that isn't,
> doesn't.
> 
> However, this requires having a much greater availability of release
> management and testing resources.

And perhaps an argument for automated tests that could prove out a release?
If you mean manual testing resources, given the scope of platform support and 
myriad branches for OpenAFS I doubt 'enough' will ever be enough :)  If we 
could bend those resources to creating and maintaining functional tests then 
that might be a better use of time.  Definitely a challenge though.

> Coupled with the biweekly release schedule, I believe it is important
> that the web site provide guidance to end users as to which release
> build is known to be reliable on which platforms.   Otherwise, users
> will
> always go for the most recent build and frequently end up with something
> much less reliable than we would prefer.
> 
> Jeffrey Altman


FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop!
Check it out at http://www.inbox.com/earth


___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Chaz Chandler]

no objection here, esp. if there's anyone out there with the spare time for and 
interest in testing them.

> -Original Message-
> From: ja...@rampaginggeek.com
> Sent: Thu, 13 Sep 2012 20:21:08 -0400
> To: sha...@gmail.com
> Subject: Re: [OpenAFS] buildbot and packages
> 
> Are there any objections to doing this for non-windows platforms? It
> could be a nightly build.
> 
> On 09/13/2012 12:35 PM, Derrick Brashear wrote:
>> we talked about doing it for releases; this would be a generalized case
>> of it
>> 
>> possibly, but i can see it becoming a support nightmare. you're running
>> what??
>> 
>> 
>> On Thu, Sep 13, 2012 at 12:21 PM, Ken Dreyer 
>> wrote:
>>> Would it be feasible or desirable to have Buildbot actually provide
>>> install-able packages (eg. RPMs on Linux, MSIs on Windows)? It could
>>> help new users confirm "yes, this change fixes my bug".
>>> 
> ___
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info


FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop!
Check it out at http://www.inbox.com/earth


___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Jeffrey Altman]

My big concern is that nightly installable builds will be a support 
nightmare.
There are a large number of users that will always take the latest no
matter what.

I realize there is an argument to be made for users being free to do
hang themselves.  But I question whether that is what organizational
help desks are prepared to support.

The real issue that needs to be addressed is how to produce supportable
releases on a more frequent basis.  In the past the consensus among the
gatekeepers has been to move to a biweekly release cycle.  Whatever is
ready for a given release date gets pulled up and anything that isn't, 
doesn't.

However, this requires having a much greater availability of release
management and testing resources.

Coupled with the biweekly release schedule, I believe it is important
that the web site provide guidance to end users as to which release
build is known to be reliable on which platforms.   Otherwise, users 
will
always go for the most recent build and frequently end up with something
much less reliable than we would prefer.

Jeffrey Altman


On Thursday, September 13, 2012 8:21:08 PM, Jason Edgecombe wrote:
> Are there any objections to doing this for non-windows platforms? It
> could be a nightly build.
>
> On 09/13/2012 12:35 PM, Derrick Brashear wrote:
>> we talked about doing it for releases; this would be a generalized
>> case of it
>>
>> possibly, but i can see it becoming a support nightmare. you're
>> running what??
>>
>>
>> On Thu, Sep 13, 2012 at 12:21 PM, Ken Dreyer 
>> wrote:
>>> Would it be feasible or desirable to have Buildbot actually provide
>>> install-able packages (eg. RPMs on Linux, MSIs on Windows)? It could
>>> help new users confirm "yes, this change fixes my bug".
>>>
> ___
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info



signature.asc
Description: OpenPGP digital signature

Re: [OpenAFS] buildbot and packages

[Jason Edgecombe]

Are there any objections to doing this for non-windows platforms? It 
could be a nightly build.


On 09/13/2012 12:35 PM, Derrick Brashear wrote:

we talked about doing it for releases; this would be a generalized case of it

possibly, but i can see it becoming a support nightmare. you're running what??


On Thu, Sep 13, 2012 at 12:21 PM, Ken Dreyer  wrote:

Would it be feasible or desirable to have Buildbot actually provide
install-able packages (eg. RPMs on Linux, MSIs on Windows)? It could
help new users confirm "yes, this change fixes my bug".


___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] buildbot and packages

[Jeffrey Altman]

At least on Windows, 'no'.   The builders have no access to 
certificates and private keys necessary for digital signatures and do 
not build either the documentation nor the installation packages.  
Building complete installation packages would increase the build time 
by more than a factor of three.   Publicly distributed binaries also 
have symbols and binaries registered with Microsoft which is a very 
manual process.



On Thursday, September 13, 2012 12:21:25 PM, Ken Dreyer wrote:
> Would it be feasible or desirable to have Buildbot actually provide
> install-able packages (eg. RPMs on Linux, MSIs on Windows)? It could
> help new users confirm "yes, this change fixes my bug".
>
> - Ken
> ___
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info



signature.asc
Description: OpenPGP digital signature

Re: [OpenAFS] buildbot and packages

[Derrick Brashear]

we talked about doing it for releases; this would be a generalized case of it

possibly, but i can see it becoming a support nightmare. you're running what??


On Thu, Sep 13, 2012 at 12:21 PM, Ken Dreyer  wrote:
> Would it be feasible or desirable to have Buildbot actually provide
> install-able packages (eg. RPMs on Linux, MSIs on Windows)? It could
> help new users confirm "yes, this change fixes my bug".
>
> - Ken
> ___
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>



-- 
Derrick
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info