I'm going to cc this discussion onto infrastructure. I think it is a valid topic which we need to address there in more detail.
You seem right on target with your e-mails. At least your understanding of the situation matches mine :-D. In review...
1) we don't want "snapshots" available through www.apache.org/dist/ nor the mirrors of that location. We push only official, versioned, pmc-endorsed, signed files to our mirrors, and everything published through www.apache.org/dist goes to our mirrors. I think that rule is not up for debate (nor do I see a reason to debate it).
2) The ASF does have several machines available to us, and we're still figuring out the optimal way to use each of those machines. Suffice to say, there is hardware, bandwidth and disk space, and infrastructure is here to facilitate the needs of other ASF projects. That said, infrastructure also has other concerns to take into account (like security), but in principle the goal is to help you meet your goal.
3) Having a distribution-setup-per-project does not scale and is a maintainance nightmare. Hence there is a need for multiple projects to co-operate on a common setup.
4) The ASF is not interested in providing redistribution of non-ASF files (ie performing the role that ibiblio's maven repository has been serving so well lately). I've heard several reasons for that; the most important one is probably that its stretching the stated goals of the foundation by quite a bit ("ASF is in the software creation business, not the software distribution business").
5) We do want snapshot builds available for "developer" use. The old (current) recommended model for this is that people push these to either their own public_html directory or cvs.apache.org/dist. IIRC, historically, use of home directories for this was discouraged because of disk space limits (the cvs machine was a seperate one), but that's no longer a problem.
6) A while back, [EMAIL PROTECTED] was established as a "subcommittee" to deal with this whole repository topic and figure out these complex issues (the "new" model), and decide on what to do. I'm not sure what happened there and if a consensus was ever reached, but its probably the most appropriate forum for achieving it.
Just explain what you need, why you need it, and keep all the above context in mind (as I think you already do). Please work on [EMAIL PROTECTED] to gain a consensus about that stuff. Then approach infrastructure@ with some rough info on what you need (ie number of GB, expected bandwidth, seperate unix group or other kind of permission scheme, etc etc), and we can make something happen. Especially if there's volunteers to help out :-D
A final note is that I've been also dabbling in a related area using the gump box: automated nightly builds. Info on that is @ http://wiki.apache.org/gump/NightlyBuilds.
Now some loose comments just to make sure this is all clear...
Jason van Zyl wrote:On Sat, 2004-07-17 at 21:03, Brett Porter wrote:
I don't know very much about cvs.apache.org/repository.
What I was suggesting was that snapshots that we want to keep should go to /dist/java-repository (which is what happens now).
What goes into /dist needs to be versioned, signed, pmc-endorsed, and be intented to live "forever" (ie several years). This is the agreement with the ASF mirrors. Please don't just push snapshots there.
Eventually we should actually have a ASF Repository structure that all Apache releases go
well, we do have that already (the mirror guidelines). Its just that it doesn't work that well for nightly builds, snapshots, etc.
Most projects out there do maintain major version releases in dist, and if they are removed from the projects releases directory, its upto the mirror to decide its cleanup strategy in rsync, this means some mirrors will not delete copies, while others will. I beleive archive.apache.org is meant to be such a mirror that maintains all the content that was ever in the www.apache.org/dist directory.
yep.
As such, IMHO, the restictions placed on dist are to reduce the archiving of any unofficial releases. So, IMHO placing any "dated build" releases into the ASF Repository violates the requirements because they will be "archived" and they are not a full release.
yep.
I recognize its almost impossible for Infrustructure to enforce this policy;
Infrastructure won't. Any problems will just get pushed back to the relevant PMC for resolution. ASF projects are self-managing and hence responsible for this enforcement :-D
for example; managing the release of every single maven plugin within Infrustructure is probibly not in their interest. What is in their interest is being able to maintain a secure release directory with properly signed releases.
yes. Everyone's interest, in fact :-D
I'm convinced we are at a point where Infrastructure needs to reflect on the current release strategy and attempt to open it up so that we can manage more complex release scenarios. We also need to establish a course of action to migrate the directory structures to that structure defined by the ASF Repository Spec.
seems like it. I'll point out most of the non-java projects (ie httpd) have no particular reason to want to migrate (no incentive for them). So "spec backward compatibility" is something to consider as well...consistent structure is nice.
The battle here for us is to maintain "group cohesion" between all the Apache projects in terms of where and how to release content, no matter how painful interaction can become. So, pushing content off to a different location isn't very beneficial in this regard. It may be tedious and frustrating to deal with, but we need to maintain such interaction.
That's indeed optimal. Doesn't mean that in the meantime "stopgap" solutions are out of the question. IE just setting up /www/cvs.apache.org/repository and giving everyone in apsite write permissions and syncing that to ibiblio is not bad per se.
What this means to me is that we have a few possible solutions for the future direction of ASF Repository:
1.) do not allow any interim content into www/dist. Interm content gets placed under cvs.apache.org/repository. This gives us a means to allow testing of interim content while not having all the mirrors (including ibibilio and the archive) get flooded with it.
I think it is very hard to change the relationship between the ASF and all its mirrors, ie this is a hard-to-change rule. Some smart technical solution is probably a lot easier.
2.) Establish a means to filter what gets into the archive (and possibly the mirrors). Store all content under one repository location (in dist). This is allot more complex and will require effort across multiple projects and infrastructure to establish as a practice. (I'm not convinced this will be possible).
well I can imagine that the maven release/artifact/dist plugins understand when something is a "release" as opposed to a snapshot or temporary build easily enough. Excalibur is currently just parsing the <version/> and deciding the publication path based on that...
3.) Establish the ASF Repository independent of the www/dist directory and manage all the content within it, providing our own mechanisms for archiving and allowing interim builds. Initial structure will be "Maven" like, migrating however, to the ASF Repository Specification. The problem is that this fractures the community into those that want to release in the old directory structure and those that want to release in the Repository.
Another downside is that releases of people using the "new" setup won't be mirrored. Bandwidth may be cheap, but everyone downloading maven 1.0 from the ASF machines instead of a local mirror still is not (both in terms of money and efficient network usage).
So, theres allot of discussion and voting that needs to continue on this subject to determine the best course of action, I think that the current compromise is not an effective solution for the long term, It reflects my initial efforts to establish a repository for ASF projects and now that we have our footing we need more member involvement and decision making to establish the proper direction for this to go into. This kind of growth is hard, changing the way a project releases is a project in and of itself.
All the more thanks go out to all the people trying! :-D
Do you know what the lifespan is for snapshots stored in cvs.apache.org/repository
There is no "rule" on that from infrastructure, nor do I believe there's a process that cleans it out...
cheers,
- LSD
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
