Re: Getting Maven component and plugin releases to /dist -- space
On Sun, 23 Dec 2012, Benson Margulies wrote: Date: Sun, 23 Dec 2012 14:46:57 +0100 From: Benson Margulies To: "infrastruct...@apache.org" , Maven Developers List Subject: Re: Getting Maven component and plugin releases to /dist -- space As far as I can determine, there should never be / should never have been one of these items pushed to repository.apache.org except after a full release vote. If a vote wasn't passed, the staging repo should have been dropped, not released. As far as I am concerned, this whole set are full, voted, releases that should have originally gone to /dist (and thence to archive), and the only sorting out is between dist and archive. So, as I see it, the Maven PMC has now given infrastructure@ all the information you need about the disk space demands of this bit of retroactive release management. With the holiday upcoming, I'll wait until 28 December for further comments or objections before proceeding. Hi Benson, sorry for being dense ; ..., but do you have a list handy showing which files you want to add to /dist/ and where? from -> to ... -> dist/... Either all of them (including the ones really only destined for archive.a.o), or only the 'real', 'recent' stuff, or both. Just curious, are the .md5's and .asc's on these to-be-added artifacts included and checked for validity ? Thanks a lot, regards, HPP On Sat, Dec 22, 2012 at 7:43 PM, Benson Margulies wrote: Releases to go directly to the archive: Total size 64852152 Releases to go to the live /dist: Total size 21369565 The details are in additional files checked into the same directory. On Sat, Dec 22, 2012 at 6:23 PM, Benson Margulies wrote: On Sat, Dec 22, 2012 at 5:11 PM, Gavin McDonald wrote: -Original Message- From: Benson Margulies [mailto:bimargul...@gmail.com] Sent: Sunday, 23 December 2012 8:31 AM To: Maven Developers List Cc: infrastruct...@apache.org Subject: Getting Maven component and plugin releases to /dist -- space https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus- releases/inventory.sizes contains my attempt to inventory all the Maven releases that are on repository.apache.org but are not on /dist. The bottom line, literally, is 86221717 bytes -- not counting the detached signature files. Infra@, please let us know if this poses a problem; we won't actually check anything into /dist before hearing from you (and taking care of some organizational matters of our own). I see beta versions in there, they don’t belong in dist. Yes they do, they were formal, voted, legal, Apache releases, that just happened to have 'beta' in their names. I see multiple versions of the same product in there. Only the LATEST version of each released product belongs in dist. (And certainly no need to be bothering mirrors with outdated and since replaced versions.) The 'archive' section of the policy needs to be complied with. If you want to distinguish the disk space of the two, I can. You should recompile your list and total size based on removing the above. I can certainly arrange subtraction of the items that need to just to the archive. We'll see what the situation is after that. Gav... _ Henk P. Penning, ICT-beta R Uithof WISK-412 _/ \_ Faculty of Science, Utrecht UniversityT +31 30 253 4106 / \_/ \ Budapestlaan 6, 3584CD Utrecht, NLF +31 30 253 4553 \_/ \_/ http://www.staff.science.uu.nl/~penni101/ M penn...@uu.nl \_/ - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Getting Maven component and plugin releases to /dist -- space
On Sun, 23 Dec 2012, Benson Margulies wrote: Date: Sun, 23 Dec 2012 15:26:18 +0100 From: Benson Margulies To: Henk P. Penning Cc: Maven Developers List , "infrastruct...@apache.org" Subject: Re: Getting Maven component and plugin releases to /dist -- space Hi Benson, excellent. As far as I can see, everything can go directly into "archive.a.o/dist/maven/". The only point of having stuff in /dist/ is that projects can refer users/downloaders to that stuff on a mirror. Since there will be no such pointers to the stuff you want to add, there is no point in having it on the mirrors, and therefore in /dist/. Remember that /dist/ is only a cache for the 'recent' stuff in "archive.a.o". Nothing in /dist/ is there permanently. If you would only put 'your' stuff in archive.a.o, it would avoid the problem of cleaning up in "/dist/maven/{plugins,components)". Actually, I am opposed to the "automated" addition of stuff to /dist/, if the removal or that stuff isn't "automated" by the same mechanism too ; "cleanup" is as important as "adding stuff". [ privately I still wonder why this stuff has to be archive.a.o'ed ; if this stuff, why not everything else ; if not everything, why this stuff? If these artefact are special, why doesn't it suffice to just have a list of them somewhere? ] HPP If you look in https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus-releases, you will find two files, inventory.archive and inventory.latest. These list the release archives that, as far as I can tell, make up Maven PMC releases that (a) were voted, and (b) were never put on /dist. The 'latest' file contains current releases, and the 'archive' file contains not-current releases. So, my intention is to push the files listed in the 'latest' file to /dist, and the others directly to archive. As for 'where' on /dist, my proposal to the Maven PMC is going to be to have two subdirectories of the Maven /dist: 'plugins' and 'components'. However, the Maven PMC is still in the early phases of confronting the requirement to do this at all, so it will be a few days, I expect, before there's a consensus. I'd be grateful if you have anything to add here; as you know, I fell into this project by discovering the hard way that it had to be done. This hasn't been my historical area of expertise. All these files currently reside on repository.apache.org. The Nexus configuration there forces each file to be accompanied by valid signatures when you deploy them there. The python program I've written (which you can also find there) just pulls metadata from Nexus. The next python program will actually pull files, including their accompanying detached signatures. Since the Maven project releases something about once a week, this next script will be driven by the Nexus metadata, not by the files. So the inventory is likely to creep up a bit by the time we go live. It wouldn't hurt to run a verification pass on the signatures before actually pushing these files live. Have you got a script that examines an entire directory? I suspect that you do. Regards, Benson _ Henk P. Penning, ICT-beta R Uithof WISK-412 _/ \_ Faculty of Science, Utrecht UniversityT +31 30 253 4106 / \_/ \ Budapestlaan 6, 3584CD Utrecht, NLF +31 30 253 4553 \_/ \_/ http://www.staff.science.uu.nl/~penni101/ M penn...@uu.nl \_/ - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Getting Maven component and plugin releases to /dist -- space
On Sun, Dec 23, 2012 at 10:26 AM, Henk P. Penning wrote: > On Sun, 23 Dec 2012, Benson Margulies wrote: > > Date: Sun, 23 Dec 2012 15:26:18 +0100 >> From: Benson Margulies >> To: Henk P. Penning >> Cc: Maven Developers List , >> "infrastruct...@apache.org" >> >> Subject: Re: Getting Maven component and plugin releases to /dist -- space >> > > Hi Benson, > > excellent. > > As far as I can see, everything can go directly into > "archive.a.o/dist/maven/". > > The only point of having stuff in /dist/ is that projects > can refer users/downloaders to that stuff on a mirror. > Since there will be no such pointers to the stuff you want > to add, there is no point in having it on the mirrors, and > therefore in /dist/. > I would completely agree with that. But it's not what the policy says. It says it must go to dist. Do we really mean archives for everything and dist only for things that are linked from webpages so they can be mirrored? That's completely sensible.
Re: Getting Maven component and plugin releases to /dist -- space
Henk, I'm sorry that I'm effectively top-posting; for some reason gmail doesn't want to set up interlinear commentary on your message. If I miss something, please let me know. If you look in https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus-releases, you will find two files, inventory.archive and inventory.latest. These list the release archives that, as far as I can tell, make up Maven PMC releases that (a) were voted, and (b) were never put on /dist. The 'latest' file contains current releases, and the 'archive' file contains not-current releases. So, my intention is to push the files listed in the 'latest' file to /dist, and the others directly to archive. As for 'where' on /dist, my proposal to the Maven PMC is going to be to have two subdirectories of the Maven /dist: 'plugins' and 'components'. However, the Maven PMC is still in the early phases of confronting the requirement to do this at all, so it will be a few days, I expect, before there's a consensus. I'd be grateful if you have anything to add here; as you know, I fell into this project by discovering the hard way that it had to be done. This hasn't been my historical area of expertise. All these files currently reside on repository.apache.org. The Nexus configuration there forces each file to be accompanied by valid signatures when you deploy them there. The python program I've written (which you can also find there) just pulls metadata from Nexus. The next python program will actually pull files, including their accompanying detached signatures. Since the Maven project releases something about once a week, this next script will be driven by the Nexus metadata, not by the files. So the inventory is likely to creep up a bit by the time we go live. It wouldn't hurt to run a verification pass on the signatures before actually pushing these files live. Have you got a script that examines an entire directory? I suspect that you do. Regards, Benson - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Getting Maven component and plugin releases to /dist -- space
As far as I can determine, there should never be / should never have been one of these items pushed to repository.apache.org except after a full release vote. If a vote wasn't passed, the staging repo should have been dropped, not released. As far as I am concerned, this whole set are full, voted, releases that should have originally gone to /dist (and thence to archive), and the only sorting out is between dist and archive. So, as I see it, the Maven PMC has now given infrastructure@ all the information you need about the disk space demands of this bit of retroactive release management. With the holiday upcoming, I'll wait until 28 December for further comments or objections before proceeding. On Sat, Dec 22, 2012 at 7:43 PM, Benson Margulies wrote: > Releases to go directly to the archive: Total size 64852152 > > Releases to go to the live /dist: Total size 21369565 > > The details are in additional files checked into the same directory. > > > On Sat, Dec 22, 2012 at 6:23 PM, Benson Margulies > wrote: >> On Sat, Dec 22, 2012 at 5:11 PM, Gavin McDonald >> wrote: >>> >>> -Original Message- From: Benson Margulies [mailto:bimargul...@gmail.com] Sent: Sunday, 23 December 2012 8:31 AM To: Maven Developers List Cc: infrastruct...@apache.org Subject: Getting Maven component and plugin releases to /dist -- space https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus- releases/inventory.sizes contains my attempt to inventory all the Maven releases that are on repository.apache.org but are not on /dist. The bottom line, literally, is 86221717 bytes -- not counting the detached signature files. Infra@, please let us know if this poses a problem; we won't actually check anything into /dist before hearing from you (and taking care of some organizational matters of our own). >>> >>> I see beta versions in there, they don’t belong in dist. >> >> Yes they do, they were formal, voted, legal, Apache releases, that >> just happened to have 'beta' in their names. >> >>> I see multiple versions of the same product in there. Only the LATEST >>> version of each >>> released product belongs in dist. (And certainly no need to be bothering >>> mirrors >>> with outdated and since replaced versions.) >> >> The 'archive' section of the policy needs to be complied with. If you >> want to distinguish the disk space of the two, I can. >> >> >>> >>> You should recompile your list and total size based on removing the above. >> >> I can certainly arrange subtraction of the items that need to just to >> the archive. >> >>> >>> We'll see what the situation is after that. >>> >>> Gav... >>> - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Getting Maven component and plugin releases to /dist -- space
Releases to go directly to the archive: Total size 64852152 Releases to go to the live /dist: Total size 21369565 The details are in additional files checked into the same directory. On Sat, Dec 22, 2012 at 6:23 PM, Benson Margulies wrote: > On Sat, Dec 22, 2012 at 5:11 PM, Gavin McDonald > wrote: >> >> >>> -Original Message- >>> From: Benson Margulies [mailto:bimargul...@gmail.com] >>> Sent: Sunday, 23 December 2012 8:31 AM >>> To: Maven Developers List >>> Cc: infrastruct...@apache.org >>> Subject: Getting Maven component and plugin releases to /dist -- space >>> >>> https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus- >>> releases/inventory.sizes >>> >>> contains my attempt to inventory all the Maven releases that are on >>> repository.apache.org but are not on /dist. >>> >>> The bottom line, literally, is 86221717 bytes -- not counting the detached >>> signature files. >>> >>> Infra@, please let us know if this poses a problem; we won't actually check >>> anything into /dist before hearing from you (and taking care of some >>> organizational matters of our own). >> >> I see beta versions in there, they don’t belong in dist. > > Yes they do, they were formal, voted, legal, Apache releases, that > just happened to have 'beta' in their names. > >> I see multiple versions of the same product in there. Only the LATEST >> version of each >> released product belongs in dist. (And certainly no need to be bothering >> mirrors >> with outdated and since replaced versions.) > > The 'archive' section of the policy needs to be complied with. If you > want to distinguish the disk space of the two, I can. > > >> >> You should recompile your list and total size based on removing the above. > > I can certainly arrange subtraction of the items that need to just to > the archive. > >> >> We'll see what the situation is after that. >> >> Gav... >> - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Getting Maven component and plugin releases to /dist -- space
On Sat, Dec 22, 2012 at 5:11 PM, Gavin McDonald wrote: > > >> -Original Message- >> From: Benson Margulies [mailto:bimargul...@gmail.com] >> Sent: Sunday, 23 December 2012 8:31 AM >> To: Maven Developers List >> Cc: infrastruct...@apache.org >> Subject: Getting Maven component and plugin releases to /dist -- space >> >> https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus- >> releases/inventory.sizes >> >> contains my attempt to inventory all the Maven releases that are on >> repository.apache.org but are not on /dist. >> >> The bottom line, literally, is 86221717 bytes -- not counting the detached >> signature files. >> >> Infra@, please let us know if this poses a problem; we won't actually check >> anything into /dist before hearing from you (and taking care of some >> organizational matters of our own). > > I see beta versions in there, they don’t belong in dist. Yes they do, they were formal, voted, legal, Apache releases, that just happened to have 'beta' in their names. > I see multiple versions of the same product in there. Only the LATEST version > of each > released product belongs in dist. (And certainly no need to be bothering > mirrors > with outdated and since replaced versions.) The 'archive' section of the policy needs to be complied with. If you want to distinguish the disk space of the two, I can. > > You should recompile your list and total size based on removing the above. I can certainly arrange subtraction of the items that need to just to the archive. > > We'll see what the situation is after that. > > Gav... > - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Getting Maven component and plugin releases to /dist -- space
Gavin McDonald wrote on Sun, Dec 23, 2012 at 08:41:22 +1030: > > > > -Original Message- > > From: Benson Margulies [mailto:bimargul...@gmail.com] > > Sent: Sunday, 23 December 2012 8:31 AM > > To: Maven Developers List > > Cc: infrastruct...@apache.org > > Subject: Getting Maven component and plugin releases to /dist -- space > > > > https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus- > > releases/inventory.sizes > > > > contains my attempt to inventory all the Maven releases that are on > > repository.apache.org but are not on /dist. > > > > The bottom line, literally, is 86221717 bytes -- not counting the detached > > signature files. > > > > Infra@, please let us know if this poses a problem; we won't actually check > > anything into /dist before hearing from you (and taking care of some > > organizational matters of our own). > > I see beta versions in there, they don’t belong in dist. Why? If they have 3 +1's and the rest of the legal ducks in a row, they can go on dist/. But contrarily, .0 versions that haven't passed voting (eg, serf 0.5.0) don't belong on dist/. - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
RE: Getting Maven component and plugin releases to /dist -- space
> -Original Message- > From: Benson Margulies [mailto:bimargul...@gmail.com] > Sent: Sunday, 23 December 2012 8:31 AM > To: Maven Developers List > Cc: infrastruct...@apache.org > Subject: Getting Maven component and plugin releases to /dist -- space > > https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus- > releases/inventory.sizes > > contains my attempt to inventory all the Maven releases that are on > repository.apache.org but are not on /dist. > > The bottom line, literally, is 86221717 bytes -- not counting the detached > signature files. > > Infra@, please let us know if this poses a problem; we won't actually check > anything into /dist before hearing from you (and taking care of some > organizational matters of our own). I see beta versions in there, they don’t belong in dist. I see multiple versions of the same product in there. Only the LATEST version of each released product belongs in dist. (And certainly no need to be bothering mirrors with outdated and since replaced versions.) You should recompile your list and total size based on removing the above. We'll see what the situation is after that. Gav... - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Getting Maven component and plugin releases to /dist -- space
On 22/12/2012 22:00, Benson Margulies wrote: > https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus-releases/inventory.sizes > > contains my attempt to inventory all the Maven releases that are on > repository.apache.org but are not on /dist. > > The bottom line, literally, is 86221717 bytes -- not counting the > detached signature files. > > Infra@, please let us know if this poses a problem; we won't actually > check anything into /dist before hearing from you (and taking care of > some organizational matters of our own). Only the latest version needs to go on /dist. All versions need to be on archive.a.o ~86MB is not a significant amount. The current size of /dist is 10s of GB. It might be simplest to copy everything to /dist, wait for it to copy to the archives and then delete all but the latest versions from /dist. Mark - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org