Re: Bootstraping a repository manager
On 19/05/2011 12:49 PM, Wendy Smoak wrote: On Thu, May 19, 2011 at 12:40 PM, Heck, Gus (Patrick) wrote: Hmm, how would I go about convincing maven-plugins such as clean to only use the latest versions (in some cases it just seems to be a dep on the pom file, not any actual jars). Don't know where I would find the handle for this in maven, unless one does it by editing the poms that are wandering off into past versions I suppose. No, don't mess with the released poms for the plugins. Ron was talking about controlling what dependencies are used in your own projects. Correct. I have enough trouble with my own stuff and I am not deploying any Maven plug-in code. Ron There are two separate things here: - the plugins and other libraries Maven uses to build your code - the actual artifacts the build produces that contain your compiled code, config files, etc. Are you really that concerned about how many parent poms Maven needs to download to do its job? For Maven plugins with an internal (disconnected) repository, you will go mad trying to download-and-upload each thing individually. There will be hundreds of Maven/Plexus/Commons/Etc jars to find. Just let Maven gather it all up, give it a once over, and put it in your repo. Or is the real issue that you want to make sure you know exactly what your code depends on and what you are putting in production or delivering to customers? In my experience, that's the bigger issue. - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Bootstraping a repository manager
On 19/05/2011 12:48 PM, Heck, Gus (Patrick) wrote: Interesting however that nobody has yet mentioned the apache home grown archiva... It is used by a number of people here but the support for Nexus by Sonotype is very good and you can start with the Community version and as your needs get more sophisticated, move to the Pro version. You might already be there if some of conversation that has gone on here is correct but that is beyond my experience. I can only say that I wish I had got Nexus earlier in the project, it really makes Maven much easier to use. Ron -Gus Patrick Heck ||| Principal Software Engineer Aspen Technology, Inc. ||| +1 781-221-6483 ||| www.aspentech.com -Original Message- From: Ron Wheeler [mailto:rwhee...@artifact-software.com] Sent: Thursday, May 19, 2011 12:03 PM To: users@maven.apache.org Subject: Re: Bootstraping a repository manager I am just using the Free version and although talking about competing repo managers is discouraged in the form, I am a big fan of the Nexus approach and would heartily recommend it to anyone who is using Maven Ron On 19/05/2011 10:35 AM, Brian Fox wrote: It's also worth mentioning that Nexus Professional's Procurement feature is built for exactly the use case you have. It's meant to have a hard firewall like separation between internal and external artifacts and rules that allow you to approve whitelist/blacklist style, or by wildcard or other runtime evaluation to define what can be used internally. On Thu, May 19, 2011 at 9:24 AM, Ron Wheeler wrote: If you followed my directions or Wendy's, you will have a repo that contains only the dependencies (artifact and exact versions) that your software requires and nothing else. I don't know Artifactory but I do use Nexus so you will have to translate my instructions. In Nexus, you can see each of the artifacts that you have requested and extract the license information. If you start at the top of the list and work your way down in order, you will not miss anything unless you skip one by mistake. Keep a spreadsheet as a checklist if you worry about that. A bit of a PITA but not as bad as you are envisioning. You will also see where you are using more than one version of an artifact so you can limit the dependencies that you may want to clean up. Ron On 19/05/2011 3:23 AM, Nicola Musatti wrote: Once you had Maven collect all the dependencies for you you can use something like find repository_storage_root -name '*.jar' if you are on Linux/UNIX or Windows's search tool to obtain a list of all your dependencies. With that you can use something like http://mvnrepository.com/ to gather information about each jar and make your decisions. This would take you a lot less than your approach. Cheers, Nicola Musatti Heck, Gus (Patrick) wrote: Hi Wendy, I don't have a requirement to build from source, but in the case of the maven plugins, nowhere on the web seems to point to a place where I can download the finished product (aside from letting maven find it, and who knows what other dependencies). I am adverse to letting maven just pull things because then I have a very long list of items to check and verify arranged in a tree, and it's all too easy to miss a subtree and think I've got everything covered. Basically the problem is I make a really crappy computer when it comes to tree-traversals and not losing my place when someone comes over to chat, I go to lunch, or I have to go home at the end of the day. :-) I don't want to trust that I won't make a mistake, I'd rather take a bit more time and let the build system ensure that I didn't miss anything. If I act as gatekeeper, I know that the project won't build properly until I've applied each dependency required to build the project. (and then same for test... etc). This shouldn't be any more burdensome than finding the transitive dependencies for a good sized ant project, which I've done many times before, except that maven plug-in folks seem to squirrel their stuff away where only maven can find it, so I'm having trouble getting maven working to start with. -Gus -Original Message- From: Wendy Smoak [mailto:wsm...@gmail.com] Sent: Wednesday, May 18, 2011 6:35 PM To: Maven Users List Subject: Re: Bootstraping a repository manager On Wed, May 18, 2011 at 5:21 PM, Heck, Gus (Patrick) wrote: I can't seem to find a place to download something that I can upload directly to artifactory, so I tried to start with the first plugin that was failing, and build that and see if mvn deploy would deploy it to artifactory. (First, question... is that a reasonable idea?) That would be: http://repo1.maven.org/maven2 However it is not going to be fun to find each thing you need and upload it to your internal repository manager. (I work with a company that doe
Re: Bootstraping a repository manager
I have a policy of explicitly stating the version that I want to build with and to deploy to production. It may or may not be the latest but it is one that the team agrees is the "best" version to run (it supports the methods that we use, we have used it long enough to be comfortable, the vendor or producing organization has no reported critical deficiencies, etc.) If someone wants to propose a later version, that has have an impact analysis and then be agreed upon and approved by me. I am a pretty reasonable rubber stamp when the previous steps have been done. I want reproducible builds that deploy as secure and robust applications. This does not always require the latest version and I am not sure that leaving that decision up to Maven is a good idea. Ron On 19/05/2011 12:40 PM, Heck, Gus (Patrick) wrote: Hmm, how would I go about convincing maven-plugins such as clean to only use the latest versions (in some cases it just seems to be a dep on the pom file, not any actual jars). Don't know where I would find the handle for this in maven, unless one does it by editing the poms that are wandering off into past versions I suppose. That of course introduces some chance of running into any cases where there's a backwards compatibility problem, but keeping the list of dependencies shorter might be worth that risk. -Gus Patrick Heck ||| Principal Software Engineer Aspen Technology, Inc. ||| +1 781-221-6483 ||| www.aspentech.com -Original Message- From: Ron Wheeler [mailto:rwhee...@artifact-software.com] Sent: Thursday, May 19, 2011 12:30 PM To: users@maven.apache.org Subject: Re: Bootstraping a repository manager On 19/05/2011 11:26 AM, Heck, Gus (Patrick) wrote: Oh yes, and I'll be doing some new searches now that I have your name as a search key since you say you've discussed this a lot. :) Searches on settings.xml, or repository or most of the keywords I can think of for this just have way too many hits. The help is definitely appreciated, sorry if I sounded a bit sour. On the plus side, I'm now running down dependencies now that I've figured out where to find things in "central" (and where central is). Interestingly, a simple maven-clean from maven 3.0.2 pulls dependencies for multiple versions of maven and maven-parent. Seems a little funky to me, but I guess it's probably a product of plug-ins having a separate release schedule. This sounds similar to the problem with third party libraries where you end up with a lot of different versions of log4j, for example, and you just pick one - the highest, provide it in a shared libray and then use exclusions to prevent other jars from loading old versions. I believe in strict and explicit control over all code (including the 3rd party stuff) that runs in production and went to the trouble of making this happen for our development team. Ron -Gus Patrick Heck ||| Principal Software Engineer Aspen Technology, Inc. ||| +1 781-221-6483 ||| www.aspentech.com -Original Message- From: Ron Wheeler [mailto:rwhee...@artifact-software.com] Sent: Wednesday, May 18, 2011 10:22 PM To: users@maven.apache.org Subject: Re: Bootstraping a repository manager The other nice thing about Maven is that no matter what one is building or how unusual one thinks their environment is, someone has already built something similar and someone has already worked out a best practice in a similar environment. It is also supported by a great forum where you get advice that you could not find or afford and it is free here. Ron On 18/05/2011 9:32 PM, Wendy Smoak wrote: On Wed, May 18, 2011 at 7:56 PM, Heck, Gus (Patrick) wrote: As I explained in another response, I want the software to tell me when I've got enough stuff to build (much like test driven development), rather than trusting I covered everything post-hoc. As for doing it with ant projects, I've certainly been there. At one job I wrote a custom ant task that looked at /lib, a properties file and a directory called /licenses. If the jar in lib didn't have an entry in the properties file, or the value for that jar in the properties file didn't match the name of a file in /licenses, the build failed. Not really that hard to implement. As you say, the big time sink is in hunting down the licenses, but in the current job I have to do that anyway to include a copy of the license with the approval form for the lawyers... nothing maven can do about that. The only irritation is that some basic artifacts seem to be hard to find and instead of an answer such as "the artifacts you need can be found here" I get a chorus of "don't do that" Hm? I already told you where the things are. In the central repository. (And you need both the jar and the pom for it to work.) Typically the only organizations that can afford to be this paranoid are really big ones. I work with one o
Re: Bootstraping a repository manager
> what it takes from just plexus and maven to do a 'mvn clean'. I think > there's got to be something wrong when you need 38 poms and 4 jars. What > could maven legitimately want with the pom file for 5 different versions > of plexus-utils? This happens due to transitive dependencies which are released depending on a particular version of an artifact, which itself depends on various dependencies of various versions, so on and so on. And then new versions of artifacts get released which depend on new versions of the same set of artifacts (but backwards-compatibility is not guaranteed nor assumed). You could clean things up a bit through the generous use of tags and by declaring all the plugins you use in your builds and locking-down the versions of those plugins with [1.2.3] style version tags. Relatively few people (that are vocal or are active Maven developers/contributors) seem to care about this or see this as a problem. As such, this is a situation that is unlikely to change. Wayne - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
RE: Bootstraping a repository manager
Actually, I'm finding that it's not the jars that are a pain... here's what it takes from just plexus and maven to do a 'mvn clean'. I think there's got to be something wrong when you need 38 poms and 4 jars. What could maven legitimately want with the pom file for 5 different versions of plexus-utils? bash-4.1$ find -name *.jar ./org/apache/maven/plugins/maven-clean-plugin/2.3/maven-clean-plugin-2.3 .jar ./org/apache/maven/shared/file-management/1.2.1/file-management-1.2.1.ja r ./org/apache/maven/shared/maven-shared-io/1.1/maven-shared-io-1.1.jar ./org/codehaus/plexus/plexus-utils/1.5.6/plexus-utils-1.5.6.jar bash-4.1$ find -name *.pom ./org/apache/apache/3/apache-3.pom ./org/apache/apache/4/apache-4.pom ./org/apache/maven/maven/2.0.2/maven-2.0.2.pom ./org/apache/maven/maven/2.0.6/maven-2.0.6.pom ./org/apache/maven/maven-artifact/2.0.2/maven-artifact-2.0.2.pom ./org/apache/maven/maven-artifact/2.0.6/maven-artifact-2.0.6.pom ./org/apache/maven/maven-artifact-manager/2.0.2/maven-artifact-manager-2 .0.2.pom ./org/apache/maven/maven-artifact-manager/2.0.6/maven-artifact-manager-2 .0.6.pom ./org/apache/maven/maven-model/2.0.6/maven-model-2.0.6.pom ./org/apache/maven/maven-parent/5/maven-parent-5.pom ./org/apache/maven/maven-parent/7/maven-parent-7.pom ./org/apache/maven/maven-parent/9/maven-parent-9.pom ./org/apache/maven/maven-plugin-api/2.0.6/maven-plugin-api-2.0.6.pom ./org/apache/maven/maven-plugin-registry/2.0.6/maven-plugin-registry-2.0 .6.pom ./org/apache/maven/maven-profile/2.0.6/maven-profile-2.0.6.pom ./org/apache/maven/maven-project/2.0.6/maven-project-2.0.6.pom ./org/apache/maven/maven-repository-metadata/2.0.2/maven-repository-meta data-2.0.2.pom ./org/apache/maven/maven-repository-metadata/2.0.6/maven-repository-meta data-2.0.6.pom ./org/apache/maven/maven-settings/2.0.6/maven-settings-2.0.6.pom ./org/apache/maven/plugins/maven-clean-plugin/2.3/maven-clean-plugin-2.3 .pom ./org/apache/maven/plugins/maven-plugins/12/maven-plugins-12.pom ./org/apache/maven/shared/file-management/1.2.1/file-management-1.2.1.po m ./org/apache/maven/shared/maven-shared-components/10/maven-shared-compon ents-10.pom ./org/apache/maven/shared/maven-shared-components/8/maven-shared-compone nts-8.pom ./org/apache/maven/shared/maven-shared-io/1.1/maven-shared-io-1.1.pom ./org/apache/maven/wagon/wagon/1.0-alpha-6/wagon-1.0-alpha-6.pom ./org/apache/maven/wagon/wagon-provider-api/1.0-alpha-6/wagon-provider-a pi-1.0-alpha-6.pom ./org/codehaus/plexus/plexus/1.0.11/plexus-1.0.11.pom ./org/codehaus/plexus/plexus/1.0.12/plexus-1.0.12.pom ./org/codehaus/plexus/plexus/1.0.4/plexus-1.0.4.pom ./org/codehaus/plexus/plexus-container-default/1.0-alpha-9/plexus-contai ner-default-1.0-alpha-9.pom ./org/codehaus/plexus/plexus-container-default/1.0-alpha-9-stable-1/plex us-container-default-1.0-alpha-9-stable-1.pom ./org/codehaus/plexus/plexus-containers/1.0.3/plexus-containers-1.0.3.po m ./org/codehaus/plexus/plexus-utils/1.0.4/plexus-utils-1.0.4.pom ./org/codehaus/plexus/plexus-utils/1.1/plexus-utils-1.1.pom ./org/codehaus/plexus/plexus-utils/1.4.1/plexus-utils-1.4.1.pom ./org/codehaus/plexus/plexus-utils/1.4.6/plexus-utils-1.4.6.pom ./org/codehaus/plexus/plexus-utils/1.5.6/plexus-utils-1.5.6.pom Now off to run-down junit and classworlds... Patrick Heck ||| Principal Software Engineer Aspen Technology, Inc. ||| +1 781-221-6483 ||| www.aspentech.com -Original Message- From: Gordon Cody [mailto:gc...@zafinlabs.com] Sent: Thursday, May 19, 2011 1:35 PM To: Maven Users List Subject: Re: Bootstraping a repository manager Hi Gus We seriously considered trying to implement the type of controls you talked about, but there are literally thousands of files you would have to manually install once you figure out what they all are. What we ended up doing was using the mirroOf feature in settings.xml and allow artifactory to download what it needed and keep it forever. We ran each of the standard mvn commands (and some others) that developers use, in order to populate our artifactory. Once we had all the necessary 3rd party bits & pieces we required, we configured artifactorys 'global offline' mode. Subsequently new jars or new versions of jars are only added manually after approval. Prior to working like this we got burned (once) when an artifact we required (a dependency of a dependency) was missing when we were trying to do a release. That was a bad day we will not ever repeat. -Gord - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email, and any attachments the
RE: Bootstraping a repository manager
That's good news :) Patrick Heck ||| Principal Software Engineer Aspen Technology, Inc. ||| +1 781-221-6483 ||| www.aspentech.com -Original Message- From: Ansgar Konermann [mailto:ansgar.konerm...@googlemail.com] Sent: Thursday, May 19, 2011 1:52 PM To: users@maven.apache.org Subject: Re: Bootstraping a repository manager Am 19.05.2011 01:56, schrieb Heck, Gus (Patrick): > in the current job I have to do that anyway to include a copy of the license > with the approval form for the lawyers... nothing maven can do about that. Not yet. But there's something in the Maven development pipeline IIRC. Maybe one of the Maven gurus can light this up a bit, but as far as I understood the rumors, maven is going to have some kind of license management features in the not so distant future. Best regards Ansgar - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email, and any attachments thereto, is strictly prohibited. If you receive this email in error please immediately notify the sender and permanently delete the original copy and any copy of any e-mail, and any printout thereof.
Re: Bootstraping a repository manager
Am 19.05.2011 01:56, schrieb Heck, Gus (Patrick): > in the current job I have to do that anyway to include a copy of the license > with the approval form for the lawyers... nothing maven can do about that. Not yet. But there's something in the Maven development pipeline IIRC. Maybe one of the Maven gurus can light this up a bit, but as far as I understood the rumors, maven is going to have some kind of license management features in the not so distant future. Best regards Ansgar - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
RE: Bootstraping a repository manager
Not familiar with Archiva but I do know that the Nexus Pro product has a feature set geared at this use case. Don't recall if it is just the use of meta-data or what but you can proxy stuff openly but then *promote* things to a Definitive Software Library status and run authoritative builds against that. Curt Yanko | Continuous Integration Services | UnitedHealth Group IT Making IT Happen, one build at a time, 600 times a day > -Original Message- > From: Heck, Gus (Patrick) [mailto:gus.h...@aspentech.com] > Sent: Thursday, May 19, 2011 12:49 PM > To: users@maven.apache.org > Subject: RE: Bootstraping a repository manager > > Interesting however that nobody has yet mentioned the apache > home grown archiva... > > -Gus > > Patrick Heck > ||| Principal Software Engineer > Aspen Technology, Inc. ||| +1 781-221-6483 ||| www.aspentech.com > > -Original Message- > From: Ron Wheeler [mailto:rwhee...@artifact-software.com] > Sent: Thursday, May 19, 2011 12:03 PM > To: users@maven.apache.org > Subject: Re: Bootstraping a repository manager > > I am just using the Free version and although talking about > competing repo managers is discouraged in the form, I am a > big fan of the Nexus approach and would heartily recommend it > to anyone who is using Maven > > Ron > > On 19/05/2011 10:35 AM, Brian Fox wrote: > > It's also worth mentioning that Nexus Professional's Procurement > > feature is built for exactly the use case you have. It's > meant to have > > a hard firewall like separation between internal and external > > artifacts and rules that allow you to approve whitelist/blacklist > > style, or by wildcard or other runtime evaluation to define > what can > > be used internally. > > > > On Thu, May 19, 2011 at 9:24 AM, Ron Wheeler > > wrote: > >> If you followed my directions or Wendy's, you will have a repo that > contains > >> only the dependencies (artifact and exact versions) that your > software > >> requires and nothing else. > >> I don't know Artifactory but I do use Nexus so you will have to > translate my > >> instructions. > >> > >> In Nexus, you can see each of the artifacts that you have requested > and > >> extract the license information. > >> If you start at the top of the list and work your way down > in order, > you > >> will not miss anything unless you skip one by mistake. Keep a > spreadsheet as > >> a checklist if you worry about that. > >> > >> A bit of a PITA but not as bad as you are envisioning. > >> > >> You will also see where you are using more than one version of an > artifact > >> so you can limit the dependencies that you may want to clean up. > >> > >> Ron > >> > >> > >> > >> On 19/05/2011 3:23 AM, Nicola Musatti wrote: > >>> Once you had Maven collect all the dependencies for you > you can use > >>> something like > >>> > >>> find repository_storage_root -name '*.jar' > >>> > >>> if you are on Linux/UNIX or Windows's search tool to obtain a list > of all > >>> your dependencies. With that you can use something like > >>> http://mvnrepository.com/ to gather information about each jar and > make your > >>> decisions. This would take you a lot less than your approach. > >>> > >>> Cheers, > >>> Nicola Musatti > >>> > >>> Heck, Gus (Patrick) wrote: > >>>> Hi Wendy, > >>>> > >>>> I don't have a requirement to build from source, but in > the case of > the > >>>> maven plugins, nowhere on the web seems to point to a > place where I > can > >>>> download the finished product (aside from letting maven find it, > and who > >>>> knows what other dependencies). I am adverse to letting > maven just > pull > >>>> things because then I have a very long list of items to check and > verify > >>>> arranged in a tree, and it's all too easy to miss a subtree and > think > >>>> I've got everything covered. Basically the problem is I make a > really > >>>> crappy computer when it comes to tree-traversals and not > losing my > place > >>>> when someone comes over to chat, I go to lunch, or I have to go > home at > >>>> the end of the day. :-) I don't want to trust that I > won't make a
Re: Bootstraping a repository manager
Hi Gus We seriously considered trying to implement the type of controls you talked about, but there are literally thousands of files you would have to manually install once you figure out what they all are. What we ended up doing was using the mirroOf feature in settings.xml and allow artifactory to download what it needed and keep it forever. We ran each of the standard mvn commands (and some others) that developers use, in order to populate our artifactory. Once we had all the necessary 3rd party bits & pieces we required, we configured artifactorys 'global offline' mode. Subsequently new jars or new versions of jars are only added manually after approval. Prior to working like this we got burned (once) when an artifact we required (a dependency of a dependency) was missing when we were trying to do a release. That was a bad day we will not ever repeat. -Gord - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Bootstraping a repository manager
On Thu, May 19, 2011 at 12:48 PM, Heck, Gus (Patrick) wrote: > Interesting however that nobody has yet mentioned the apache home grown > archiva... Most open source projects don't do a lot of advertising. You know where to find us if you want to give it a try. ;) -- Wendy - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Bootstraping a repository manager
On Thu, May 19, 2011 at 12:40 PM, Heck, Gus (Patrick) wrote: > Hmm, how would I go about convincing maven-plugins such as clean to only > use the latest versions (in some cases it just seems to be a dep on the > pom file, not any actual jars). Don't know where I would find the handle > for this in maven, unless one does it by editing the poms that are > wandering off into past versions I suppose. No, don't mess with the released poms for the plugins. Ron was talking about controlling what dependencies are used in your own projects. There are two separate things here: - the plugins and other libraries Maven uses to build your code - the actual artifacts the build produces that contain your compiled code, config files, etc. Are you really that concerned about how many parent poms Maven needs to download to do its job? For Maven plugins with an internal (disconnected) repository, you will go mad trying to download-and-upload each thing individually. There will be hundreds of Maven/Plexus/Commons/Etc jars to find. Just let Maven gather it all up, give it a once over, and put it in your repo. Or is the real issue that you want to make sure you know exactly what your code depends on and what you are putting in production or delivering to customers? In my experience, that's the bigger issue. -- Wendy - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
RE: Bootstraping a repository manager
Interesting however that nobody has yet mentioned the apache home grown archiva... -Gus Patrick Heck ||| Principal Software Engineer Aspen Technology, Inc. ||| +1 781-221-6483 ||| www.aspentech.com -Original Message- From: Ron Wheeler [mailto:rwhee...@artifact-software.com] Sent: Thursday, May 19, 2011 12:03 PM To: users@maven.apache.org Subject: Re: Bootstraping a repository manager I am just using the Free version and although talking about competing repo managers is discouraged in the form, I am a big fan of the Nexus approach and would heartily recommend it to anyone who is using Maven Ron On 19/05/2011 10:35 AM, Brian Fox wrote: > It's also worth mentioning that Nexus Professional's Procurement > feature is built for exactly the use case you have. It's meant to have > a hard firewall like separation between internal and external > artifacts and rules that allow you to approve whitelist/blacklist > style, or by wildcard or other runtime evaluation to define what can > be used internally. > > On Thu, May 19, 2011 at 9:24 AM, Ron Wheeler > wrote: >> If you followed my directions or Wendy's, you will have a repo that contains >> only the dependencies (artifact and exact versions) that your software >> requires and nothing else. >> I don't know Artifactory but I do use Nexus so you will have to translate my >> instructions. >> >> In Nexus, you can see each of the artifacts that you have requested and >> extract the license information. >> If you start at the top of the list and work your way down in order, you >> will not miss anything unless you skip one by mistake. Keep a spreadsheet as >> a checklist if you worry about that. >> >> A bit of a PITA but not as bad as you are envisioning. >> >> You will also see where you are using more than one version of an artifact >> so you can limit the dependencies that you may want to clean up. >> >> Ron >> >> >> >> On 19/05/2011 3:23 AM, Nicola Musatti wrote: >>> Once you had Maven collect all the dependencies for you you can use >>> something like >>> >>> find repository_storage_root -name '*.jar' >>> >>> if you are on Linux/UNIX or Windows's search tool to obtain a list of all >>> your dependencies. With that you can use something like >>> http://mvnrepository.com/ to gather information about each jar and make your >>> decisions. This would take you a lot less than your approach. >>> >>> Cheers, >>> Nicola Musatti >>> >>> Heck, Gus (Patrick) wrote: >>>> Hi Wendy, >>>> >>>> I don't have a requirement to build from source, but in the case of the >>>> maven plugins, nowhere on the web seems to point to a place where I can >>>> download the finished product (aside from letting maven find it, and who >>>> knows what other dependencies). I am adverse to letting maven just pull >>>> things because then I have a very long list of items to check and verify >>>> arranged in a tree, and it's all too easy to miss a subtree and think >>>> I've got everything covered. Basically the problem is I make a really >>>> crappy computer when it comes to tree-traversals and not losing my place >>>> when someone comes over to chat, I go to lunch, or I have to go home at >>>> the end of the day. :-) I don't want to trust that I won't make a >>>> mistake, I'd rather take a bit more time and let the build system ensure >>>> that I didn't miss anything. >>>> >>>> If I act as gatekeeper, I know that the project won't build properly >>>> until I've applied each dependency required to build the project. (and >>>> then same for test... etc). This shouldn't be any more burdensome than >>>> finding the transitive dependencies for a good sized ant project, which >>>> I've done many times before, except that maven plug-in folks seem to >>>> squirrel their stuff away where only maven can find it, so I'm having >>>> trouble getting maven working to start with. >>>> >>>> -Gus >>>> >>>> -Original Message- >>>> From: Wendy Smoak [mailto:wsm...@gmail.com] >>>> Sent: Wednesday, May 18, 2011 6:35 PM >>>> To: Maven Users List >>>> Subject: Re: Bootstraping a repository manager >>>> >>>> On Wed, May 18, 2011 at 5:21 PM, Heck, Gus (Patrick) >>>> wrote: >>>>> I can't seem to find a
RE: Bootstraping a repository manager
Hmm, how would I go about convincing maven-plugins such as clean to only use the latest versions (in some cases it just seems to be a dep on the pom file, not any actual jars). Don't know where I would find the handle for this in maven, unless one does it by editing the poms that are wandering off into past versions I suppose. That of course introduces some chance of running into any cases where there's a backwards compatibility problem, but keeping the list of dependencies shorter might be worth that risk. -Gus Patrick Heck ||| Principal Software Engineer Aspen Technology, Inc. ||| +1 781-221-6483 ||| www.aspentech.com -Original Message- From: Ron Wheeler [mailto:rwhee...@artifact-software.com] Sent: Thursday, May 19, 2011 12:30 PM To: users@maven.apache.org Subject: Re: Bootstraping a repository manager On 19/05/2011 11:26 AM, Heck, Gus (Patrick) wrote: > Oh yes, and I'll be doing some new searches now that I have your name as > a search key since you say you've discussed this a lot. :) Searches on > settings.xml, or repository or most of the keywords I can think of for > this just have way too many hits. The help is definitely appreciated, > sorry if I sounded a bit sour. > > On the plus side, I'm now running down dependencies now that I've > figured out where to find things in "central" (and where central is). > > Interestingly, a simple maven-clean from maven 3.0.2 pulls dependencies > for multiple versions of maven and maven-parent. Seems a little funky to > me, but I guess it's probably a product of plug-ins having a separate > release schedule. This sounds similar to the problem with third party libraries where you end up with a lot of different versions of log4j, for example, and you just pick one - the highest, provide it in a shared libray and then use exclusions to prevent other jars from loading old versions. I believe in strict and explicit control over all code (including the 3rd party stuff) that runs in production and went to the trouble of making this happen for our development team. Ron > -Gus > > Patrick Heck > ||| Principal Software Engineer > Aspen Technology, Inc. ||| +1 781-221-6483 ||| www.aspentech.com > > -Original Message- > From: Ron Wheeler [mailto:rwhee...@artifact-software.com] > Sent: Wednesday, May 18, 2011 10:22 PM > To: users@maven.apache.org > Subject: Re: Bootstraping a repository manager > > The other nice thing about Maven is that no matter what one is building > or how unusual one thinks their environment is, someone has already > built something similar and someone has already worked out a best > practice in a similar environment. > > It is also supported by a great forum where you get advice that you > could not find or afford and it is free here. > > Ron > > On 18/05/2011 9:32 PM, Wendy Smoak wrote: >> On Wed, May 18, 2011 at 7:56 PM, Heck, Gus (Patrick) >>wrote: >>> As I explained in another response, I want the software to tell me > when I've got enough stuff to build (much like test driven development), > rather than trusting I covered everything post-hoc. As for doing it with > ant projects, I've certainly been there. At one job I wrote a custom ant > task that looked at /lib, a properties file and a directory called > /licenses. If the jar in lib didn't have an entry in the properties > file, or the value for that jar in the properties file didn't match the > name of a file in /licenses, the build failed. Not really that hard to > implement. As you say, the big time sink is in hunting down the > licenses, but in the current job I have to do that anyway to include a > copy of the license with the approval form for the lawyers... nothing > maven can do about that. The only irritation is that some basic > artifacts seem to be hard to find and instead of an answer such as "the > artifacts you need can be found here" I get a chorus of "don't do that" >> Hm? I already told you where the things are. In the central >> repository. (And you need both the jar and the pom for it to work.) >> >> Typically the only organizations that can afford to be this paranoid >> are really big ones. I work with one of those. The internal Maven >> repo does not talk to the internet. Instead, developers fill out a >> web form to request a new artifact to be uploaded, someone approves it >> if it's in the standards, or sends them off to the reference >> architecture team to explain why they need it, and then someone else >> presses the buttons in Archiva to upload it. >> >> Plugins are a special case. Those we handle within the build team, as >> they typically want all kinds of dependencies. So for thos
Re: Bootstraping a repository manager
On 19/05/2011 11:26 AM, Heck, Gus (Patrick) wrote: Oh yes, and I'll be doing some new searches now that I have your name as a search key since you say you've discussed this a lot. :) Searches on settings.xml, or repository or most of the keywords I can think of for this just have way too many hits. The help is definitely appreciated, sorry if I sounded a bit sour. On the plus side, I'm now running down dependencies now that I've figured out where to find things in "central" (and where central is). Interestingly, a simple maven-clean from maven 3.0.2 pulls dependencies for multiple versions of maven and maven-parent. Seems a little funky to me, but I guess it's probably a product of plug-ins having a separate release schedule. This sounds similar to the problem with third party libraries where you end up with a lot of different versions of log4j, for example, and you just pick one - the highest, provide it in a shared libray and then use exclusions to prevent other jars from loading old versions. I believe in strict and explicit control over all code (including the 3rd party stuff) that runs in production and went to the trouble of making this happen for our development team. Ron -Gus Patrick Heck ||| Principal Software Engineer Aspen Technology, Inc. ||| +1 781-221-6483 ||| www.aspentech.com -Original Message- From: Ron Wheeler [mailto:rwhee...@artifact-software.com] Sent: Wednesday, May 18, 2011 10:22 PM To: users@maven.apache.org Subject: Re: Bootstraping a repository manager The other nice thing about Maven is that no matter what one is building or how unusual one thinks their environment is, someone has already built something similar and someone has already worked out a best practice in a similar environment. It is also supported by a great forum where you get advice that you could not find or afford and it is free here. Ron On 18/05/2011 9:32 PM, Wendy Smoak wrote: On Wed, May 18, 2011 at 7:56 PM, Heck, Gus (Patrick) wrote: As I explained in another response, I want the software to tell me when I've got enough stuff to build (much like test driven development), rather than trusting I covered everything post-hoc. As for doing it with ant projects, I've certainly been there. At one job I wrote a custom ant task that looked at /lib, a properties file and a directory called /licenses. If the jar in lib didn't have an entry in the properties file, or the value for that jar in the properties file didn't match the name of a file in /licenses, the build failed. Not really that hard to implement. As you say, the big time sink is in hunting down the licenses, but in the current job I have to do that anyway to include a copy of the license with the approval form for the lawyers... nothing maven can do about that. The only irritation is that some basic artifacts seem to be hard to find and instead of an answer such as "the artifacts you need can be found here" I get a chorus of "don't do that" Hm? I already told you where the things are. In the central repository. (And you need both the jar and the pom for it to work.) Typically the only organizations that can afford to be this paranoid are really big ones. I work with one of those. The internal Maven repo does not talk to the internet. Instead, developers fill out a web form to request a new artifact to be uploaded, someone approves it if it's in the standards, or sends them off to the reference architecture team to explain why they need it, and then someone else presses the buttons in Archiva to upload it. Plugins are a special case. Those we handle within the build team, as they typically want all kinds of dependencies. So for those we connect to the internet and use the Maven Assembly plugin to create a repository bundle by listing the plugin as a dependency. Maven goes off and gets everything that plugin needs, and the whole bundle is uploaded. Those artifacts generally aren't going to get into your end product, they are just used during the build. By the way, I am a bit irritated that maven put's the repository information out on the developer's settings.xml. I'd much rather be able to manage that centrally, and not leave it as something that they could screw up, or forget to do after their hard drive fails and the set things up a second time. Is there a way to lock things into a single repository without settings.xml? That's where it's done, with a mirror in settings.xml. You can create your own Maven distribution with the settings.xml you want them to use packaged inside it (in the /conf directory.) I assume if you're this strict about dependencies, developers are not downloading random software from the internet and installing it on their machines, so provide the Maven distro you want them to use. - To unsubs
Re: Bootstraping a repository manager
I am just using the Free version and although talking about competing repo managers is discouraged in the form, I am a big fan of the Nexus approach and would heartily recommend it to anyone who is using Maven Ron On 19/05/2011 10:35 AM, Brian Fox wrote: It's also worth mentioning that Nexus Professional's Procurement feature is built for exactly the use case you have. It's meant to have a hard firewall like separation between internal and external artifacts and rules that allow you to approve whitelist/blacklist style, or by wildcard or other runtime evaluation to define what can be used internally. On Thu, May 19, 2011 at 9:24 AM, Ron Wheeler wrote: If you followed my directions or Wendy's, you will have a repo that contains only the dependencies (artifact and exact versions) that your software requires and nothing else. I don't know Artifactory but I do use Nexus so you will have to translate my instructions. In Nexus, you can see each of the artifacts that you have requested and extract the license information. If you start at the top of the list and work your way down in order, you will not miss anything unless you skip one by mistake. Keep a spreadsheet as a checklist if you worry about that. A bit of a PITA but not as bad as you are envisioning. You will also see where you are using more than one version of an artifact so you can limit the dependencies that you may want to clean up. Ron On 19/05/2011 3:23 AM, Nicola Musatti wrote: Once you had Maven collect all the dependencies for you you can use something like find repository_storage_root -name '*.jar' if you are on Linux/UNIX or Windows's search tool to obtain a list of all your dependencies. With that you can use something like http://mvnrepository.com/ to gather information about each jar and make your decisions. This would take you a lot less than your approach. Cheers, Nicola Musatti Heck, Gus (Patrick) wrote: Hi Wendy, I don't have a requirement to build from source, but in the case of the maven plugins, nowhere on the web seems to point to a place where I can download the finished product (aside from letting maven find it, and who knows what other dependencies). I am adverse to letting maven just pull things because then I have a very long list of items to check and verify arranged in a tree, and it's all too easy to miss a subtree and think I've got everything covered. Basically the problem is I make a really crappy computer when it comes to tree-traversals and not losing my place when someone comes over to chat, I go to lunch, or I have to go home at the end of the day. :-) I don't want to trust that I won't make a mistake, I'd rather take a bit more time and let the build system ensure that I didn't miss anything. If I act as gatekeeper, I know that the project won't build properly until I've applied each dependency required to build the project. (and then same for test... etc). This shouldn't be any more burdensome than finding the transitive dependencies for a good sized ant project, which I've done many times before, except that maven plug-in folks seem to squirrel their stuff away where only maven can find it, so I'm having trouble getting maven working to start with. -Gus -Original Message- From: Wendy Smoak [mailto:wsm...@gmail.com] Sent: Wednesday, May 18, 2011 6:35 PM To: Maven Users List Subject: Re: Bootstraping a repository manager On Wed, May 18, 2011 at 5:21 PM, Heck, Gus (Patrick) wrote: I can't seem to find a place to download something that I can upload directly to artifactory, so I tried to start with the first plugin that was failing, and build that and see if mvn deploy would deploy it to artifactory. (First, question... is that a reasonable idea?) That would be: http://repo1.maven.org/maven2 However it is not going to be fun to find each thing you need and upload it to your internal repository manager. (I work with a company that does it this way.) Especially for plugins. Is there any middle ground? Let the repository manager proxy into a segregated directory, vet everything in there and then move it to the pristine internal repo? Unfortunately, as soon as I did svn checkout http://svn.apache.org/repos/asf/maven/plugins/tags/maven-clean-plugin-2. 4.1 maven-clean-plugin cd maven-clean-plugin mvn deploy Hmm... first, do you *really* have a requirement to build everything from source? Chi riceve il presente messaggio e' tenuto a verificare se lo stesso non gli sia pervenuto per errore. In tal caso e' pregato di avvisare immediatamente il mittente e, tenuto conto delle responsabilita' connesse all'indebito utilizzo e/o divulgazione del messaggio e/o delle informazioni in esso contenute, voglia cancellare l'originale e distruggere le varie copie o stampe. The receiver of this message is required to check if he/she has receive
RE: Bootstraping a repository manager
Oh yes, and I'll be doing some new searches now that I have your name as a search key since you say you've discussed this a lot. :) Searches on settings.xml, or repository or most of the keywords I can think of for this just have way too many hits. The help is definitely appreciated, sorry if I sounded a bit sour. On the plus side, I'm now running down dependencies now that I've figured out where to find things in "central" (and where central is). Interestingly, a simple maven-clean from maven 3.0.2 pulls dependencies for multiple versions of maven and maven-parent. Seems a little funky to me, but I guess it's probably a product of plug-ins having a separate release schedule. -Gus Patrick Heck ||| Principal Software Engineer Aspen Technology, Inc. ||| +1 781-221-6483 ||| www.aspentech.com -Original Message- From: Ron Wheeler [mailto:rwhee...@artifact-software.com] Sent: Wednesday, May 18, 2011 10:22 PM To: users@maven.apache.org Subject: Re: Bootstraping a repository manager The other nice thing about Maven is that no matter what one is building or how unusual one thinks their environment is, someone has already built something similar and someone has already worked out a best practice in a similar environment. It is also supported by a great forum where you get advice that you could not find or afford and it is free here. Ron On 18/05/2011 9:32 PM, Wendy Smoak wrote: > On Wed, May 18, 2011 at 7:56 PM, Heck, Gus (Patrick) > wrote: >> As I explained in another response, I want the software to tell me when I've got enough stuff to build (much like test driven development), rather than trusting I covered everything post-hoc. As for doing it with ant projects, I've certainly been there. At one job I wrote a custom ant task that looked at /lib, a properties file and a directory called /licenses. If the jar in lib didn't have an entry in the properties file, or the value for that jar in the properties file didn't match the name of a file in /licenses, the build failed. Not really that hard to implement. As you say, the big time sink is in hunting down the licenses, but in the current job I have to do that anyway to include a copy of the license with the approval form for the lawyers... nothing maven can do about that. The only irritation is that some basic artifacts seem to be hard to find and instead of an answer such as "the artifacts you need can be found here" I get a chorus of "don't do that" > Hm? I already told you where the things are. In the central > repository. (And you need both the jar and the pom for it to work.) > > Typically the only organizations that can afford to be this paranoid > are really big ones. I work with one of those. The internal Maven > repo does not talk to the internet. Instead, developers fill out a > web form to request a new artifact to be uploaded, someone approves it > if it's in the standards, or sends them off to the reference > architecture team to explain why they need it, and then someone else > presses the buttons in Archiva to upload it. > > Plugins are a special case. Those we handle within the build team, as > they typically want all kinds of dependencies. So for those we > connect to the internet and use the Maven Assembly plugin to create a > repository bundle by listing the plugin as a dependency. Maven goes > off and gets everything that plugin needs, and the whole bundle is > uploaded. Those artifacts generally aren't going to get into your end > product, they are just used during the build. > >> By the way, I am a bit irritated that maven put's the repository information out on the developer's settings.xml. I'd much rather be able to manage that centrally, and not leave it as something that they could screw up, or forget to do after their hard drive fails and the set things up a second time. Is there a way to lock things into a single repository without settings.xml? > That's where it's done, with a mirror in settings.xml. You can create > your own Maven distribution with the settings.xml you want them to use > packaged inside it (in the /conf directory.) > > I assume if you're this strict about dependencies, developers are not > downloading random software from the internet and installing it on > their machines, so provide the Maven distro you want them to use. > - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination
RE: Bootstraping a repository manager
Well, the pom already has a element. It seems that if this is used, all one needs is an attribute to indicate that fallback to central or other repositories not explicitly listed is not desired. Perhaps an attribute named exhaustive="true|false"? -Gus -Original Message- From: Wayne Fay [mailto:wayne...@gmail.com] Sent: Thursday, May 19, 2011 11:08 AM To: Maven Users List Subject: Re: Bootstraping a repository manager [-snip-] > I'm just curious why this fairly important bit of configuration is > left out on the desktop where it is hard to manage. What alternative would you propose?? Feel free to design and implement such a system, and then donate the changed source code back to the Maven project for possible future inclusion in a future release. Personally, I have ZERO interest in such a feature and do not anticipate working on it. Wayne This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email, and any attachments thereto, is strictly prohibited. If you receive this email in error please immediately notify the sender and permanently delete the original copy and any copy of any e-mail, and any printout thereof.
Re: Bootstraping a repository manager
> As for creating my own maven, sure I can do that if it seems to be > necessary, though forking a project has significant costs, both in the > initial code and in maintenance. No one is suggesting that you "create your own Maven" but rather simply suggesting that it might be a good idea to simply use InstallAnywhere or a similar tool to create an "official" Maven installation for your company that would include, among other things, the settings.xml file or other configuration files that you deem necessary. Then you can distribute this package to everyone who needs to use Maven in your company and it will come "ready to go." > I'd actually been thinking more along the lines of a plug-in that > parses settings.xml and fails out if it doesn't like what it sees. Sounds like a job for the enforcer plugin. > I'm just curious why this fairly important bit of configuration is > left out on the desktop where it is hard to manage. What alternative would you propose?? Feel free to design and implement such a system, and then donate the changed source code back to the Maven project for possible future inclusion in a future release. Personally, I have ZERO interest in such a feature and do not anticipate working on it. Wayne - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
RE: Bootstraping a repository manager
Hi Wendy, "In central" is slightly opaque sometimes. In this case I was deceived by the top level maven directory. I didn't know that this was a very old location, and so I was getting dead-ended for a while until I figured out that everything modern was in org/apache/** As for creating my own maven, sure I can do that if it seems to be necessary, though forking a project has significant costs, both in the initial code and in maintenance. I'd actually been thinking more along the lines of a plug-in that parses settings.xml and fails out if it doesn't like what it sees. I'm just curious why this fairly important bit of configuration is left out on the desktop where it is hard to manage. The size of the companies that are locking things down is probably shrinking over time as IT managers and Execs become more savvy about licensing/IP and more companies have offshore teams. Ours was more permissive in the past, but the lockdown is in progress, and I'm trying to make sure my team can still operate when it happens. -Gus Patrick Heck ||| Principal Software Engineer Aspen Technology, Inc. ||| +1 781-221-6483 ||| www.aspentech.com -Original Message- From: Wendy Smoak [mailto:wsm...@gmail.com] Sent: Wednesday, May 18, 2011 9:33 PM To: Maven Users List Subject: Re: Bootstraping a repository manager On Wed, May 18, 2011 at 7:56 PM, Heck, Gus (Patrick) wrote: > As I explained in another response, I want the software to tell me when I've got enough stuff to build (much like test driven development), rather than trusting I covered everything post-hoc. As for doing it with ant projects, I've certainly been there. At one job I wrote a custom ant task that looked at /lib, a properties file and a directory called /licenses. If the jar in lib didn't have an entry in the properties file, or the value for that jar in the properties file didn't match the name of a file in /licenses, the build failed. Not really that hard to implement. As you say, the big time sink is in hunting down the licenses, but in the current job I have to do that anyway to include a copy of the license with the approval form for the lawyers... nothing maven can do about that. The only irritation is that some basic artifacts seem to be hard to find and instead of an answer such as "the artifacts you need can be found here" I get a chorus of "don't do that" Hm? I already told you where the things are. In the central repository. (And you need both the jar and the pom for it to work.) Typically the only organizations that can afford to be this paranoid are really big ones. I work with one of those. The internal Maven repo does not talk to the internet. Instead, developers fill out a web form to request a new artifact to be uploaded, someone approves it if it's in the standards, or sends them off to the reference architecture team to explain why they need it, and then someone else presses the buttons in Archiva to upload it. Plugins are a special case. Those we handle within the build team, as they typically want all kinds of dependencies. So for those we connect to the internet and use the Maven Assembly plugin to create a repository bundle by listing the plugin as a dependency. Maven goes off and gets everything that plugin needs, and the whole bundle is uploaded. Those artifacts generally aren't going to get into your end product, they are just used during the build. > By the way, I am a bit irritated that maven put's the repository information out on the developer's settings.xml. I'd much rather be able to manage that centrally, and not leave it as something that they could screw up, or forget to do after their hard drive fails and the set things up a second time. Is there a way to lock things into a single repository without settings.xml? That's where it's done, with a mirror in settings.xml. You can create your own Maven distribution with the settings.xml you want them to use packaged inside it (in the /conf directory.) I assume if you're this strict about dependencies, developers are not downloading random software from the internet and installing it on their machines, so provide the Maven distro you want them to use. -- Wendy - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email, and any attachments thereto, is strictly prohibited. If you receive this email in error please immediately notify the sender and permanently delete the original copy and any copy of any e
Re: Bootstraping a repository manager
Brian you should have enclosed that within [advertisement][/advertisement] tags or at least pointed out that you work for Sonatype who make Nexus [which is a great piece of software IMHO, and I do not work for Sonatype] -Stephen On 19 May 2011 15:35, Brian Fox wrote: > It's also worth mentioning that Nexus Professional's Procurement > feature is built for exactly the use case you have. It's meant to have > a hard firewall like separation between internal and external > artifacts and rules that allow you to approve whitelist/blacklist > style, or by wildcard or other runtime evaluation to define what can > be used internally. > > On Thu, May 19, 2011 at 9:24 AM, Ron Wheeler > wrote: >> If you followed my directions or Wendy's, you will have a repo that contains >> only the dependencies (artifact and exact versions) that your software >> requires and nothing else. >> I don't know Artifactory but I do use Nexus so you will have to translate my >> instructions. >> >> In Nexus, you can see each of the artifacts that you have requested and >> extract the license information. >> If you start at the top of the list and work your way down in order, you >> will not miss anything unless you skip one by mistake. Keep a spreadsheet as >> a checklist if you worry about that. >> >> A bit of a PITA but not as bad as you are envisioning. >> >> You will also see where you are using more than one version of an artifact >> so you can limit the dependencies that you may want to clean up. >> >> Ron >> >> >> >> On 19/05/2011 3:23 AM, Nicola Musatti wrote: >>> >>> Once you had Maven collect all the dependencies for you you can use >>> something like >>> >>> find repository_storage_root -name '*.jar' >>> >>> if you are on Linux/UNIX or Windows's search tool to obtain a list of all >>> your dependencies. With that you can use something like >>> http://mvnrepository.com/ to gather information about each jar and make your >>> decisions. This would take you a lot less than your approach. >>> >>> Cheers, >>> Nicola Musatti >>> >>> Heck, Gus (Patrick) wrote: >>>> >>>> Hi Wendy, >>>> >>>> I don't have a requirement to build from source, but in the case of the >>>> maven plugins, nowhere on the web seems to point to a place where I can >>>> download the finished product (aside from letting maven find it, and who >>>> knows what other dependencies). I am adverse to letting maven just pull >>>> things because then I have a very long list of items to check and verify >>>> arranged in a tree, and it's all too easy to miss a subtree and think >>>> I've got everything covered. Basically the problem is I make a really >>>> crappy computer when it comes to tree-traversals and not losing my place >>>> when someone comes over to chat, I go to lunch, or I have to go home at >>>> the end of the day. :-) I don't want to trust that I won't make a >>>> mistake, I'd rather take a bit more time and let the build system ensure >>>> that I didn't miss anything. >>>> >>>> If I act as gatekeeper, I know that the project won't build properly >>>> until I've applied each dependency required to build the project. (and >>>> then same for test... etc). This shouldn't be any more burdensome than >>>> finding the transitive dependencies for a good sized ant project, which >>>> I've done many times before, except that maven plug-in folks seem to >>>> squirrel their stuff away where only maven can find it, so I'm having >>>> trouble getting maven working to start with. >>>> >>>> -Gus >>>> >>>> -Original Message- >>>> From: Wendy Smoak [mailto:wsm...@gmail.com] >>>> Sent: Wednesday, May 18, 2011 6:35 PM >>>> To: Maven Users List >>>> Subject: Re: Bootstraping a repository manager >>>> >>>> On Wed, May 18, 2011 at 5:21 PM, Heck, Gus (Patrick) >>>> wrote: >>>>> >>>>> I can't seem to find a place to >>>>> download something that I can upload directly to artifactory, so I >>>> >>>> tried >>>>> >>>>> to start with the first plugin that was failing, and build that and >>>> >>>> see >>>>> >>>>> if mvn deploy would deplo
Re: Bootstraping a repository manager
It's also worth mentioning that Nexus Professional's Procurement feature is built for exactly the use case you have. It's meant to have a hard firewall like separation between internal and external artifacts and rules that allow you to approve whitelist/blacklist style, or by wildcard or other runtime evaluation to define what can be used internally. On Thu, May 19, 2011 at 9:24 AM, Ron Wheeler wrote: > If you followed my directions or Wendy's, you will have a repo that contains > only the dependencies (artifact and exact versions) that your software > requires and nothing else. > I don't know Artifactory but I do use Nexus so you will have to translate my > instructions. > > In Nexus, you can see each of the artifacts that you have requested and > extract the license information. > If you start at the top of the list and work your way down in order, you > will not miss anything unless you skip one by mistake. Keep a spreadsheet as > a checklist if you worry about that. > > A bit of a PITA but not as bad as you are envisioning. > > You will also see where you are using more than one version of an artifact > so you can limit the dependencies that you may want to clean up. > > Ron > > > > On 19/05/2011 3:23 AM, Nicola Musatti wrote: >> >> Once you had Maven collect all the dependencies for you you can use >> something like >> >> find repository_storage_root -name '*.jar' >> >> if you are on Linux/UNIX or Windows's search tool to obtain a list of all >> your dependencies. With that you can use something like >> http://mvnrepository.com/ to gather information about each jar and make your >> decisions. This would take you a lot less than your approach. >> >> Cheers, >> Nicola Musatti >> >> Heck, Gus (Patrick) wrote: >>> >>> Hi Wendy, >>> >>> I don't have a requirement to build from source, but in the case of the >>> maven plugins, nowhere on the web seems to point to a place where I can >>> download the finished product (aside from letting maven find it, and who >>> knows what other dependencies). I am adverse to letting maven just pull >>> things because then I have a very long list of items to check and verify >>> arranged in a tree, and it's all too easy to miss a subtree and think >>> I've got everything covered. Basically the problem is I make a really >>> crappy computer when it comes to tree-traversals and not losing my place >>> when someone comes over to chat, I go to lunch, or I have to go home at >>> the end of the day. :-) I don't want to trust that I won't make a >>> mistake, I'd rather take a bit more time and let the build system ensure >>> that I didn't miss anything. >>> >>> If I act as gatekeeper, I know that the project won't build properly >>> until I've applied each dependency required to build the project. (and >>> then same for test... etc). This shouldn't be any more burdensome than >>> finding the transitive dependencies for a good sized ant project, which >>> I've done many times before, except that maven plug-in folks seem to >>> squirrel their stuff away where only maven can find it, so I'm having >>> trouble getting maven working to start with. >>> >>> -Gus >>> >>> -Original Message- >>> From: Wendy Smoak [mailto:wsm...@gmail.com] >>> Sent: Wednesday, May 18, 2011 6:35 PM >>> To: Maven Users List >>> Subject: Re: Bootstraping a repository manager >>> >>> On Wed, May 18, 2011 at 5:21 PM, Heck, Gus (Patrick) >>> wrote: >>>> >>>> I can't seem to find a place to >>>> download something that I can upload directly to artifactory, so I >>> >>> tried >>>> >>>> to start with the first plugin that was failing, and build that and >>> >>> see >>>> >>>> if mvn deploy would deploy it to artifactory. (First, question... is >>>> that a reasonable idea?) >>> >>> That would be: http://repo1.maven.org/maven2 >>> >>> However it is not going to be fun to find each thing you need and >>> upload it to your internal repository manager. (I work with a company >>> that does it this way.) Especially for plugins. Is there any middle >>> ground? Let the repository manager proxy into a segregated directory, >>> vet everything in there and then move it to the pristine internal >>> repo? >>> >>>>
Re: Bootstraping a repository manager
If you followed my directions or Wendy's, you will have a repo that contains only the dependencies (artifact and exact versions) that your software requires and nothing else. I don't know Artifactory but I do use Nexus so you will have to translate my instructions. In Nexus, you can see each of the artifacts that you have requested and extract the license information. If you start at the top of the list and work your way down in order, you will not miss anything unless you skip one by mistake. Keep a spreadsheet as a checklist if you worry about that. A bit of a PITA but not as bad as you are envisioning. You will also see where you are using more than one version of an artifact so you can limit the dependencies that you may want to clean up. Ron On 19/05/2011 3:23 AM, Nicola Musatti wrote: Once you had Maven collect all the dependencies for you you can use something like find repository_storage_root -name '*.jar' if you are on Linux/UNIX or Windows's search tool to obtain a list of all your dependencies. With that you can use something like http://mvnrepository.com/ to gather information about each jar and make your decisions. This would take you a lot less than your approach. Cheers, Nicola Musatti Heck, Gus (Patrick) wrote: Hi Wendy, I don't have a requirement to build from source, but in the case of the maven plugins, nowhere on the web seems to point to a place where I can download the finished product (aside from letting maven find it, and who knows what other dependencies). I am adverse to letting maven just pull things because then I have a very long list of items to check and verify arranged in a tree, and it's all too easy to miss a subtree and think I've got everything covered. Basically the problem is I make a really crappy computer when it comes to tree-traversals and not losing my place when someone comes over to chat, I go to lunch, or I have to go home at the end of the day. :-) I don't want to trust that I won't make a mistake, I'd rather take a bit more time and let the build system ensure that I didn't miss anything. If I act as gatekeeper, I know that the project won't build properly until I've applied each dependency required to build the project. (and then same for test... etc). This shouldn't be any more burdensome than finding the transitive dependencies for a good sized ant project, which I've done many times before, except that maven plug-in folks seem to squirrel their stuff away where only maven can find it, so I'm having trouble getting maven working to start with. -Gus -Original Message- From: Wendy Smoak [mailto:wsm...@gmail.com] Sent: Wednesday, May 18, 2011 6:35 PM To: Maven Users List Subject: Re: Bootstraping a repository manager On Wed, May 18, 2011 at 5:21 PM, Heck, Gus (Patrick) wrote: I can't seem to find a place to download something that I can upload directly to artifactory, so I tried to start with the first plugin that was failing, and build that and see if mvn deploy would deploy it to artifactory. (First, question... is that a reasonable idea?) That would be: http://repo1.maven.org/maven2 However it is not going to be fun to find each thing you need and upload it to your internal repository manager. (I work with a company that does it this way.) Especially for plugins. Is there any middle ground? Let the repository manager proxy into a segregated directory, vet everything in there and then move it to the pristine internal repo? Unfortunately, as soon as I did svn checkout http://svn.apache.org/repos/asf/maven/plugins/tags/maven-clean-plugin-2. 4.1 maven-clean-plugin cd maven-clean-plugin mvn deploy Hmm... first, do you *really* have a requirement to build everything from source? Chi riceve il presente messaggio e' tenuto a verificare se lo stesso non gli sia pervenuto per errore. In tal caso e' pregato di avvisare immediatamente il mittente e, tenuto conto delle responsabilita' connesse all'indebito utilizzo e/o divulgazione del messaggio e/o delle informazioni in esso contenute, voglia cancellare l'originale e distruggere le varie copie o stampe. The receiver of this message is required to check if he/she has received it erroneously. If so, the receiver is requested to immediately inform the sender and - in consideration of the responsibilities arising from undue use and/or disclosure of the message and/or the information contained therein - destroy the original message and any copy or printout thereof. - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Bootstraping a repository manager
Once you had Maven collect all the dependencies for you you can use something like find repository_storage_root -name '*.jar' if you are on Linux/UNIX or Windows's search tool to obtain a list of all your dependencies. With that you can use something like http://mvnrepository.com/ to gather information about each jar and make your decisions. This would take you a lot less than your approach. Cheers, Nicola Musatti Heck, Gus (Patrick) wrote: Hi Wendy, I don't have a requirement to build from source, but in the case of the maven plugins, nowhere on the web seems to point to a place where I can download the finished product (aside from letting maven find it, and who knows what other dependencies). I am adverse to letting maven just pull things because then I have a very long list of items to check and verify arranged in a tree, and it's all too easy to miss a subtree and think I've got everything covered. Basically the problem is I make a really crappy computer when it comes to tree-traversals and not losing my place when someone comes over to chat, I go to lunch, or I have to go home at the end of the day. :-) I don't want to trust that I won't make a mistake, I'd rather take a bit more time and let the build system ensure that I didn't miss anything. If I act as gatekeeper, I know that the project won't build properly until I've applied each dependency required to build the project. (and then same for test... etc). This shouldn't be any more burdensome than finding the transitive dependencies for a good sized ant project, which I've done many times before, except that maven plug-in folks seem to squirrel their stuff away where only maven can find it, so I'm having trouble getting maven working to start with. -Gus -Original Message- From: Wendy Smoak [mailto:wsm...@gmail.com] Sent: Wednesday, May 18, 2011 6:35 PM To: Maven Users List Subject: Re: Bootstraping a repository manager On Wed, May 18, 2011 at 5:21 PM, Heck, Gus (Patrick) wrote: I can't seem to find a place to download something that I can upload directly to artifactory, so I tried to start with the first plugin that was failing, and build that and see if mvn deploy would deploy it to artifactory. (First, question... is that a reasonable idea?) That would be: http://repo1.maven.org/maven2 However it is not going to be fun to find each thing you need and upload it to your internal repository manager. (I work with a company that does it this way.) Especially for plugins. Is there any middle ground? Let the repository manager proxy into a segregated directory, vet everything in there and then move it to the pristine internal repo? Unfortunately, as soon as I did svn checkout http://svn.apache.org/repos/asf/maven/plugins/tags/maven-clean-plugin-2. 4.1 maven-clean-plugin cd maven-clean-plugin mvn deploy Hmm... first, do you *really* have a requirement to build everything from source? Chi riceve il presente messaggio e' tenuto a verificare se lo stesso non gli sia pervenuto per errore. In tal caso e' pregato di avvisare immediatamente il mittente e, tenuto conto delle responsabilita' connesse all'indebito utilizzo e/o divulgazione del messaggio e/o delle informazioni in esso contenute, voglia cancellare l'originale e distruggere le varie copie o stampe. The receiver of this message is required to check if he/she has received it erroneously. If so, the receiver is requested to immediately inform the sender and - in consideration of the responsibilities arising from undue use and/or disclosure of the message and/or the information contained therein - destroy the original message and any copy or printout thereof. - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Bootstraping a repository manager
The other nice thing about Maven is that no matter what one is building or how unusual one thinks their environment is, someone has already built something similar and someone has already worked out a best practice in a similar environment. It is also supported by a great forum where you get advice that you could not find or afford and it is free here. Ron On 18/05/2011 9:32 PM, Wendy Smoak wrote: On Wed, May 18, 2011 at 7:56 PM, Heck, Gus (Patrick) wrote: As I explained in another response, I want the software to tell me when I've got enough stuff to build (much like test driven development), rather than trusting I covered everything post-hoc. As for doing it with ant projects, I've certainly been there. At one job I wrote a custom ant task that looked at /lib, a properties file and a directory called /licenses. If the jar in lib didn't have an entry in the properties file, or the value for that jar in the properties file didn't match the name of a file in /licenses, the build failed. Not really that hard to implement. As you say, the big time sink is in hunting down the licenses, but in the current job I have to do that anyway to include a copy of the license with the approval form for the lawyers... nothing maven can do about that. The only irritation is that some basic artifacts seem to be hard to find and instead of an answer such as "the artifacts you need can be found here" I get a chorus of "don't do that" Hm? I already told you where the things are. In the central repository. (And you need both the jar and the pom for it to work.) Typically the only organizations that can afford to be this paranoid are really big ones. I work with one of those. The internal Maven repo does not talk to the internet. Instead, developers fill out a web form to request a new artifact to be uploaded, someone approves it if it's in the standards, or sends them off to the reference architecture team to explain why they need it, and then someone else presses the buttons in Archiva to upload it. Plugins are a special case. Those we handle within the build team, as they typically want all kinds of dependencies. So for those we connect to the internet and use the Maven Assembly plugin to create a repository bundle by listing the plugin as a dependency. Maven goes off and gets everything that plugin needs, and the whole bundle is uploaded. Those artifacts generally aren't going to get into your end product, they are just used during the build. By the way, I am a bit irritated that maven put's the repository information out on the developer's settings.xml. I'd much rather be able to manage that centrally, and not leave it as something that they could screw up, or forget to do after their hard drive fails and the set things up a second time. Is there a way to lock things into a single repository without settings.xml? That's where it's done, with a mirror in settings.xml. You can create your own Maven distribution with the settings.xml you want them to use packaged inside it (in the /conf directory.) I assume if you're this strict about dependencies, developers are not downloading random software from the internet and installing it on their machines, so provide the Maven distro you want them to use. - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Bootstraping a repository manager
On Wed, May 18, 2011 at 7:56 PM, Heck, Gus (Patrick) wrote: > As I explained in another response, I want the software to tell me when I've > got enough stuff to build (much like test driven development), rather than > trusting I covered everything post-hoc. As for doing it with ant projects, > I've certainly been there. At one job I wrote a custom ant task that looked > at /lib, a properties file and a directory called /licenses. If the jar in > lib didn't have an entry in the properties file, or the value for that jar in > the properties file didn't match the name of a file in /licenses, the build > failed. Not really that hard to implement. As you say, the big time sink is > in hunting down the licenses, but in the current job I have to do that anyway > to include a copy of the license with the approval form for the lawyers... > nothing maven can do about that. The only irritation is that some basic > artifacts seem to be hard to find and instead of an answer such as "the > artifacts you need can be found here" I get a chorus of "don't do that" Hm? I already told you where the things are. In the central repository. (And you need both the jar and the pom for it to work.) Typically the only organizations that can afford to be this paranoid are really big ones. I work with one of those. The internal Maven repo does not talk to the internet. Instead, developers fill out a web form to request a new artifact to be uploaded, someone approves it if it's in the standards, or sends them off to the reference architecture team to explain why they need it, and then someone else presses the buttons in Archiva to upload it. Plugins are a special case. Those we handle within the build team, as they typically want all kinds of dependencies. So for those we connect to the internet and use the Maven Assembly plugin to create a repository bundle by listing the plugin as a dependency. Maven goes off and gets everything that plugin needs, and the whole bundle is uploaded. Those artifacts generally aren't going to get into your end product, they are just used during the build. > By the way, I am a bit irritated that maven put's the repository information > out on the developer's settings.xml. I'd much rather be able to manage that > centrally, and not leave it as something that they could screw up, or forget > to do after their hard drive fails and the set things up a second time. Is > there a way to lock things into a single repository without settings.xml? That's where it's done, with a mirror in settings.xml. You can create your own Maven distribution with the settings.xml you want them to use packaged inside it (in the /conf directory.) I assume if you're this strict about dependencies, developers are not downloading random software from the internet and installing it on their machines, so provide the Maven distro you want them to use. -- Wendy - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
RE: Bootstraping a repository manager
As I explained in another response, I want the software to tell me when I've got enough stuff to build (much like test driven development), rather than trusting I covered everything post-hoc. As for doing it with ant projects, I've certainly been there. At one job I wrote a custom ant task that looked at /lib, a properties file and a directory called /licenses. If the jar in lib didn't have an entry in the properties file, or the value for that jar in the properties file didn't match the name of a file in /licenses, the build failed. Not really that hard to implement. As you say, the big time sink is in hunting down the licenses, but in the current job I have to do that anyway to include a copy of the license with the approval form for the lawyers... nothing maven can do about that. The only irritation is that some basic artifacts seem to be hard to find and instead of an answer such as "the artifacts you need can be found here" I get a chorus of "don't do that" By the way, I am a bit irritated that maven put's the repository information out on the developer's settings.xml. I'd much rather be able to manage that centrally, and not leave it as something that they could screw up, or forget to do after their hard drive fails and the set things up a second time. Is there a way to lock things into a single repository without settings.xml? Malicious circumvention is of course a whole different topic, and would be impossible to prevent without an encrypted form of the pom anyway. -Gus -Original Message- From: Benson Margulies [mailto:bimargul...@gmail.com] Sent: Wednesday, May 18, 2011 7:35 PM To: Maven Users List Subject: Re: Bootstraping a repository manager Managing a maven build under these constraints is prohibitively expensive for a small operation if you apply it to maven itself. The thing you are building may have a modest enough dependency collection, but maven itself? I think you are looking an an inventory in the hundreds. It's a gigantic undertaking. Hmm, maybe someone could make some money selling 'trusted maven' -- a canned repo full of plugins and their deps where the seller takes legal responsibility for license and provenance :-) Again, a question here is provenance versus license. Say that you carefully set up an environment to load dependencies only from central. You run a build. You use the available tools to load the contents of your repo into a repo manager, and you tell everyone else to configure maven to load from that and nowhere else. Now you have provenance settled (if you trust central). Maven doesn't 'load things from anywhere'. It loads things from where you tell it to load things from. By default, that's central, period. Could a rogue developer circumvent? Of course, on their local machine. Can they push unwanted things into your repo? Not if you set the access control properly. Next you have dependency licenses to worry about. Well, even if you converted back to ant, you'd still have dependencies to worry about. At least maven will make you a nice report of the licenses of the dependencies as declared in the pom. With ant, you'd be googling away tracking down the original sources and licenses of all the dependencies. So I don't see how the maven-ness of this thing makes your problem materially harder. It's a big problem either way. - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email, and any attachments thereto, is strictly prohibited. If you receive this email in error please immediately notify the sender and permanently delete the original copy and any copy of any e-mail, and any printout thereof.
Re: Bootstraping a repository manager
1) Connect to the Internet, 2) Set up your repo to point to the external repos that you trust. 3) Load your repo by doing builds that you do not install or deploy. 4) Upload the 3rd party libraries that are not available from public Maven Repos - software with license restrictions 5) Disconnect from the internet 6) Delete any libraries that you want to disallow - can't imagine what these would be since you did put them in a POM for a reason. 7) Fix the poms that call up disallowed libraries to use different libraries. 8) Start building We have done other things to restrict the libraries that users use. I have bored people to death here describing this approach so you can likely find it in the archives. Ron On 18/05/2011 5:36 PM, Heck, Gus (Patrick) wrote: Hi Brian, I think you missed the entire point of the exercise. The point is to NOT allow it to pull from outside sources, as a proxy or not. The intent is to load it up manually and never ever let it fetch serve any dependency or plugin I haven't personally uploaded, and intentionally provided for use. This relates to a coming set of network access restrictions as well as overseas developers not accidentally downloading export controlled, or locally illegal software. Also, you will note that I am currently working with artifactory. -Gus -Original Message- From: Brian Fox [mailto:bri...@infinity.nu] Sent: Wednesday, May 18, 2011 5:26 PM To: Maven Users List Subject: Re: Bootstraping a repository manager You don't need to bootsrap it, just setup a repo like Nexus and let it proxy on demand the things you need. In that case a bootstrap might simply mean run all our builds and/or run mvn dependency:go-offline to resolve everything you need. On Wed, May 18, 2011 at 5:21 PM, Heck, Gus (Patrick) wrote: Hi Folks, The list archives return hundreds or thousands of not very relevant results when I search them, so this may have been answered, but I can't find it. In any case, I'm also somewhat new to maven so I'm possibly asking silly questions for that reason too. In either case I apologize in advance... I've got a project, that is derived from another project that was set up with maven (else, I'd probably still be using ant). In our company, our policies make it very important that I know exactly what is going into the project, so I can't just let maven go slurping things up from anywhere it feels like on the web. To this end, I'm testing out the free version of artifactory, which looks like a nice solution to that problem. I've managed to get my maven pointed at artifactory, and wiped out my local repository. I also deleted artifactory's references to the outside world. Now my builds complain that I can't find stuff, and the messages are clearly looking at my artifactory. Perfect so far. However, as pleased as I am to have broken my build, now I need to get it going again :-). My first challenge seems to be getting the basic maven plugins into the repository. I can't seem to find a place to download something that I can upload directly to artifactory, so I tried to start with the first plugin that was failing, and build that and see if mvn deploy would deploy it to artifactory. (First, question... is that a reasonable idea?) Unfortunately, as soon as I did svn checkout http://svn.apache.org/repos/asf/maven/plugins/tags/maven-clean-plugin-2. 4.1 maven-clean-plugin cd maven-clean-plugin mvn deploy I got [INFO] Scanning for projects... Downloading: http://heckp2.corp.aspentech.com:8080/artifactory/libs-release/org/apach e/maven/plugins/maven-plugins/18/maven-plugins-18.pom Downloading: http://heckp2.corp.aspentech.com:8080/artifactory/libs-snapshot/org/apac he/maven/plugins/maven-plugins/18/maven-plugins-18.pom [ERROR] The build could not read 1 project -> [Help 1] [ERROR] [ERROR] The project org.apache.maven.plugins:maven-clean-plugin:2.4.1 (C:\cygwin\home\gus\maven\plugins\maven-clean-plugin\pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact org.apache.maven.plugins:maven-plugins:pom:18 in central (http://heckp2.corp.aspentech.com:8080/artifactory/libs-release) and 'parent.relativePath' points at wrong local POM @ line 25, column 11 -> [Help 2] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingExceptio n [ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelExcept ion bash-4.1$ So I went and looked at the subversion repository and I found maven-plugins-18, but when I tried to build that I got.. [INFO] Scanning for projects... Downloading: http://heckp2.corp.aspentech.c
RE: Bootstraping a repository manager
Ansgar, I'm not new to building stuff, just new to maven. Unrolling transitive dependencies is old hat for folks who grew up on ant :). It's more work, but in the end there's no question of what is or is not in your project, and I want to be sure that I only pull the dependencies I actually need, since I will likely wind up filling out an approval form for each of them at some point. Another concern of course is which FOS licenses are being depended upon. Another facet of this is the need to provide a location that can be scanned with a tool that detects GPL'd code, or other such problems. I don't understand your bit about test dependencies. I don't need to run the tests for things I am using. The release engineers for those projects presumably did that already. Note that the only reason I went to the source for the plugin was I couldn't find a binary. -Gus -Original Message- From: Ansgar Konermann [mailto:ansgar.konerm...@googlemail.com] Sent: Wednesday, May 18, 2011 6:34 PM To: users@maven.apache.org Subject: Re: Bootstraping a repository manager Am 18.05.2011 23:21, schrieb Heck, Gus (Patrick): > In our company, our > policies make it very important that I know exactly what is going into > the project, so I can't just let maven go slurping things up from > anywhere it feels like on the web. In a network allowed to access the outside world: - clean out your local repository (rm -rf ~/.m2/repository) - run your build - review what's inside your local repository and decide whether that's okay according to your policies The other way round (upload everything which is needed manually) completely thwarts the idea of automatic dependency management. To make it short: this is probably a dead end. The approach you outlined would mean to download and build from source each and every project your build is depending upon. Unless you are a world-class-superhero-build-and-release-manager with a lot of spare time, or your project has only a tiny set of (transitive) dependencies, this probably won't work. I doubt one person can accrue the knowledge required to build all these software modules which get deployed to the central maven repository in a reasonable time. If you don't believe, run mvn dependency:tree on your project, this will give you the dependency tree for compile-time dependencies. Add to this the test-dependencies needed to execute the tests of your compile-time dependencies which happens before packaging (i. e. you won't get a JAR if the tests don't run). If this is not enough to stop you from pursuing your initial approach, add all the maven plugins which make up your build system, and their dependencies. Best regards Ansgar - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email, and any attachments thereto, is strictly prohibited. If you receive this email in error please immediately notify the sender and permanently delete the original copy and any copy of any e-mail, and any printout thereof. - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Bootstraping a repository manager
Managing a maven build under these constraints is prohibitively expensive for a small operation if you apply it to maven itself. The thing you are building may have a modest enough dependency collection, but maven itself? I think you are looking an an inventory in the hundreds. It's a gigantic undertaking. Hmm, maybe someone could make some money selling 'trusted maven' -- a canned repo full of plugins and their deps where the seller takes legal responsibility for license and provenance :-) Again, a question here is provenance versus license. Say that you carefully set up an environment to load dependencies only from central. You run a build. You use the available tools to load the contents of your repo into a repo manager, and you tell everyone else to configure maven to load from that and nowhere else. Now you have provenance settled (if you trust central). Maven doesn't 'load things from anywhere'. It loads things from where you tell it to load things from. By default, that's central, period. Could a rogue developer circumvent? Of course, on their local machine. Can they push unwanted things into your repo? Not if you set the access control properly. Next you have dependency licenses to worry about. Well, even if you converted back to ant, you'd still have dependencies to worry about. At least maven will make you a nice report of the licenses of the dependencies as declared in the pom. With ant, you'd be googling away tracking down the original sources and licenses of all the dependencies. So I don't see how the maven-ness of this thing makes your problem materially harder. It's a big problem either way. - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
RE: Bootstraping a repository manager
Hi Wendy, I don't have a requirement to build from source, but in the case of the maven plugins, nowhere on the web seems to point to a place where I can download the finished product (aside from letting maven find it, and who knows what other dependencies). I am adverse to letting maven just pull things because then I have a very long list of items to check and verify arranged in a tree, and it's all too easy to miss a subtree and think I've got everything covered. Basically the problem is I make a really crappy computer when it comes to tree-traversals and not losing my place when someone comes over to chat, I go to lunch, or I have to go home at the end of the day. :-) I don't want to trust that I won't make a mistake, I'd rather take a bit more time and let the build system ensure that I didn't miss anything. If I act as gatekeeper, I know that the project won't build properly until I've applied each dependency required to build the project. (and then same for test... etc). This shouldn't be any more burdensome than finding the transitive dependencies for a good sized ant project, which I've done many times before, except that maven plug-in folks seem to squirrel their stuff away where only maven can find it, so I'm having trouble getting maven working to start with. -Gus -Original Message- From: Wendy Smoak [mailto:wsm...@gmail.com] Sent: Wednesday, May 18, 2011 6:35 PM To: Maven Users List Subject: Re: Bootstraping a repository manager On Wed, May 18, 2011 at 5:21 PM, Heck, Gus (Patrick) wrote: > I can't seem to find a place to > download something that I can upload directly to artifactory, so I tried > to start with the first plugin that was failing, and build that and see > if mvn deploy would deploy it to artifactory. (First, question... is > that a reasonable idea?) That would be: http://repo1.maven.org/maven2 However it is not going to be fun to find each thing you need and upload it to your internal repository manager. (I work with a company that does it this way.) Especially for plugins. Is there any middle ground? Let the repository manager proxy into a segregated directory, vet everything in there and then move it to the pristine internal repo? > Unfortunately, as soon as I did > > svn checkout > http://svn.apache.org/repos/asf/maven/plugins/tags/maven-clean-plugin-2. > 4.1 maven-clean-plugin > cd maven-clean-plugin > mvn deploy Hmm... first, do you *really* have a requirement to build everything from source? -- Wendy - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email, and any attachments thereto, is strictly prohibited. If you receive this email in error please immediately notify the sender and permanently delete the original copy and any copy of any e-mail, and any printout thereof. - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Bootstraping a repository manager
Am 18.05.2011 23:36, schrieb Heck, Gus (Patrick): > This relates to a coming set of network access restrictions as well as > overseas developers not accidentally downloading export controlled, or > locally illegal software. Hi, regarding the overseas developers: try a repository manager with decent support for access control. I know that Sonatype Nexus does support quite fine-grained control over who can access which artifacts. You could have two roles for your developers, one which is allowed to download export-controlled artifacts, the other one is not. You would have to instruct your developers how to set up their maven settings.xml, so that maven knows which login credentials to present to the repository manager. Best regards Ansgar - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Bootstraping a repository manager
On Wed, May 18, 2011 at 5:21 PM, Heck, Gus (Patrick) wrote: > I can't seem to find a place to > download something that I can upload directly to artifactory, so I tried > to start with the first plugin that was failing, and build that and see > if mvn deploy would deploy it to artifactory. (First, question... is > that a reasonable idea?) That would be: http://repo1.maven.org/maven2 However it is not going to be fun to find each thing you need and upload it to your internal repository manager. (I work with a company that does it this way.) Especially for plugins. Is there any middle ground? Let the repository manager proxy into a segregated directory, vet everything in there and then move it to the pristine internal repo? > Unfortunately, as soon as I did > > svn checkout > http://svn.apache.org/repos/asf/maven/plugins/tags/maven-clean-plugin-2. > 4.1 maven-clean-plugin > cd maven-clean-plugin > mvn deploy Hmm... first, do you *really* have a requirement to build everything from source? -- Wendy - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Bootstraping a repository manager
Am 18.05.2011 23:21, schrieb Heck, Gus (Patrick): > In our company, our > policies make it very important that I know exactly what is going into > the project, so I can't just let maven go slurping things up from > anywhere it feels like on the web. In a network allowed to access the outside world: - clean out your local repository (rm -rf ~/.m2/repository) - run your build - review what's inside your local repository and decide whether that's okay according to your policies The other way round (upload everything which is needed manually) completely thwarts the idea of automatic dependency management. To make it short: this is probably a dead end. The approach you outlined would mean to download and build from source each and every project your build is depending upon. Unless you are a world-class-superhero-build-and-release-manager with a lot of spare time, or your project has only a tiny set of (transitive) dependencies, this probably won't work. I doubt one person can accrue the knowledge required to build all these software modules which get deployed to the central maven repository in a reasonable time. If you don't believe, run mvn dependency:tree on your project, this will give you the dependency tree for compile-time dependencies. Add to this the test-dependencies needed to execute the tests of your compile-time dependencies which happens before packaging (i. e. you won't get a JAR if the tests don't run). If this is not enough to stop you from pursuing your initial approach, add all the maven plugins which make up your build system, and their dependencies. Best regards Ansgar - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
RE: Bootstraping a repository manager
Hi Brian, I think you missed the entire point of the exercise. The point is to NOT allow it to pull from outside sources, as a proxy or not. The intent is to load it up manually and never ever let it fetch serve any dependency or plugin I haven't personally uploaded, and intentionally provided for use. This relates to a coming set of network access restrictions as well as overseas developers not accidentally downloading export controlled, or locally illegal software. Also, you will note that I am currently working with artifactory. -Gus -Original Message- From: Brian Fox [mailto:bri...@infinity.nu] Sent: Wednesday, May 18, 2011 5:26 PM To: Maven Users List Subject: Re: Bootstraping a repository manager You don't need to bootsrap it, just setup a repo like Nexus and let it proxy on demand the things you need. In that case a bootstrap might simply mean run all our builds and/or run mvn dependency:go-offline to resolve everything you need. On Wed, May 18, 2011 at 5:21 PM, Heck, Gus (Patrick) wrote: > Hi Folks, > > The list archives return hundreds or thousands of not very relevant > results when I search them, so this may have been answered, but I can't > find it. In any case, I'm also somewhat new to maven so I'm possibly > asking silly questions for that reason too. In either case I apologize > in advance... > > I've got a project, that is derived from another project that was set up > with maven (else, I'd probably still be using ant). In our company, our > policies make it very important that I know exactly what is going into > the project, so I can't just let maven go slurping things up from > anywhere it feels like on the web. To this end, I'm testing out the free > version of artifactory, which looks like a nice solution to that > problem. I've managed to get my maven pointed at artifactory, and wiped > out my local repository. I also deleted artifactory's references to the > outside world. Now my builds complain that I can't find stuff, and the > messages are clearly looking at my artifactory. Perfect so far. > > However, as pleased as I am to have broken my build, now I need to get > it going again :-). My first challenge seems to be getting the basic > maven plugins into the repository. I can't seem to find a place to > download something that I can upload directly to artifactory, so I tried > to start with the first plugin that was failing, and build that and see > if mvn deploy would deploy it to artifactory. (First, question... is > that a reasonable idea?) > > Unfortunately, as soon as I did > > svn checkout > http://svn.apache.org/repos/asf/maven/plugins/tags/maven-clean-plugin-2. > 4.1 maven-clean-plugin > cd maven-clean-plugin > mvn deploy > > I got > > [INFO] Scanning for projects... > Downloading: > http://heckp2.corp.aspentech.com:8080/artifactory/libs-release/org/apach > e/maven/plugins/maven-plugins/18/maven-plugins-18.pom > Downloading: > http://heckp2.corp.aspentech.com:8080/artifactory/libs-snapshot/org/apac > he/maven/plugins/maven-plugins/18/maven-plugins-18.pom > [ERROR] The build could not read 1 project -> [Help 1] > [ERROR] > [ERROR] The project org.apache.maven.plugins:maven-clean-plugin:2.4.1 > (C:\cygwin\home\gus\maven\plugins\maven-clean-plugin\pom.xml) has 1 > error > [ERROR] Non-resolvable parent POM: Could not find artifact > org.apache.maven.plugins:maven-plugins:pom:18 in central > (http://heckp2.corp.aspentech.com:8080/artifactory/libs-release) and > 'parent.relativePath' points at wrong local POM @ line 25, column 11 -> > [Help 2] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, > please read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingExceptio > n > [ERROR] [Help 2] > http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelExcept > ion > bash-4.1$ > > So I went and looked at the subversion repository and I found > maven-plugins-18, but when I tried to build that I got.. > > [INFO] Scanning for projects... > Downloading: > http://heckp2.corp.aspentech.com:8080/artifactory/libs-release/org/apach > e/maven/maven-parent/16/maven-parent-16.pom > Downloading: > http://heckp2.corp.aspentech.com:8080/artifactory/libs-snapshot/org/apac > he/maven/maven-parent/16/maven-parent-16.pom > [ERROR] The build could not read 1 project -> [Help 1] > [ERROR] > [ERROR] The project org.apache.maven.plugins:maven-plugins:18 > (C:\cygwin\home\gus\maven\plugins
Re: Bootstraping a repository manager
You don't need to bootsrap it, just setup a repo like Nexus and let it proxy on demand the things you need. In that case a bootstrap might simply mean run all our builds and/or run mvn dependency:go-offline to resolve everything you need. On Wed, May 18, 2011 at 5:21 PM, Heck, Gus (Patrick) wrote: > Hi Folks, > > The list archives return hundreds or thousands of not very relevant > results when I search them, so this may have been answered, but I can't > find it. In any case, I'm also somewhat new to maven so I'm possibly > asking silly questions for that reason too. In either case I apologize > in advance... > > I've got a project, that is derived from another project that was set up > with maven (else, I'd probably still be using ant). In our company, our > policies make it very important that I know exactly what is going into > the project, so I can't just let maven go slurping things up from > anywhere it feels like on the web. To this end, I'm testing out the free > version of artifactory, which looks like a nice solution to that > problem. I've managed to get my maven pointed at artifactory, and wiped > out my local repository. I also deleted artifactory's references to the > outside world. Now my builds complain that I can't find stuff, and the > messages are clearly looking at my artifactory. Perfect so far. > > However, as pleased as I am to have broken my build, now I need to get > it going again :-). My first challenge seems to be getting the basic > maven plugins into the repository. I can't seem to find a place to > download something that I can upload directly to artifactory, so I tried > to start with the first plugin that was failing, and build that and see > if mvn deploy would deploy it to artifactory. (First, question... is > that a reasonable idea?) > > Unfortunately, as soon as I did > > svn checkout > http://svn.apache.org/repos/asf/maven/plugins/tags/maven-clean-plugin-2. > 4.1 maven-clean-plugin > cd maven-clean-plugin > mvn deploy > > I got > > [INFO] Scanning for projects... > Downloading: > http://heckp2.corp.aspentech.com:8080/artifactory/libs-release/org/apach > e/maven/plugins/maven-plugins/18/maven-plugins-18.pom > Downloading: > http://heckp2.corp.aspentech.com:8080/artifactory/libs-snapshot/org/apac > he/maven/plugins/maven-plugins/18/maven-plugins-18.pom > [ERROR] The build could not read 1 project -> [Help 1] > [ERROR] > [ERROR] The project org.apache.maven.plugins:maven-clean-plugin:2.4.1 > (C:\cygwin\home\gus\maven\plugins\maven-clean-plugin\pom.xml) has 1 > error > [ERROR] Non-resolvable parent POM: Could not find artifact > org.apache.maven.plugins:maven-plugins:pom:18 in central > (http://heckp2.corp.aspentech.com:8080/artifactory/libs-release) and > 'parent.relativePath' points at wrong local POM @ line 25, column 11 -> > [Help 2] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, > please read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingExceptio > n > [ERROR] [Help 2] > http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelExcept > ion > bash-4.1$ > > So I went and looked at the subversion repository and I found > maven-plugins-18, but when I tried to build that I got.. > > [INFO] Scanning for projects... > Downloading: > http://heckp2.corp.aspentech.com:8080/artifactory/libs-release/org/apach > e/maven/maven-parent/16/maven-parent-16.pom > Downloading: > http://heckp2.corp.aspentech.com:8080/artifactory/libs-snapshot/org/apac > he/maven/maven-parent/16/maven-parent-16.pom > [ERROR] The build could not read 1 project -> [Help 1] > [ERROR] > [ERROR] The project org.apache.maven.plugins:maven-plugins:18 > (C:\cygwin\home\gus\maven\plugins\maven-plugins\pom.xml) has 1 error > [ERROR] Non-resolvable parent POM: Could not find artifact > org.apache.maven:maven-parent:pom:16 in central > (http://heckp2.corp.aspentech.com:8080/artifactory/libs-release) and > 'parent.relativePath' points at wrong local POM @ line 23, column 11 -> > [Help 2] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, > please read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingExceptio > n > [ERROR] [Help 2] > http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelExcept > ion > > When I look in subversion, I don't see a maven-parent... > > So, now I don't see any clear path to walk up to wherever the root of > this is... > > What is the best way to get fresh clean maven plugins for installation > into a repository manager without allowing it