Re: Maven Repository Manager: Will it avoid aggregation of repositories?
Hi, the issue is fxed and released. Please try it! The current release relies on the following solution: Repositories are divided into groups (String repo.getGroupId()). Repositories IN the same groups are merged, otherwise they are spoofed (as before). The ordering of reposes are still important, but beside repositoryOrder (list of all reposes configured independent of groupId) we have subchains by groups with same order of members. The algorithm: if we have request for non mergeable item, use who serves first -- wins by repositoryOrder. if we have request for mergable item, use who serves first -- wins to get initial copy. Then, iterate over the found item's originating repo group, and merge the result (resultlist). Thus, using current Px RC2 a solution to Barrie's inital problem would be: repo A: group A repo B: group B repo C: group C or even more sophisticated: A: group One B: group Two C: group Two So, A may be used to spoof (inhouse), but B and C (if A does not serve what it needs) will be merging metadatas. RC2 has a factory configuration as this: inhouse, group: private extFree, group: private extNonFree, group: private central, group: public All four reposes shares same repository logic, so they share newly implemented snapshot policies too: serveSnapshots=false, serverReleases=true (release is something that is not snapshot in Px vocabulary). Question: what should happen with maven-metadata.xml checksum (sha1, md5) after merge? Thanx in advance, ~t~ On 7/26/06, Trygve Laugstøl [EMAIL PROTECTED] wrote: Tamás Cservenák wrote: Hi, On 7/26/06, Trygve Laugstøl [EMAIL PROTECTED] wrote: That is not the problem, the problem is that the metadata from all the repositories has to be _merged_ and dispatched properly by proximity. That is why any plain http proxy will fail acting as a repository for Maven. Ah, I see. My knowledge was here faulty a little bit, sorry. At a first glance, it is easily implentable at proximityBean level (it has the list of repos), using some similar pattern as repositoryBean does for retrieval logic. Actually, externalize the current first serves wins serving algorithm to some customizable form. And this issue is only for aggregated reposes. Does anything else needs merging through reposes? Not that I can think of right now. I will create an issue for that. How can Maven solve this problem? Maven will properly merge the metadata from the different repositories but Proximity is effectively stopping half of the data coming through. sorry, i meant to say: solution could be to not aggregate repositories, you could use different base URLs for them (in POM or settings.xml). It is what Barrie suggested on the top of the thread. And it is (will be) solvable with the repo relocation i was talking about. Proximity will serve metadata properly then, no? Sounds like it. -- Trygve - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Maven Repository Manager: Will it avoid aggregation of repositories?
Tamás Cservenák wrote: Hi, Not quite. Proximity is (almost) plain HTTP proxy, but IT DOES have knowledge about maven reposes (see repository browsing or the use of ?repositoryId=central URL parameter.), but only as separate storages. What it does not have is STATE, it cannot distuingish the two consecutive HTTP GET commands from same M2, once for metadata and once for POM. That is not the problem, the problem is that the metadata from all the repositories has to be _merged_ and dispatched properly by proximity. That is why any plain http proxy will fail acting as a repository for Maven. The solution should be in POM (Maven). How can Maven solve this problem? Maven will properly merge the metadata from the different repositories but Proximity is effectively stopping half of the data coming through. -- Trygve - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Maven Repository Manager: Will it avoid aggregation of repositories?
Hi, On 7/26/06, Trygve Laugstøl [EMAIL PROTECTED] wrote: That is not the problem, the problem is that the metadata from all the repositories has to be _merged_ and dispatched properly by proximity. That is why any plain http proxy will fail acting as a repository for Maven. Ah, I see. My knowledge was here faulty a little bit, sorry. At a first glance, it is easily implentable at proximityBean level (it has the list of repos), using some similar pattern as repositoryBean does for retrieval logic. Actually, externalize the current first serves wins serving algorithm to some customizable form. And this issue is only for aggregated reposes. Does anything else needs merging through reposes? I will create an issue for that. How can Maven solve this problem? Maven will properly merge the metadata from the different repositories but Proximity is effectively stopping half of the data coming through. sorry, i meant to say: solution could be to not aggregate repositories, you could use different base URLs for them (in POM or settings.xml). It is what Barrie suggested on the top of the thread. And it is (will be) solvable with the repo relocation i was talking about. Proximity will serve metadata properly then, no? Thanx, ~t~
Re: Maven Repository Manager: Will it avoid aggregation of repositories?
Tamás Cservenák wrote: Hi, On 7/26/06, Trygve Laugstøl [EMAIL PROTECTED] wrote: That is not the problem, the problem is that the metadata from all the repositories has to be _merged_ and dispatched properly by proximity. That is why any plain http proxy will fail acting as a repository for Maven. Ah, I see. My knowledge was here faulty a little bit, sorry. At a first glance, it is easily implentable at proximityBean level (it has the list of repos), using some similar pattern as repositoryBean does for retrieval logic. Actually, externalize the current first serves wins serving algorithm to some customizable form. And this issue is only for aggregated reposes. Does anything else needs merging through reposes? Not that I can think of right now. I will create an issue for that. How can Maven solve this problem? Maven will properly merge the metadata from the different repositories but Proximity is effectively stopping half of the data coming through. sorry, i meant to say: solution could be to not aggregate repositories, you could use different base URLs for them (in POM or settings.xml). It is what Barrie suggested on the top of the thread. And it is (will be) solvable with the repo relocation i was talking about. Proximity will serve metadata properly then, no? Sounds like it. -- Trygve - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Maven Repository Manager: Will it avoid aggregation of repositories?
Hi, I created 3 issues: http://trac.abstracthorizon.org/proximity/ticket/3 http://trac.abstracthorizon.org/proximity/ticket/17 http://trac.abstracthorizon.org/proximity/ticket/18 and before a few days already got a ticket with similar request: http://trac.abstracthorizon.org/proximity/ticket/16 So, 3 + 16 + 17 + 18 should do it :) Thanx for help! ~t~ On 7/26/06, Trygve Laugstøl [EMAIL PROTECTED] wrote: Tamás Cservenák wrote: Hi, On 7/26/06, Trygve Laugstøl [EMAIL PROTECTED] wrote: That is not the problem, the problem is that the metadata from all the repositories has to be _merged_ and dispatched properly by proximity. That is why any plain http proxy will fail acting as a repository for Maven. Ah, I see. My knowledge was here faulty a little bit, sorry. At a first glance, it is easily implentable at proximityBean level (it has the list of repos), using some similar pattern as repositoryBean does for retrieval logic. Actually, externalize the current first serves wins serving algorithm to some customizable form. And this issue is only for aggregated reposes. Does anything else needs merging through reposes? Not that I can think of right now. I will create an issue for that. How can Maven solve this problem? Maven will properly merge the metadata from the different repositories but Proximity is effectively stopping half of the data coming through. sorry, i meant to say: solution could be to not aggregate repositories, you could use different base URLs for them (in POM or settings.xml). It is what Barrie suggested on the top of the thread. And it is (will be) solvable with the repo relocation i was talking about. Proximity will serve metadata properly then, no? Sounds like it. -- Trygve - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Maven Repository Manager: Will it avoid aggregation of repositories?
Something else of note I discovered using Proximity: If you have multiple repositories mirrored, you need to keep the snapshot repositories away from the release repositories. In my case, I was using Proximity to mirror central but it also had several other repositories in its configuration including a couple snapshot repositories. My original thinking was that I could do this and avoid needing to specify individual repositories in my master POM. The problem with this arrangement is that if Proximity returns metadata for a snapshot release, Maven will be unable to retrieve the artifact and POM because central is defined as releases only. I don't know if this problem would be eliminated by using the new global mirror setting that will be coming out in 2.0.5 or not. -Original Message- From: Brett Porter [mailto:[EMAIL PROTECTED] Sent: Monday, July 24, 2006 23:42 To: Maven Developers List Subject: Re: Maven Repository Manager: Will it avoid aggregation of repositories? Hi Barrie, While the current implementation is aggregating them, I'll keep this in mind when I review it in the next couple of days. It's probably best to keep the repositories separate, but to have a single index for the rest of the operations so it can appear as an aggregate (and be able to access a view over the aggregate from certain URLs). - Brett On 25/07/2006 1:37 PM, Barrie Treloar wrote: I just stumbled into a feature of maven-proxy which caused me some problems. Assume you have three repositories defined in your maven-proxy: A, B and C. In your pom you have specified the repository A and in your settings have configured A to point to your maven-proxy installation. Because maven-proxy aggregates the repositories the artifacts from the B and C repositories are available even though you have not defined them in your pom. Normally this would not show up as an issue. The problem manifests as soon as you start using SNAPSHOT version of artifacts, perhaps as part of doing some work on maven itself. Due to the aggregation you will find that you are using SNAPSHOT versions instead of the more stable released versions. My workaround was to not put SNAPSHOT repositories into your maven-proxy configuration and to create a separate maven-proxy installation that just contains the SNAPSHOTS. I think maven-proxy should have used the repository id as part of the mirror URL so that aggregation was never a side effect. So something like: http://maven-proxy:/central/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Apache Maven - http://maven.apache.org/ Better Builds with Maven - http://library.mergere.com/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This communication is the property of Qwest and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Maven Repository Manager: Will it avoid aggregation of repositories?
On 7/25/06, Allison, Bob [EMAIL PROTECTED] wrote: If you have multiple repositories mirrored, you need to keep the snapshot repositories away from the release repositories. In my case, I was using Proximity to mirror central but it also had several other repositories in its configuration including a couple snapshot repositories. My original thinking was that I could do this and avoid needing to specify individual repositories in my master POM. The problem with this arrangement is that if Proximity returns metadata for a snapshot release, Maven will be unable to retrieve the artifact and POM because central is defined as releases only. Maybe this is what was causing me problems. I had included snapshot repositories in maven-proxy and it is failing for the metadata reason you note above. I think this is another reason to avoid aggregation. To many unforseen problems. I want an internal mirror site that is seeded on an as needed basis by contacting the appropriate repositories. I should be able to do that without needing to configure the mirror if maven can pass on the URL request to the mirror program. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Maven Repository Manager: Will it avoid aggregation of repositories?
Allison, Bob wrote: Something else of note I discovered using Proximity: If you have multiple repositories mirrored, you need to keep the snapshot repositories away from the release repositories. In my case, I was using Proximity to mirror central but it also had several other repositories in its configuration including a couple snapshot repositories. My original thinking was that I could do this and avoid needing to specify individual repositories in my master POM. The problem with this arrangement is that if Proximity returns metadata for a snapshot release, Maven will be unable to retrieve the artifact and POM because central is defined as releases only. The problem here is that Proximity is nothing but a plain HTTP proxy and has no knowledge of Maven repositories which falls apart with two repositories with the same metadata file. The first one will probably cached and the latter will never be used. So if the first repository is the snapshot repository and the latter the release repository it will probably never download the release. This is a bit disappointing as Proximity is basically what maven-proxy always has been delivering. -- Trygve - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Maven Repository Manager: Will it avoid aggregation of repositories?
Proximity does only use the first metadata that is found in the list of sources. If central is defined first, it ALWAYS wins and you get release metadata by default. In this case, however, it found metadata for a plugin in a snapshot repository (it had never been released, so there was no data for it in central). Maven did not know that the metadata came from a snapshot repository (it was just using central because nothing else was configured) so when Maven tried to get the POM that went with the metadata it just retrieved it barfed all over itself because central is configured to only serve releases. If MRM, combined with a wildcard mirror in the settings, will allow Maven to retrieve the snapshot properly, then things will be OK. This means that MAVEN has to know the difference between the repositories, which probably means that all of the various repositories need to be configured in the project POM. MRM will probably also need to keep the metadata from each repository separate so that the resulting metadata in the local repository points to the correct remote repository (even though they are all mirrored in the same place). -Original Message- From: Trygve Laugstøl [mailto:[EMAIL PROTECTED] Sent: Tuesday, July 25, 2006 09:11 To: Maven Developers List Subject: Re: Maven Repository Manager: Will it avoid aggregation of repositories? Allison, Bob wrote: Something else of note I discovered using Proximity: If you have multiple repositories mirrored, you need to keep the snapshot repositories away from the release repositories. In my case, I was using Proximity to mirror central but it also had several other repositories in its configuration including a couple snapshot repositories. My original thinking was that I could do this and avoid needing to specify individual repositories in my master POM. The problem with this arrangement is that if Proximity returns metadata for a snapshot release, Maven will be unable to retrieve the artifact and POM because central is defined as releases only. The problem here is that Proximity is nothing but a plain HTTP proxy and has no knowledge of Maven repositories which falls apart with two repositories with the same metadata file. The first one will probably cached and the latter will never be used. So if the first repository is the snapshot repository and the latter the release repository it will probably never download the release. This is a bit disappointing as Proximity is basically what maven-proxy always has been delivering. -- Trygve - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This communication is the property of Qwest and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Maven Repository Manager: Will it avoid aggregation of repositories?
Hi, Not quite. Proximity is (almost) plain HTTP proxy, but IT DOES have knowledge about maven reposes (see repository browsing or the use of ?repositoryId=central URL parameter.), but only as separate storages. What it does not have is STATE, it cannot distuingish the two consecutive HTTP GET commands from same M2, once for metadata and once for POM. The solution should be in POM (Maven). ~t~ On 7/25/06, Trygve Laugstøl [EMAIL PROTECTED] wrote: The problem here is that Proximity is nothing but a plain HTTP proxy and has no knowledge of Maven repositories which falls apart with two repositories with the same metadata file. The first one will probably cached and the latter will never be used. So if the first repository is the snapshot repository and the latter the release repository it will probably never download the release. This is a bit disappointing as Proximity is basically what maven-proxy always has been delivering.
Re: Maven Repository Manager: Will it avoid aggregation of repositories?
Just to add: a planned feature of Proximity (awaited for RC2, but my every day job is pressing me lately) would be repository relocation. Behind the curtains, this feature would be implemented just by prefixing the repository namespace within Proximity, like: http://localhost:8080/px-webapp/repository - the current repo Px URL Lets say, you add central repo proxied from http://repo1.maven.org/maven2to px, prefixed with central, the result would be: http://localhost:8080/px-webapp/repository/central/... This way, you could shift/relocate your snapshot-able reposes with snapshot and set properly your POM/settings.xml and have separated reposes (with current M2 2.0.4!), have proxied ALL reposes and have a single proximity instance: central mirror - Px on http://localhost:8080/px-webapp/repository/central/ snapshot mirror - Px on http://localhost:8080/px-webapp/repository/snapshot/ etc. Any opinion is welcome. Thanx. ~t~ On 7/25/06, Barrie Treloar [EMAIL PROTECTED] wrote: In your pom you have specified the repository A and in your settings have configured A to point to your maven-proxy installation. Because maven-proxy aggregates the repositories the artifacts from the B and C repositories are available even though you have not defined them in your pom.
Re: Maven Repository Manager: Will it avoid aggregation of repositories?
Hi Barrie, While the current implementation is aggregating them, I'll keep this in mind when I review it in the next couple of days. It's probably best to keep the repositories separate, but to have a single index for the rest of the operations so it can appear as an aggregate (and be able to access a view over the aggregate from certain URLs). - Brett On 25/07/2006 1:37 PM, Barrie Treloar wrote: I just stumbled into a feature of maven-proxy which caused me some problems. Assume you have three repositories defined in your maven-proxy: A, B and C. In your pom you have specified the repository A and in your settings have configured A to point to your maven-proxy installation. Because maven-proxy aggregates the repositories the artifacts from the B and C repositories are available even though you have not defined them in your pom. Normally this would not show up as an issue. The problem manifests as soon as you start using SNAPSHOT version of artifacts, perhaps as part of doing some work on maven itself. Due to the aggregation you will find that you are using SNAPSHOT versions instead of the more stable released versions. My workaround was to not put SNAPSHOT repositories into your maven-proxy configuration and to create a separate maven-proxy installation that just contains the SNAPSHOTS. I think maven-proxy should have used the repository id as part of the mirror URL so that aggregation was never a side effect. So something like: http://maven-proxy:/central/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Apache Maven - http://maven.apache.org/ Better Builds with Maven - http://library.mergere.com/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
