Re: [equinox-dev] Jetty and commons logging
On Jan 6, 2008 1:18 AM, Jeff McAffer [EMAIL PROTECTED] wrote: I won't disagree and would certainly welcome anyone who wanted to create/maintain such a repo. It seems that someone has done this at maven.org. The Maven user community should encourage that person to continue doing the work. I don't really see eclipse.org or the projects hosting another copy of all their bundles laid out in the standard Maven dir structure. Our mirrors would be upset at the doubling in size and we have to maintain the update site structure for the Eclipse user community anyway. As I understand Maven, it is at least in theory possible to define different kinds of repos so, for example, Eclipse update sites could be directly accessed from Maven. No idea what is involved in that but it feels like the right kind of direction. The problem with the Eclipse downloads is that there's actually no real way of knowing where to look for something. Quite apart from the arbitrary tools/platform/technology project splits, each directory is versioned in an S-3.4M2-200709210919 directory whose name is computed based on the time that something was run, rather than anything detectable. In order to provide my 'direct download links' I had to manually grep for the random-date-string in order to update my links each time a new drop was released. Not only that, but eclipse.org doesn't provide any really useful way of finding out what plugins are hosted, or even where. It also ends up wasting a *heap* of bandwith; the -win32-win32-, -win32-wpf-, -linux-gtk-, -linux-motif-, combos are all pretty much 95%+ the same, with only a few native bits different from each of them. There's no 'one' place to download the individual plugins. Even the update sites aren't helpful. You can't browse to them, and even when you do (e.g. via a mirror) it's not clear as to which plugin is stored in which location. Lastly, there's no metadata available. One of the things which will have been observed from the P2 work is that it is necessary to have a (small) downloadable metadata file that can be downloaded without needing to download the bits that the metadata refers to. In Maven, this is the POM, and without it, a maven repo is just an HTTP store. Whether the P2 work can be leveraged to have a Maven-P2 link (or vice versa) is something that could be investigated, but the problem is that if you rely on e.g. additions to the maven repository then they are either done sporadically or erroneously. In the case of Eclipse, several (different) people have uploaded variants of some of the core Eclipse plugins, but then never get updated as newer versions are available. If this process were baked into the Eclipse release process, more people might be tempted to build upon OSGi systems than they are at the moment. For those that are interested in building OGSi with Maven, then take a look at Pax Construct: http://wiki.ops4j.org/confluence/display/ops4j/Pax+Construct There's also Pax Runner, which allows bundles to be launched inside different OSGi engines, such as Equinox and Felix: http://wiki.ops4j.org/confluence/display/ops4j/Pax+Runner Alex ___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev
Re: [equinox-dev] Jetty and commons logging
Alex Blewitt wrote: Even the update sites aren't helpful. You can't browse to them, and even when you do (e.g. via a mirror) it's not clear as to which plugin is stored in which location. While I agree that there's certainly room for improvement, I don't think it's that bad. You will find most things on the Europa update site (or the Ganymede staging site if you want to be bleeding edge). You can create a target platform suitable for your builds using the current update manager and/or you can make use of the Orbit map files. The structure is not perfect but its definitely usable for automating downloads and builds. Lastly, there's no metadata available. Yes there is. You have the site.xml in every update site. All features have a feature.xml and all OSGi bundles have a manifest.mf. Getting to that meta-data can sometimes be a pain since it's embedded but features are most often very small by nature. The manifest.mf is a bit more cumbersome though. A tool that does some caching of what it reads can do a fairly good job of resolving and downloading without wasting too much bandwidth. - thomas ___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev
Re: [equinox-dev] Jetty and commons logging
Hey Alex, Please note that I started my reply by saying I don't disagree. The point is that the team available to do this sort of work is small and the diskspace/mirror capacity to do it is similarly restricted. I'm not happy about it either. As with many of these things, if there are folks in the community who care deeply about this and want to do the work/supply the space, we'd be happy to work with them. You are also correct in observing that in the p2 work we did indeed decide to have separately downloadable repository (metadata and artifact) indexing. Further our general approach has been to allow for individual bundle download/install rather than whole feature management. These moves have been a big win and should go a long way towards addressing the valid usecases you point out. I completley agree that the update site structure is lacking in these areas but disagree with you that Maven addresses the issues. The POM file certainly can represent the metadata in a bundle but its primary focus is to drive the build. So, for example, if the build generates the dependency information into the output manifest.mf (e.g., using bnd or mangen style processing) then that dependency info is not going to be in the POM. The standard maven repo does not have an index so it still does not help you find something unless you know exact ids and version numbers. If you have that information then you can get it from a Maven repo or Update site equally well. (yes, I realize that there is a Maven repo indexer. This is akin to the update site digest but with a finer grain) I don't mean to slam Maven at all. It is a fine tool and many people like it. In the course of the p2 work we have learned some things from Maven and a careful eye will observe some Mavenesque characteristics in p2. The standard Maven repo structure however does not meet our requirements (we spent quite a bit of time with various Maven folks going over the possibilities here). As a design point we opted out of defining one repo format in favour of allowing people to add their own. This opens the door for people with a Maven (or other) bent to add the support they need. Similarly, our default repo structure is not that complicated so someone in the Maven world should be able to write a Maven repo class to manage p2's simple repos or even use p2 API to write them (thus mapping onto any p2 repo). Anyway, we are way off topic here. It suffices to say that the orginal poster's problems were caused by someone creating and publishing a faulty commons.logging bundle. I doubt that was created by the eclipse team or if it was, it was a prerelease version (the Europa version has the necessary dependency version numbers). In the ideal world it would actually be the Apache folks who produced these bundles and we would all just consume them. But I'm ahead of myself... Jeff Alex Blewitt [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 01/06/2008 01:35 PM Please respond to Equinox development mailing list equinox-dev@eclipse.org To Equinox development mailing list equinox-dev@eclipse.org cc Subject Re: [equinox-dev] Jetty and commons logging On Jan 6, 2008 1:18 AM, Jeff McAffer [EMAIL PROTECTED] wrote: I won't disagree and would certainly welcome anyone who wanted to create/maintain such a repo. It seems that someone has done this at maven.org. The Maven user community should encourage that person to continue doing the work. I don't really see eclipse.org or the projects hosting another copy of all their bundles laid out in the standard Maven dir structure. Our mirrors would be upset at the doubling in size and we have to maintain the update site structure for the Eclipse user community anyway. As I understand Maven, it is at least in theory possible to define different kinds of repos so, for example, Eclipse update sites could be directly accessed from Maven. No idea what is involved in that but it feels like the right kind of direction. The problem with the Eclipse downloads is that there's actually no real way of knowing where to look for something. Quite apart from the arbitrary tools/platform/technology project splits, each directory is versioned in an S-3.4M2-200709210919 directory whose name is computed based on the time that something was run, rather than anything detectable. In order to provide my 'direct download links' I had to manually grep for the random-date-string in order to update my links each time a new drop was released. Not only that, but eclipse.org doesn't provide any really useful way of finding out what plugins are hosted, or even where. It also ends up wasting a *heap* of bandwith; the -win32-win32-, -win32-wpf-, -linux-gtk-, -linux-motif-, combos are all pretty much 95%+ the same, with only a few native bits different from each of them. There's no 'one' place to download the individual plugins. Even the update sites aren't helpful. You can't
[equinox-dev] Maven and all that [was: Jetty and commons logging]
[redoing the subject to be more accurate as we are well and truly off the original topic...] That doc talks about releases but does not define the term. Do you know what the expectation is? One of our builds can be 200MB or so and we end up running at least one and sometimes multiple a day in some (unhappy) cases. Last I knew ibiblio was not keen on this sort of publishing. Our weekly integration builds would be a candidate though they might prefer only the 6week milestones. Either way the process is the same. We run several such builds during the integration/milestone week and in the end declare one of them as good and promote it. We do not apriori know which build is going to be the one as a mess o' testing is done after the build (automated and manual). A publishing step such as discussed here would have to be after the fact and so cannot be built into the build itself. What is really needed is a tool that would take an Eclipse build (e.g., update site or download zip or ...) and fire it into the desired repo in the desired form. This would accommodate the myriad of different build mechanisms/processes that exist today at Eclipse and allow anyone who wanted the content in another repo to do so without having to depend on the particular project build team/process. With this in hand we could talk about getting particular build teams to call the tool when they promote a build. Alternatively and more generally it would be interesting to have an RSS feed (forgive me Kim) for build promotions. External parties can then start sucking content as soon as it is released. Jeff Hal Hildebrand [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 01/06/2008 09:08 PM Please respond to Equinox development mailing list equinox-dev@eclipse.org To Equinox development mailing list equinox-dev@eclipse.org cc Subject Re: [equinox-dev] Jetty and commons logging I guess I'm a little confused... There's a few sites which will do the mirroring for you, like ibiblio.org. Ibiblio, and the myriad of mirrors for the site, host a plethora of maven artifacts for projects all over. I'm not completely up on the mechanisms for publishing to these repositories - i.e. what are the protocols, contacts, etc - but considering that plenty of projects and code bases publish there means it can't be too hard to do. Further, tools exist for Ant and Maven to make this publishing part of the build release process. Or, at the very least, a target of the build which can be manually called when appropriate. So, it seems like everyone could be happy. Those actually build outside of Eclipse (strange, I know) could have access to Eclipse artifacts from a maven repository hosted by Ibiblio rather than having to provide your own space, and you Eclipse folks would have this folded into your build in a painless fashion. This link has the Apache documentation on publishing to their site which is mirrored by Ibiblio: http://www.apache.org/dev/release-publishing.html Although this wouldn't be what you'd be doing, I'm fairly certain the process would be similar. Hope this helps all involved... On Jan 6, 2008, at 5:29 PM, Jeff McAffer wrote: Hey Alex, Please note that I started my reply by saying I don't disagree. The point is that the team available to do this sort of work is small and the diskspace/mirror capacity to do it is similarly restricted. I'm not happy about it either. As with many of these things, if there are folks in the community who care deeply about this and want to do the work/supply the space, we'd be happy to work with them. You are also correct in observing that in the p2 work we did indeed decide to have separately downloadable repository (metadata and artifact) indexing. Further our general approach has been to allow for individual bundle download/install rather than whole feature management. These moves have been a big win and should go a long way towards addressing the valid usecases you point out. I completley agree that the update site structure is lacking in these areas but disagree with you that Maven addresses the issues. The POM file certainly can represent the metadata in a bundle but its primary focus is to drive the build. So, for example, if the build generates the dependency information into the output manifest.mf (e.g., using bnd or mangen style processing) then that dependency info is not going to be in the POM. The standard maven repo does not have an index so it still does not help you find something unless you know exact ids and version numbers. If you have that information then you can get it from a Maven repo or Update site equally well. (yes, I realize that there is a Maven repo indexer. This is akin to the update site digest but with a finer grain) I don't mean to slam Maven at all. It is a fine tool and many people like it. In the course of the p2 work we have learned some things from Maven and a careful eye
Re: [equinox-dev] Maven and all that [was: Jetty and commons logging]
I think the determination is entirely up to you. Maven has the concept of SNAPSHOTS, which are incrementals of a release. Myself, I wouldn't bother with these, as just having officially released artifacts would be enough to start with and given the prodigious output... WRT your suggestion at the end, I think that's a great idea given what you point out about your process. And it sounds like an excellent project for someone who wants to see the artifacts made available in Maven. Myself, I'll see what I can do about investigating the building of such a tool. +1 on the RSS feed. That would be bitchin'. On Jan 6, 2008, at 6:45 PM, Jeff McAffer wrote: [redoing the subject to be more accurate as we are well and truly off the original topic...] That doc talks about releases but does not define the term. Do you know what the expectation is? One of our builds can be 200MB or so and we end up running at least one and sometimes multiple a day in some (unhappy) cases. Last I knew ibiblio was not keen on this sort of publishing. Our weekly integration builds would be a candidate though they might prefer only the 6week milestones. Either way the process is the same. We run several such builds during the integration/milestone week and in the end declare one of them as good and promote it. We do not apriori know which build is going to be the one as a mess o' testing is done after the build (automated and manual). A publishing step such as discussed here would have to be after the fact and so cannot be built into the build itself. What is really needed is a tool that would take an Eclipse build (e.g., update site or download zip or ...) and fire it into the desired repo in the desired form. This would accommodate the myriad of different build mechanisms/processes that exist today at Eclipse and allow anyone who wanted the content in another repo to do so without having to depend on the particular project build team/ process. With this in hand we could talk about getting particular build teams to call the tool when they promote a build. Alternatively and more generally it would be interesting to have an RSS feed (forgive me Kim) for build promotions. External parties can then start sucking content as soon as it is released. Jeff Hal Hildebrand [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 01/06/2008 09:08 PM Please respond to Equinox development mailing list equinox-dev@eclipse.org To Equinox development mailing list equinox-dev@eclipse.org cc Subject Re: [equinox-dev] Jetty and commons logging I guess I'm a little confused... There's a few sites which will do the mirroring for you, like ibiblio.org. Ibiblio, and the myriad of mirrors for the site, host a plethora of maven artifacts for projects all over. I'm not completely up on the mechanisms for publishing to these repositories - i.e. what are the protocols, contacts, etc - but considering that plenty of projects and code bases publish there means it can't be too hard to do. Further, tools exist for Ant and Maven to make this publishing part of the build release process. Or, at the very least, a target of the build which can be manually called when appropriate. So, it seems like everyone could be happy. Those actually build outside of Eclipse (strange, I know) could have access to Eclipse artifacts from a maven repository hosted by Ibiblio rather than having to provide your own space, and you Eclipse folks would have this folded into your build in a painless fashion. This link has the Apache documentation on publishing to their site which is mirrored by Ibiblio: http://www.apache.org/dev/release-publishing.html Although this wouldn't be what you'd be doing, I'm fairly certain the process would be similar. Hope this helps all involved... On Jan 6, 2008, at 5:29 PM, Jeff McAffer wrote: Hey Alex, Please note that I started my reply by saying I don't disagree. The point is that the team available to do this sort of work is small and the diskspace/mirror capacity to do it is similarly restricted. I'm not happy about it either. As with many of these things, if there are folks in the community who care deeply about this and want to do the work/supply the space, we'd be happy to work with them. You are also correct in observing that in the p2 work we did indeed decide to have separately downloadable repository (metadata and artifact) indexing. Further our general approach has been to allow for individual bundle download/install rather than whole feature management. These moves have been a big win and should go a long way towards addressing the valid usecases you point out. I completley agree that the update site structure is lacking in these areas but disagree with you that Maven addresses the issues. The POM file certainly can represent the metadata in a bundle but its primary focus is to drive
Re: [equinox-dev] Maven and all that [was: Jetty and commons logging]
With the discussion of meta data, I'm surprised no one mentioned OSGi repositories ... is there a reason for that? (Other than me missing the whole point from quickly scanning this mailing list :) It's not heavily advertised, I guess, but we actually produce OSGi repository meta data for our yearly releases, such as http://download.eclipse.org/releases/europa/repository.zip And this in turn becomes part of the (browsable) OSGi repository at http://www2.osgi.org/obr/browse?_xml=1cmd=repository Just FYI, FWIW ___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev
