Re: [RT] Using FileInstall
On Tue, Aug 10, 2010 at 3:09 PM, Carsten Ziegeler cziege...@apache.org wrote: ...Therefore instead of enhancing FileInstall and add all our features, I think it makes more sense to leverage our OSGi installer and add file support to it - like we already have jcr support. I'm glad to hear that - we spent lots of time getting the OSGi installer to do the right thing with bundle updates. ...Basically all we need is a simple bundle doing file scans and triggering the osgi installer Exactly - our OSGi installer module does all the hard stuff. -Bertrand
Re: [RT] Using FileInstall
On 09.08.2010 21:14, Justin Edelson wrote: The problem is that the current scheme doesn't handle the case where the launchpad JAR is updated in a consistent manner. Whenever this is raised, the answer is to deploy bundles via sling:install or the web console. In my experience, this technique just doesn't scale past a single node. Perhaps Ace is a better answer for this, but I think updating the JAR should have a consistent effect. Well, actually, the current launchpad always has supported bundle updates by using a new Launchpad JAR file file. Because on startup the launcher checks, whether the standalone JAR file has a more recent last modification time than the previously launched JAR file in the same sling.home location. If so, the embedded bundles are unpackage. Next, the unpack location is scanned for updates since the last installation. If so, any updated bundles (bundles are never downgraded by this mechanism) or new bundles are installed. So, we already have this mechanis, and this should keep on working of course. It is my belief that the only way to fix this situation is by loading bundles from VFS layer we have rather than extracting the bundles. I could (of course) be wrong about this and I have not prioritized this (I've got a set of RightScripts which will get me by in the near term). But my fear is that by removing the code you're suggesting we remove, fixing this issue becomes that much harder. With using File Install and copying stuff to a well defined directory we enable other use cases. If people want to deploy or remove stuff at runtime, they can just drop/remove stuff from this directory. At minimum, I'd like to see us recommend against doing this. If we need to extract files from the archive, let's do that in a directory where we tell users not to touch and then create a 'dropins' directory for user-managed files. If users manipulate the extracted files, things can go haywire when Launchpad decides to re-extract the archive. Well, the goal of unpacking in effect was to enable administrators to put more bundles to be installed on next startup (or to be installed on first startup if Sling is distributed as a ZIP containing the launchpad and the bundles). Regards Felix There are other scenarios possible where for example the launchpad app does not contain all the bundles anymore, but they are picked up by File Install etc. We already have an alternate scenario - launchpad:run and launchpad:start use bundles (OK, artifacts) from the local Maven repository. And now that Aether has been released, I'm going to start experimenting with this in Launchpad itself. If you can give me about 10 days, I'll have a different ConfigAdmin solution. Sorry to be a stick in the mud about this Justin On Aug 9, 2010, at 11:38 AM, Carsten Ziegeler cziege...@apache.org wrote: I agree that having to copy the bundles (and other files) before they get installed is odd (especially as they are copied once more by the framework). But on the other hand this is imho a minor issue compared to what we gain. Let's assume that when using Sling disk space is not that important. The copying of the files is transparent to the user: a launchpad is created (either jar or war) which contains everything you want to install (bundles, configs etc). With this use case in mind it doesn't play a role if these things are copied, if File Install is used etc. It just works. If we would use File Install, we add a well known and used instrument to install stuff into an OSGi framework. So if people are known and used to FileInstall, they are already familiar with this stuff. We don't have to add all the support in Sling - which I think is a good thing; we already have code for bundles and deployment packages. And imho we really need support for config files. With using FileInstall we just reuse code. Code which is developed and used by a much wider base. With using File Install and copying stuff to a well defined directory we enable other use cases. If people want to deploy or remove stuff at runtime, they can just drop/remove stuff from this directory. There are other scenarios possible where for example the launchpad app does not contain all the bundles anymore, but they are picked up by File Install etc. So in short :) I don't see any downside of this except the disk space - but I see several advantages. Regards Carsten -- Carsten Ziegeler cziege...@apache.org
Re: [RT] Using FileInstall
Hi, just a heads up here; I looked more closely into Apache Felix FileInstall (which I should have done before...) and it seems that it doesn't meet our requirements. We have some logic regarding bundle versions implemented which comes into play for updates (only bundles with a higher version number is installed etc.) and removes. Adding this to FileInstall would be an incompatible behaviour. Then there is the missing start level support. Therefore instead of enhancing FileInstall and add all our features, I think it makes more sense to leverage our OSGi installer and add file support to it - like we already have jcr support. Basically all we need is a simple bundle doing file scans and triggering the osgi installer. Regards Carsten -- Carsten Ziegeler cziege...@apache.org
Re: [RT] Using FileInstall
Hi, On 06.08.2010 21:07, Justin Edelson wrote: Apologies in advance for reordering the bits from Carsten's email, but I think my response will make more sense this way. On Wed, Aug 4, 2010 at 9:53 AM, Carsten Ziegeler cziege...@apache.org wrote: Currently the launchpad copies the initial set of bundles to a file directory and creates a structure based on the start levels there. Bundles are then picked up from there and installed. Personally, I'd like to revisit this design. It seems unnecessary to copy the bundles to the file system vs. reading them directly from the InputStream. AFAICT, this was originally done to allow for additional bundles (i.e. ones not in the launchpad archive) to be placed in the startup directory, but that's exactly where FileInstall *should* be used. That is, as I understand it, the exact goal right now: Instead of duplicating implementation code with File Install we leverage File Install. This makes our Launchpad code base smaller and easier to understand and maintain while even extending the overall functionality. What I would like to see is that the ResourceProvider interface (the launchpad one, not the Sling API one) is treated like a virtual file system and that resources from this VFS are used directly by: 1) the bits in launchpad which install/start bundles 2) some ConfigAdmin bridge thingy 3) the jackrabbit.server bundle (in order to load the JR config from inside the archive) I agree, that the Jackrabbit Configuration file is a real issue right now. Because it currently is hard to easily overwrite in a fresh installation. We should look for a solution to this problem Yes... disk is cheap, but it seems wasteful that in a typical Sling install you end up with three copies of each bundle - one inside the archive, one in the startup directory, and one in the Felix cache. Yes, this ssems odd and overkill. But I valuate this not as critical as duplicating code. If this is a concern I suggest generating a ZIP (or compressed tar) file consisting of something like this: * raw launchpad with just the log and file install bundles * bundles and configurations to be installed After unpacking the package file can be discarded. (Of course I still think the all-encompassing executable JAR-file a major asset of Sling) while trying to implement handling of initial configurations for the ConfigAdmin within our launchpad, I noticed that this is more difficult than I thought. The final problem is a class loading issue as the launchpad does not contain the compendium class (for config admin). For the same reason, the current support of deployment packages is broken in launchpad. The other obvious solution is to embed ConfigAdmin in Launchpad. I'm sure there are reasons not to do this, but I suspect that these are somewhat academic in nature given how stable the Felix ConfigAdmin implementation appears to be. It is really about the Configuration Admin service package. Of course we will never integrate the Configuration Admin implementation with Launchpad Base (this would severly violate the separation of concerns principle). Still, I would discourage adding the Configuration Admin service package to launchpad base. Because this might cause problems should the API be extended in the future (or increase the version number as has been done in R 4.2). While trying to workarund these problems I came back to an idea I had a long time ago: instead of doing all these installs (bundle, configs, depl. pcks etc.) in our own code, why not use existing stuff? The FileInstall bundle from Apache Felix covers most of our use cases - there are some minor pieces missing. But we can add the required features there and avoid duplicate code by just using File Install. I'm not sure these missing pieces are so minor... last I looked, File Install doesn't support start levels. Although I'm not sure Sling *needs* start levels, they are useful from an administrative perspective. Sling definitely doesn't require start levels. We introduced them to workaround an implementation issue with early Felix framework releases with respect to finding a consistent classspace for imports/exports and to somehow create kind of a layered system setup. And yes, File Install does not have support for start levels yet and such support will have to be added. But I would assume, that we are not the only ones with a requirement to add start level support to file install. It seems to me that what's really needed is a refactoring of ConfigAdmin and FileInstall (and JCR Install for that matter) to provide a unified, standalone config parsing library. I believe I saw some discussion on felix-dev about having FileInstall use the config parsing bits from the ConfigAdmin bundle. This is better than having code duplication, but a standalone library still seems better to me. A standalone library with a single file (or two files) in it is probably not worth
Re: [RT] Using FileInstall
On second thought... s/fails/falls short for/ On Aug 9, 2010, at 7:27 AM, Justin Edelson justinedel...@gmail.com wrote: On Aug 9, 2010, at 4:59 AM, Felix Meschberger fmesc...@gmail.com wrote: Hi, On 06.08.2010 21:07, Justin Edelson wrote: Apologies in advance for reordering the bits from Carsten's email, but I think my response will make more sense this way. On Wed, Aug 4, 2010 at 9:53 AM, Carsten Ziegeler cziege...@apache.org wrote: Currently the launchpad copies the initial set of bundles to a file directory and creates a structure based on the start levels there. Bundles are then picked up from there and installed. Personally, I'd like to revisit this design. It seems unnecessary to copy the bundles to the file system vs. reading them directly from the InputStream. AFAICT, this was originally done to allow for additional bundles (i.e. ones not in the launchpad archive) to be placed in the startup directory, but that's exactly where FileInstall *should* be used. That is, as I understand it, the exact goal right now: Instead of duplicating implementation code with File Install we leverage File Install. AFAICT, the bundles (and now configs) will still be extracted from the launchpad artifact. I agree that if we're going to extract these files, we should use FileInstall. However, I don't agree that we should be doing this extraction in the first place. This makes our Launchpad code base smaller and easier to understand and maintain while even extending the overall functionality. What I would like to see is that the ResourceProvider interface (the launchpad one, not the Sling API one) is treated like a virtual file system and that resources from this VFS are used directly by: 1) the bits in launchpad which install/start bundles 2) some ConfigAdmin bridge thingy 3) the jackrabbit.server bundle (in order to load the JR config from inside the archive) I agree, that the Jackrabbit Configuration file is a real issue right now. Because it currently is hard to easily overwrite in a fresh installation. We should look for a solution to this problem Yes... disk is cheap, but it seems wasteful that in a typical Sling install you end up with three copies of each bundle - one inside the archive, one in the startup directory, and one in the Felix cache. Yes, this ssems odd and overkill. But I valuate this not as critical as duplicating code. Agreed. But what I think we both value more than overkill and code duplication is ease of comprehension and this is where I think the current scheme fails us. Justin If this is a concern I suggest generating a ZIP (or compressed tar) file consisting of something like this: * raw launchpad with just the log and file install bundles * bundles and configurations to be installed After unpacking the package file can be discarded. (Of course I still think the all-encompassing executable JAR-file a major asset of Sling) while trying to implement handling of initial configurations for the ConfigAdmin within our launchpad, I noticed that this is more difficult than I thought. The final problem is a class loading issue as the launchpad does not contain the compendium class (for config admin). For the same reason, the current support of deployment packages is broken in launchpad. The other obvious solution is to embed ConfigAdmin in Launchpad. I'm sure there are reasons not to do this, but I suspect that these are somewhat academic in nature given how stable the Felix ConfigAdmin implementation appears to be. It is really about the Configuration Admin service package. Of course we will never integrate the Configuration Admin implementation with Launchpad Base (this would severly violate the separation of concerns principle). Still, I would discourage adding the Configuration Admin service package to launchpad base. Because this might cause problems should the API be extended in the future (or increase the version number as has been done in R 4.2). While trying to workarund these problems I came back to an idea I had a long time ago: instead of doing all these installs (bundle, configs, depl. pcks etc.) in our own code, why not use existing stuff? The FileInstall bundle from Apache Felix covers most of our use cases - there are some minor pieces missing. But we can add the required features there and avoid duplicate code by just using File Install. I'm not sure these missing pieces are so minor... last I looked, File Install doesn't support start levels. Although I'm not sure Sling *needs* start levels, they are useful from an administrative perspective. Sling definitely doesn't require start levels. We introduced them to workaround an implementation issue with early Felix framework releases with respect to finding a consistent classspace for imports/exports and to somehow create kind of a layered system setup. And yes, File Install does
Re: [RT] Using FileInstall
I agree that having to copy the bundles (and other files) before they get installed is odd (especially as they are copied once more by the framework). But on the other hand this is imho a minor issue compared to what we gain. Let's assume that when using Sling disk space is not that important. The copying of the files is transparent to the user: a launchpad is created (either jar or war) which contains everything you want to install (bundles, configs etc). With this use case in mind it doesn't play a role if these things are copied, if File Install is used etc. It just works. If we would use File Install, we add a well known and used instrument to install stuff into an OSGi framework. So if people are known and used to FileInstall, they are already familiar with this stuff. We don't have to add all the support in Sling - which I think is a good thing; we already have code for bundles and deployment packages. And imho we really need support for config files. With using FileInstall we just reuse code. Code which is developed and used by a much wider base. With using File Install and copying stuff to a well defined directory we enable other use cases. If people want to deploy or remove stuff at runtime, they can just drop/remove stuff from this directory. There are other scenarios possible where for example the launchpad app does not contain all the bundles anymore, but they are picked up by File Install etc. So in short :) I don't see any downside of this except the disk space - but I see several advantages. Regards Carsten -- Carsten Ziegeler cziege...@apache.org
Re: [RT] Using FileInstall
Just forgot: It's correct that FileInstall lacks some features - right now this is just start level support; adding a similar support like we have atm in Sling should be easy and straightforward. Regards Carsten -- Carsten Ziegeler cziege...@apache.org
Re: [RT] Using FileInstall
The problem is that the current scheme doesn't handle the case where the launchpad JAR is updated in a consistent manner. Whenever this is raised, the answer is to deploy bundles via sling:install or the web console. In my experience, this technique just doesn't scale past a single node. Perhaps Ace is a better answer for this, but I think updating the JAR should have a consistent effect. It is my belief that the only way to fix this situation is by loading bundles from VFS layer we have rather than extracting the bundles. I could (of course) be wrong about this and I have not prioritized this (I've got a set of RightScripts which will get me by in the near term). But my fear is that by removing the code you're suggesting we remove, fixing this issue becomes that much harder. With using File Install and copying stuff to a well defined directory we enable other use cases. If people want to deploy or remove stuff at runtime, they can just drop/remove stuff from this directory. At minimum, I'd like to see us recommend against doing this. If we need to extract files from the archive, let's do that in a directory where we tell users not to touch and then create a 'dropins' directory for user-managed files. If users manipulate the extracted files, things can go haywire when Launchpad decides to re-extract the archive. There are other scenarios possible where for example the launchpad app does not contain all the bundles anymore, but they are picked up by File Install etc. We already have an alternate scenario - launchpad:run and launchpad:start use bundles (OK, artifacts) from the local Maven repository. And now that Aether has been released, I'm going to start experimenting with this in Launchpad itself. If you can give me about 10 days, I'll have a different ConfigAdmin solution. Sorry to be a stick in the mud about this Justin On Aug 9, 2010, at 11:38 AM, Carsten Ziegeler cziege...@apache.org wrote: I agree that having to copy the bundles (and other files) before they get installed is odd (especially as they are copied once more by the framework). But on the other hand this is imho a minor issue compared to what we gain. Let's assume that when using Sling disk space is not that important. The copying of the files is transparent to the user: a launchpad is created (either jar or war) which contains everything you want to install (bundles, configs etc). With this use case in mind it doesn't play a role if these things are copied, if File Install is used etc. It just works. If we would use File Install, we add a well known and used instrument to install stuff into an OSGi framework. So if people are known and used to FileInstall, they are already familiar with this stuff. We don't have to add all the support in Sling - which I think is a good thing; we already have code for bundles and deployment packages. And imho we really need support for config files. With using FileInstall we just reuse code. Code which is developed and used by a much wider base. With using File Install and copying stuff to a well defined directory we enable other use cases. If people want to deploy or remove stuff at runtime, they can just drop/remove stuff from this directory. There are other scenarios possible where for example the launchpad app does not contain all the bundles anymore, but they are picked up by File Install etc. So in short :) I don't see any downside of this except the disk space - but I see several advantages. Regards Carsten -- Carsten Ziegeler cziege...@apache.org
Re: [RT] Using FileInstall
Hi, +1 to using fileinstall One point: I would extend the fixed bundles to be installed by the commons.log bundle, which brings LogService and Logging APIs. This should IMHO really be the very first bundle to be installed. Regards Felix On 04.08.2010 15:53, Carsten Ziegeler wrote: Hi, while trying to implement handling of initial configurations for the ConfigAdmin within our launchpad, I noticed that this is more difficult than I thought. The final problem is a class loading issue as the launchpad does not contain the compendium class (for config admin). For the same reason, the current support of deployment packages is broken in launchpad. While trying to workaround these problems I came back to an idea I had a long time ago: instead of doing all these installs (bundle, configs, depl. pcks etc.) in our own code, why not use existing stuff? The FileInstall bundle from Apache Felix covers most of our use cases - there are some minor pieces missing. But we can add the required features there and avoid duplicate code by just using File Install. Currently the launchpad copies the initial set of bundles to a file directory and creates a structure based on the start levels there. Bundles are then picked up from there and installed. Now, I would like to extend this and just copy all files over, bundles will get into the same location, all other files go into a single install directory. The only bundle which we will really install by launchpad is the File Install bundle - which will be configured to listen for the directory where everything got copied over. File Install picks up the bundles with the start levels, the configs and whatnot and just installs the stuff for us. As the directory is watched by File Install, changes are reflected as well. So basically we have the same functionality with less code :) I think getting all of this done for Sling 6 is too much. Therefore I would like to use File Install for everything else but bundles as a start: copying over those files to the install directory and start file install configured with that directory. Note, that File Install is mainly used to get initial configuration into the system - so it's more bootstrapping - for everything else jcr install can be used. Or File Install is nice for all those scenarios where no repository is used :) WDYT? Carsten
Re: [RT] Using FileInstall
On Wed, Aug 4, 2010 at 3:53 PM, Carsten Ziegeler cziege...@apache.org wrote: ...Now, I would like to extend this and just copy all files over, bundles will get into the same location, all other files go into a single install directory. The only bundle which we will really install by launchpad is the File Install bundle - which will be configured to listen for the directory where everything got copied over Sounds good. Too bad we're still duplicating a big part of the FileInstall logic in our osgi.installer module, it would be cool to merge both things at some point. -Bertrand
Re: [RT] Using FileInstall
Bertrand Delacretaz wrote On Wed, Aug 4, 2010 at 3:53 PM, Carsten Ziegeler cziege...@apache.org wrote: ...Now, I would like to extend this and just copy all files over, bundles will get into the same location, all other files go into a single install directory. The only bundle which we will really install by launchpad is the File Install bundle - which will be configured to listen for the directory where everything got copied over Sounds good. Too bad we're still duplicating a big part of the FileInstall logic in our osgi.installer module, it would be cool to merge both things at some point. Yepp, it is on my todo list :) Regards Carsten -- Carsten Ziegeler cziege...@apache.org
Re: [RT] Using FileInstall
Ok, I think the whole thing could work like this:
- resources/corebundles : contains the bundles which are installed by
the Sling base - this is usually just the log bundle and the file
install bundle.
- resources/bundles/{startLevel}/* : contains the bundles in the various
start levels - this is currently handled by Sling but will be by
FileInstall.
- resources/install : contains all other installables like deployment
packages and configurations
Like before all these are copied on startup over to the sling/startup
directory and file install is configured to watch the bundles and
install directory.
WDYT?
Regards
Carsten
--
Carsten Ziegeler
cziege...@apache.org
Re: [RT] Using FileInstall
Apologies in advance for reordering the bits from Carsten's email, but I think my response will make more sense this way. On Wed, Aug 4, 2010 at 9:53 AM, Carsten Ziegeler cziege...@apache.org wrote: Currently the launchpad copies the initial set of bundles to a file directory and creates a structure based on the start levels there. Bundles are then picked up from there and installed. Personally, I'd like to revisit this design. It seems unnecessary to copy the bundles to the file system vs. reading them directly from the InputStream. AFAICT, this was originally done to allow for additional bundles (i.e. ones not in the launchpad archive) to be placed in the startup directory, but that's exactly where FileInstall *should* be used. What I would like to see is that the ResourceProvider interface (the launchpad one, not the Sling API one) is treated like a virtual file system and that resources from this VFS are used directly by: 1) the bits in launchpad which install/start bundles 2) some ConfigAdmin bridge thingy 3) the jackrabbit.server bundle (in order to load the JR config from inside the archive) Yes... disk is cheap, but it seems wasteful that in a typical Sling install you end up with three copies of each bundle - one inside the archive, one in the startup directory, and one in the Felix cache. while trying to implement handling of initial configurations for the ConfigAdmin within our launchpad, I noticed that this is more difficult than I thought. The final problem is a class loading issue as the launchpad does not contain the compendium class (for config admin). For the same reason, the current support of deployment packages is broken in launchpad. The other obvious solution is to embed ConfigAdmin in Launchpad. I'm sure there are reasons not to do this, but I suspect that these are somewhat academic in nature given how stable the Felix ConfigAdmin implementation appears to be. While trying to workarund these problems I came back to an idea I had a long time ago: instead of doing all these installs (bundle, configs, depl. pcks etc.) in our own code, why not use existing stuff? The FileInstall bundle from Apache Felix covers most of our use cases - there are some minor pieces missing. But we can add the required features there and avoid duplicate code by just using File Install. I'm not sure these missing pieces are so minor... last I looked, File Install doesn't support start levels. Although I'm not sure Sling *needs* start levels, they are useful from an administrative perspective. It seems to me that what's really needed is a refactoring of ConfigAdmin and FileInstall (and JCR Install for that matter) to provide a unified, standalone config parsing library. I believe I saw some discussion on felix-dev about having FileInstall use the config parsing bits from the ConfigAdmin bundle. This is better than having code duplication, but a standalone library still seems better to me. All that said, I'm on vacation and haven't been able to fully process this. But I did want to raise my concerns now before this got too far. Justin Note, that File Install is mainly used to get initial configuration into the system - so it's more bootstrapping - for everything else jcr install can be used. Or File Install is nice for all those scenarios where no repository is used :) WDYT? Carsten -- Carsten Ziegeler cziege...@apache.org
