Re: Karaf/Felix skips startlevel increments ... so how to delay fileinstall hot-deploy?

[Alex Heneveld]

Thanks very much JB.

To answer your question, we use a boot feature.  But consumers might 
extend with the deploy folder, specifically placing their extension 
bundles into the deploy/ folder _before_ the first start of Karaf.  It 
is the interplay of hot-deploy `fileinstall` (OSGi startlevel) with boot 
features/bundles start order that seems to be the problem.  The deploy/ 
folder is populated when Karaf starts, but is being read too early I 
think.  It is not to do with adding items to deploy/ after startup.

As you say, the feature resolver respects the order of the `<bundle 
start-level="xxx"...` items, but it seems to do that _independently_ of 
the OSGi FrameworkStartLevel.getStartLevel() -- so `fileinstall` detects 
that `felix.fileinstall.active.level=95` is satisfied and starts 
hot-deploying ... even though the boot features at startlevel 80 are not 
yet installed.

Would that make sense?

This feels contrary to this part of the OSGi spec:

> The Framework must then increase or decrease the active start level 
by 1 until the requested start level is reached.
> The Framework must not increase to the next active start level until 
all started bundles have returned from their BundleActivator.start method

Is there any way to make sure that `FrameworkStartLevel.getStartLevel()` 
is consistent with the start-level that the feature resolver is 
starting, on first/clean startup?  (On subsequent start, it obeys start 
levels.)  I've tried adding things to startup.properties but nothing 
I've tried seems to have any impact on what `MyBundleActivator.start() { 
log(FrameworkStartLevel.getStartLevel()) }` displays, it also shows 
`100` (the `beginning` level).

Or is there any other way to support consumers populating `deploy/ ` (as 
an easy way for them to add bundles) but ensuring they don't get 
activated until _after_ our boot features?

While I've got you JB any thoughts on the post-script question, where 
the`org.apache.servicemix.bundles.dom4j` bundle being added to the 
deploy/ directory (after startup) is causing Karaf to destroy then 
recreate lots of the blueprint services we created during boot, even 
though there aren't any obvious package dependencies -- our services 
start fine without that dom4j bundle?  I've turned on all the logging I 
can find and it's not displaying any messages about why things should be 
destroyed/uninstalled.  I know you wrote the bundle and it's really 
useful but weird it is wreaking havoc when hot-deployed. Maybe the 
optional imports of sun.xxx.xxx packages is forcing some major rewiring?

Many thanks,
Alex


On 23/11/2020 12:50, Jean-Baptiste Onofre wrote:
Hi Alex,

It depends the way you deploy your bundle (especially if you use the deploy 
folder).

The start level is respected in etc/startup.properties or in a feature (as the 
resolver can evaluate the order).
However, if you drop first a bundle with startLevel 90 and then, later, a 
bundle with startLevel 85, the first bundle will be deployed before the second 
one.

So, in your case, I would use a well formed features set.

Regards
JB

Le 23 nov. 2020 à 12:44, Alex Heneveld <a...@cloudsoft.io> a écrit :


hi Karaf devs-

i have a question about start-level behaviour in karaf/felix.  the osgi spec says that start-levels 
should increase 1 by 1 during startup [1].  this doesn't seem to be happening in a clean 
karaf-based environment.  what we observe is that startlevel jumps directly to the `beginning` 
level (100 in our case) before our boot bundles (at karaf.startlevel.bundle=80) are started -- a 
log(startlevel) in one of the boot bundle activators shows "100" on a clean install.  on 
a subsequent startup it shows "80":

     /bin/start
     ....
     2020-11-23T11:32:01,159 - INFO  175 o.a.b.u.o.OsgiActivator 
[tures-3-thread-1] Starting org.apache.brooklyn.utils-common [175], at start 
level 100, state 8
     ....

     /bin/stop
     ....
     2020-11-23T11:40:04,651 - INFO  175 o.a.b.u.o.OsgiActivator 
[FelixStartLevel] Stopping org.apache.brooklyn.utils-common [175]
     ....

     /bin/start
     ....
     2020-11-23T11:40:15,431 - INFO  175 o.a.b.u.o.OsgiActivator 
[FelixStartLevel] Starting org.apache.brooklyn.utils-common [175], at start 
level 80, state 8

we are on 4.2.8.  there are related issues [2] where this has been observed, 
but this particular issue wasn't the focus; other suggestions in those issues, 
to set `featuresBootAsynchronous=false` and to add items to 
`startup.properties` are not working for us (although maybe I'm not adding the 
right bundles to startup.properties).

i totally buy the argument that declarative dependencies are better in most 
cases, but i think this is one of those use cases where relying on start-levels 
is justified.  one actual problem we're trying to solve is preventing hot 
deployment until after all the boot bundles are started.  but because 
startlevel is jumping directly to 100, these settings don't work as expected:

     felix.fileinstall.start.level=95
     felix.fileinstall.active.level=95

we'd expect based on startlevel that fileinstall shouldn't start until boot 
bundles are installed (startlevel 80).  but instead fileinstall starts trying 
to hot-deploy right away, because startlevel jumped to 100, and because our 
boot bundles aren't yet available, it fails for a while.  once the boot bundles 
are installed, the hot-deploy bundles get wired in fine and it all works, and 
the start-levels as shown in `bundle:list` are as expected (80 and 95), but 
we'd ilke not to have all the failed hot-deployment attempts, and there might 
be hot-deployed bundles that users install which interfere with the boot wiring 
in ways we don't want (offering other services, etc).  so this seems a common 
and desirable use case for startlevels to be obeyed -- useful enough anyway 
that the fileinstall authors provided those settings!

we also have another related problem that this is blocking, that we would like 
some of our bundles not to do some initalization until user-supplied hot-deploy 
bundles are installed, as discussed on the Apache Brooklyn ML (and hence the 
cross-post).

so ... is there a way to have a karaf clean startup see our boot bundles and 
start levels and not jump to 100, so it completes startlevel 80 before 
startlevel 95 kicks in? ... or some other way to have fileinstall not run until 
our boot bundles are installed?

many thanks.

best
alex

[1] 
https://docs.osgi.org/specification/osgi.core/7.0.0/framework.startlevel.html 
-- section 9.3.1

The Framework must then increase or decrease the active start level by 1 until 
the requested start level is reached.
The Framework must not increase to the next active start level until all 
started bundles have returned from their BundleActivator.start method
[2] Related issues:

https://issues.apache.org/jira/browse/KARAF-4261
https://issues.apache.org/jira/browse/KARAF-4723
https://issues.apache.org/jira/browse/KARAF-4578
https://issues.apache.org/jira/browse/KARAF-4498



PS:  related curiousity, if i set `beginning=90` in the above case and then 
manually increase the startlevel to 100 later, it works, but i have the 
`org.apache.servicemix.bundles.dom4j` bundle in my deploy/ directory, and that 
makes Karaf destroy lots of the blueprint services we created during boot.  i 
can't see why they would, as a bundle it seems pretty simple, our other bundles 
don't use the dom4j classes, the logs don't give any reason why in this case, 
and if it's hot-deployed early we don't have any issues; so again I'm grateful 
if anyone has thoughts on why this would happen!