Re: same configuration registered with multiple PIDs in ManagedServiceFactory

[ran...@opennms.org]

Looking at a diff between 4.3.6 and 4.3.7, the only thing that seems relevant 
is this change, maybe the issue with multiple threads writing the config at the 
same time needs to be solved in a different way?

My guess is this fixed the race condition by making it atomic, but it doesn’t 
stop it from happening twice in quick succession.

commit 1221b0158d2494523cf94cc3f223bd552a2467c7
Author: Grzegorz Grzybek gr.grzy...@gmail.com
Date:   Wed Feb 9 11:53:33 2022 +0100

[KARAF-7389] Prevent two threads (feature installer, CM Event Dispatcher 
through fileinstall) writing the same config file

(cherry picked from commit f3260d5ab641cdbf1bbd594875c07d974ed470a0)

diff --git 
a/features/core/src/main/java/org/apache/karaf/features/internal/service/FeatureConfigInstaller.java
 
b/features/core/src/main/java/org/apache/karaf/features/internal/service/FeatureConfigInstaller.java
index ba46eb3a2d..40bb666a34 100644
--- 
a/features/core/src/main/java/org/apache/karaf/features/internal/service/FeatureConfigInstaller.java
+++ 
b/features/core/src/main/java/org/apache/karaf/features/internal/service/FeatureConfigInstaller.java
@@ -139,2 +139,3 @@ public class FeatureConfigInstaller {
 properties.put(CONFIG_KEY, cid.pid);
+cfg.update(cfgProps);
 if (storage != null && configCfgStore) {
@@ -142,3 +143,2 @@ public class FeatureConfigInstaller {
 }
-cfg.update(cfgProps);
 try {
@@ -327,7 +327,9 @@ public class FeatureConfigInstaller {
 if (!cfgFile.exists()) {
+File tmpCfgFile = File.createTempFile(cfgFile.getName(), 
".tmp", cfgFile.getParentFile());
 if (jsonFormat) {
-Configurations.buildWriter().build(new 
FileWriter(cfgFile)).writeConfiguration(convertToDict(props));
+Configurations.buildWriter().build(new 
FileWriter(tmpCfgFile)).writeConfiguration(convertToDict(props));
 } else {
-props.save(cfgFile);
+props.save(tmpCfgFile);
 }
+tmpCfgFile.renameTo(cfgFile);
 } else {

From: Jesse White 
Date: Tuesday, May 30, 2023 at 7:11 PM
To: user@karaf.apache.org 
Subject: same configuration registered with multiple PIDs in 
ManagedServiceFactory
Hey folks,

I've encountered some odd behavior with Karaf 4.4.3, and I'd like to confirm if 
this is a bug or if there are some settings I can tune to alter the behavior.

Here's how to reproduce:

  *   Start Karaf 4.4.3 w/ JDK 11
  *   Install the managed factory example

 *   feature:repo-add 
mvn:org.apache.karaf.examples/karaf-config-example-features/4.4.3/xml

 *   feature:install karaf-config-example-managed-factory

  *   Copy the attached 'config.xml' file to the 'deploy/' directory
  *   Wait about 5 seconds, and notice the following output to the console
New configuration with pid 
org.apache.karaf.example.config.04388423-2d46-4308-8214-3dcd1e0b8fd0
key1 = value1
key2 = value2
org.apache.karaf.features.configKey = org.apache.karaf.example.config-abc
service.factoryPid = org.apache.karaf.example.config
service.pid = 
org.apache.karaf.example.config.04388423-2d46-4308-8214-3dcd1e0b8fd0
18:42:02.131 INFO [features-3-thread-1] Done.
18:42:10.999 INFO [fileinstall-/Users/jesse/labs/karaf/apache-karaf-4.4.3/etc] 
Creating configuration {org.apache.karaf.example.config~abc} from 
/Users/jesse/labs/karaf/apache-karaf-4.4.3/etc/org.apache.karaf.example.config-abc.cfg
New configuration with pid org.apache.karaf.example.config~abc
felix.fileinstall.filename = 
file:/Users/jesse/labs/karaf/apache-karaf-4.4.3/etc/org.apache.karaf.example.config-abc.cfg
key1 = value1
key2 = value2
org.apache.karaf.features.configKey = org.apache.karaf.example.config-abc
service.factoryPid = org.apache.karaf.example.config
service.pid = org.apache.karaf.example.config~abc


Note that the single config results in multiple callbacks under two separate 
pids. This seems isolated to cases where the deploy/ folder is used to write 
the config, and doesn't happen when the configuration is manually placed in 
etc/.

Following the same instructions with Karaf 4.3.6 has the expected behavior. 
Karaf 4.3.7 and later experience this issue though.

Any ideas?

Thanks,
Jesse



CONFIDENTIALITY NOTICE
This e-mail message and any attachments are only for the use of the intended 
recipient and may contain information that is privileged, confidential or 
exempt from disclosure under applicable law. If you are not the intended 
recipient, any disclosure, distribution or other use of this e-mail message or 
attachments is prohibited. If you have received this e-mail message in error, 
please delete and notify the sender immediately. Thank you.

Re: Feature XML generation

[ran...@opennms.org]

Unless you have particularly complicated needs, I would think that generating 
is better in theory. It means you only have to maintain your dependency tree 
once, rather than twice.

However, currently generation is in that valley where it’s useful for basic 
cases, but not robust enough to be able to declaratively say how to handle 
“problem” dependencies, or to layer more complicated things like conditionals 
or capabilities on top.

I’m running into a similar issue now with the features-add-to-repository mojo, 
where it doesn’t support the same excluding/blacklisting that the assembly mojo 
does. (I’m trying to upgrade our camel finally, a little at a time, but the 
2.20 feature files have a broken cxf dependency and it’s making things 
difficult. I’m about to just make an assembly and throw everything away but the 
system/ directory.)

From: Ryan Moquin 
Date: Monday, May 15, 2023 at 1:05 PM
To: user@karaf.apache.org 
Subject: Re: Feature XML generation
I like the idea of keeping it around to at least assist in the generation of 
features.  At that point, I don't mind tweaking and customizing it, it's just a 
pain to have to do it all by hand when there are a lot of dependencies.


On Sun, May 14, 2023 at 3:05 PM Jean-Baptiste Onofré 
mailto:j...@nanthrax.net>> wrote:
Hi Paul,

fair enough, but as soon as you need a "complex/detailed" features
repo, you need a XML template, so basically writing the repo by hand
:)

Regards
JB

On Thu, May 11, 2023 at 8:50 PM Paul Fraser 
mailto:pa...@qnenet.com>> wrote:
>
> Hi JB,
>
> "I'm thinking about removing the generate mojo as some point."
>
> Please do not remove it, it is most useful when using Vaadin which pulls in 
> many bundles.
>
> Paul Fraser
>
> On 12/05/2023 3:00 am, Jean-Baptiste Onofré wrote:
> > Hi Ryan
> >
> > IMHO, it's always better to create the features XML by hand (and verify).
> >
> > I'm thinking about removing the generate mojo as some point.
> >
> > Regards
> > JB
> >
> > On Tue, May 9, 2023 at 3:58 PM Ryan Moquin 
> > mailto:rkmoq...@gmail.com>> wrote:
> >> I've been getting the same error as mentioned in KARAF-5999 
> >> (https://issues.apache.org/jira/browse/KARAF-5999), but I'm just trying to 
> >> generate the features xml for a subproject.  This of course happened after 
> >> I upgraded everything to Camel 3 due to the CXF feature range usage.  Is 
> >> this going to be fixed soon?  I need to decide if I need to either revert 
> >> back to the older Camel or figure out another solution to this error:
> >>
> >> Failed to execute goal 
> >> org.apache.karaf.tooling:karaf-maven-plugin:4.2.16:features-generate-descriptor
> >>  (create-features) on project dp-processor: Unable to create features.xml 
> >> file: org.apache.maven.plugin.MojoExecutionException: Cannot locate file 
> >> for feature: org.apache.cxf.karaf:apache-cxf:xml:features:[3,4) at null -> 
> >> [Help 1]
> >>
> >> Hopefully it's something that can be rectified soon.
> >>
> >> Thanks!
> >> Ryan

Re: karaf-maven-plugin:verify and sun internal requirements

[ran...@opennms.org]

As far as I can tell (at least in Karaf 4.3, which is what we’re using), there 
are no sun.* exports in jre.properties. (just a subset of com.sun.*)

They are used a bit in the source, but config.properties in the base assembly 
and itests put them into org.osgi.framework.system.packages.extra= rather than 
relying on them being considered a core part of the JRE.

Also, the startup scripts do the jre 9+ add-exports and add-opens stuff on a 
few sun.* things for module-access purposes, but I assume those don’t translate 
into visibility of the packages inside Karaf for resolution purposes.  The 
bundles in the Karaf source that use sun.* APIs (like in diagnostic/boot/) have 
sun.misc;resolution:=optional in the exports, so it probably just happens to 
work without relying on a hard import dependency.

I’ll try to throw together a simple pom project today to make it fail.

--
Benjamin Reed (he/him)
Principal Software Engineer | Raleigh, NC
[signature_2082375981]

From: Jean-Baptiste Onofré 
Date: Saturday, April 30, 2022 at 12:29 AM
To: user@karaf.apache.org 
Subject: Re: karaf-maven-plugin:verify and sun internal requirements
Hi,

verify is based on the list of features it's loading.framework/static feature 
is the base, standard or minimal comes above framework.

So, I would start to check the features list in verify MOJO .

If you can reproduce it on a test case, I can take a look. Else, I will try to 
reproduce (afair, we are using sun.* package in Karaf or Camel features/bundles 
and verify passes).

Regards
JB

On Fri, Apr 29, 2022 at 8:30 PM ran...@opennms.org<mailto:ran...@opennms.org> 
mailto:ran...@opennms.org>> wrote:
It’s quite an extensive list of features, further down our pipeline we’re 
creating a custom Karaf distribution for internal use with the assembly goal.

That said, I have figured out how to make this work after digging into the 
plugin source a bit. It’s a little hacky but does the right thing.

First, I have a custom.properties in the resources directory of my project, 
which contains:

jre-11 = ${jre-10}, \
sun.nio, \
sun.nio.ch<http://sun.nio.ch>, \
sun.nio.cs, \
sun.nio.fs, \
sun.reflect

Second, I use the maven-dependency-plugin:unpack goal to extract 
config.properties and jre.properties from the upstream 
org.apache.karaf:apache-karaf:*:tar.gz into the same target directory that 
resources get put.

Finally, I added this to the verify configuration:

  
file:${project.build.outputDirectory}/config.properties

This way I get the “distribution” config.properties but my own overlaid 
custom.properties to put the extra modules into the list of JRE exports.

I dug around in recent openjdk8 and 11 runtimes and it appears there are a 
decent number of things that are not in the default Karaf jre.properties – I 
don’t know if this is an oversight, or on purpose.

I have no idea why it works at runtime without having to do similar 
jre.properties shenanigans, I haven’t had a chance to dig into Karaf internals 
enough to understand.

Is it worth me submitting a patch with the additional runtime exports?

Thanks,
Ben
--
Benjamin Reed (he/him)
Principal Software Engineer | Raleigh, NC
[signature_414762316]

From: Jean-Baptiste Onofré mailto:j...@nanthrax.net>>
Date: Thursday, April 28, 2022 at 1:23 PM
To: user@karaf.apache.org<mailto:user@karaf.apache.org> 
mailto:user@karaf.apache.org>>
Subject: Re: karaf-maven-plugin:verify and sun internal requirements
Hi Ben,

What features repos/features are you using in the karaf-maven-plugin 
configuration ?

Regards
JB

On Thu, Apr 28, 2022 at 4:56 PM ran...@opennms.org<mailto:ran...@opennms.org> 
mailto:ran...@opennms.org>> wrote:
I’m attempting to use karaf-maven-plugin:verify to verify our features files.

They load just fine at runtime, but the verify plugin fails on:

missing requirement [de.ruedigermoeller.fst/2.57.0] osgi.wiring.package; 
filter:="(osgi.wiring.package=sun.nio.ch<http://sun.nio.ch>)"

The sun.nio package doesn’t even appear to be defined in jre.properties, so I’m 
not sure if it’s just something that’s normally inferred from the runtime 
environment without any configuration, but I can’t seem to figure out how to 
make verify accept it without forking fst and making my own fixes to their 
manifest.

Any ideas how to deal with this without doing something hacky like shading? I 
tried wrap: on the fst bundle to override the Import-Package, but I guess wrap 
must not do anything if there’s already valid bundle metadata.

Thanks,
Ben


--
Benjamin Reed (he/him)
Principal Software Engineer | Raleigh, NC
[signature_598361874]

Re: karaf-maven-plugin:verify and sun internal requirements

[ran...@opennms.org]

It’s quite an extensive list of features, further down our pipeline we’re 
creating a custom Karaf distribution for internal use with the assembly goal.

That said, I have figured out how to make this work after digging into the 
plugin source a bit. It’s a little hacky but does the right thing.

First, I have a custom.properties in the resources directory of my project, 
which contains:

jre-11 = ${jre-10}, \
sun.nio, \
sun.nio.ch, \
sun.nio.cs, \
sun.nio.fs, \
sun.reflect

Second, I use the maven-dependency-plugin:unpack goal to extract 
config.properties and jre.properties from the upstream 
org.apache.karaf:apache-karaf:*:tar.gz into the same target directory that 
resources get put.

Finally, I added this to the verify configuration:

  
file:${project.build.outputDirectory}/config.properties

This way I get the “distribution” config.properties but my own overlaid 
custom.properties to put the extra modules into the list of JRE exports.

I dug around in recent openjdk8 and 11 runtimes and it appears there are a 
decent number of things that are not in the default Karaf jre.properties – I 
don’t know if this is an oversight, or on purpose.

I have no idea why it works at runtime without having to do similar 
jre.properties shenanigans, I haven’t had a chance to dig into Karaf internals 
enough to understand.

Is it worth me submitting a patch with the additional runtime exports?

Thanks,
Ben

--
Benjamin Reed (he/him)
Principal Software Engineer | Raleigh, NC
[signature_414762316]

From: Jean-Baptiste Onofré 
Date: Thursday, April 28, 2022 at 1:23 PM
To: user@karaf.apache.org 
Subject: Re: karaf-maven-plugin:verify and sun internal requirements
Hi Ben,

What features repos/features are you using in the karaf-maven-plugin 
configuration ?

Regards
JB

On Thu, Apr 28, 2022 at 4:56 PM ran...@opennms.org<mailto:ran...@opennms.org> 
mailto:ran...@opennms.org>> wrote:
I’m attempting to use karaf-maven-plugin:verify to verify our features files.

They load just fine at runtime, but the verify plugin fails on:

missing requirement [de.ruedigermoeller.fst/2.57.0] osgi.wiring.package; 
filter:="(osgi.wiring.package=sun.nio.ch<http://sun.nio.ch>)"

The sun.nio package doesn’t even appear to be defined in jre.properties, so I’m 
not sure if it’s just something that’s normally inferred from the runtime 
environment without any configuration, but I can’t seem to figure out how to 
make verify accept it without forking fst and making my own fixes to their 
manifest.

Any ideas how to deal with this without doing something hacky like shading? I 
tried wrap: on the fst bundle to override the Import-Package, but I guess wrap 
must not do anything if there’s already valid bundle metadata.

Thanks,
Ben


--
Benjamin Reed (he/him)
Principal Software Engineer | Raleigh, NC
[signature_598361874]

karaf-maven-plugin:verify and sun internal requirements

[ran...@opennms.org]

I’m attempting to use karaf-maven-plugin:verify to verify our features files.

They load just fine at runtime, but the verify plugin fails on:

missing requirement [de.ruedigermoeller.fst/2.57.0] osgi.wiring.package; 
filter:="(osgi.wiring.package=sun.nio.ch)"

The sun.nio package doesn’t even appear to be defined in jre.properties, so I’m 
not sure if it’s just something that’s normally inferred from the runtime 
environment without any configuration, but I can’t seem to figure out how to 
make verify accept it without forking fst and making my own fixes to their 
manifest.

Any ideas how to deal with this without doing something hacky like shading? I 
tried wrap: on the fst bundle to override the Import-Package, but I guess wrap 
must not do anything if there’s already valid bundle metadata.

Thanks,
Ben


--
Benjamin Reed (he/him)
Principal Software Engineer | Raleigh, NC
[signature_598361874]