Thanks a lot Claus! Given that we now make no distinction between bundles and standard JARs through the packaging type, I was having the OSGi MANIFEST generation triggered for bundles where its irrelevant, e.g. camel-grape, camel-spring-boot, many examples, etc. So I added a profile which is activated by default and is disabled through a new POM property camel.osgi.skip, which I added to the POMs of such modules.
I was also able to upgrade the version of the maven-bundle-plugin to the latest 3.0.1. Hopefully I'll be able to finish my tests tomorrow and merge the branch into master for a full test in JKS. >From then on... let the lambda fun begin (for features we don't intend to backport)! Cheers, *Raúl Kripalani* PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and Messaging Engineer http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani Blog: raul.io | twitter: @raulvk <https://twitter.com/raulvk> On Sun, Mar 27, 2016 at 5:24 PM, Claus Ibsen <claus.ib...@gmail.com> wrote: > Hi > > I got the OSGi tests to work with karaf 4 and they are all passing now > for the features tests. There is a few that has been @Ignored because > of known issues. eg we need a new SMX bundle for them. > > You can run these tests reliable using a bash script > > cd tests/camel-itest-karaf > ./run-tests.sh > > You can also try running using mvn with > > mvn clean install > > But the latter tend to break on my computer after 20 or so tests. The > scripts runs reliable and it also cleanup dangling karaf JVMs. But > generally its been safer to run one mvn goal per test it seems. > > I also fixed the mistakes in karaf features and the tests so they are > working again. > > Raul you should be able to use that on your branch to give the osgi tests > a go. > > > > > > > On Sat, Mar 26, 2016 at 5:47 PM, Claus Ibsen <claus.ib...@gmail.com> > wrote: > > I managed to get the CI jobs to complete now with setting higher memory. > > > > The snapshots are now deployed and the CI jobs for osgi is triggered > > > > Triggering a new build of Camel.trunk.itest.karaf > > Triggering a new build of Camel.trunk.itest.osgi > > Finished: SUCCESS > > > > > > > > On Sat, Mar 26, 2016 at 12:36 PM, Claus Ibsen <claus.ib...@gmail.com> > wrote: > >> Oh I forgot the CI jobs are here > >> https://builds.apache.org/view/A-D/view/Camel/ > >> > >> I got the link from Mueller. Beforehand it was hard/impossible to find > >> Camel from > >> https://builds.apache.org/ > >> > >> So I was for a longer time not able to see all these jobs, and just > >> got the CI emails to look at. > >> > >> But the OSGi jobs rely on the SNAPSHOT of Camel has been built prior > >> to that, and that job is unreliable. I think we need to look at making > >> the -notest also publish the SNAPSHOT to the snapshot mvn repo, so the > >> CI jobs can pickup these new JARs. > >> > >> If we get that more reliable then Camel end users can also easier use > >> SNAPSHOT instead of having to build locally to be sure they get a > >> clean build of it, otherwise you can end up with mixed builds or > >> stale/old snapshots. > >> > >> > >> > >> On Sat, Mar 26, 2016 at 12:33 PM, Claus Ibsen <claus.ib...@gmail.com> > wrote: > >>> On Sat, Mar 26, 2016 at 12:12 PM, Raul Kripalani <ra...@apache.org> > wrote: > >>>> Agree with your points. > >>>> > >>>> 1. If you use lambdas on 2.18, you can't backport the code. > >>>> > >>>> 2. I'll spend some time this weekend installing the bundles on Karaf. > Karaf > >>>> 4 is OK, or are we sticking with older versions for 2.x? > >>>> > >>> > >>> Karaf 2.x is dropped. Use Karaf 4.x as the primary karaf version. > >>> We may want to add a -Pkaraf3 profile to run against older Karaf 3.x > containers. > >>> > >>> > >>>> 3. I'll also run our OSGi itests, but it may be worth having Jenkins > do all > >>>> this. Could you configure a job in JKS to build this branch? I don't > have > >>>> job admin rights I believe. > >>>> > >>> > >>> We already have some CI jobs to run 2 osgi based tests > >>> > >>> - tests/camel-itest-karaf > >>> - tests/camel-itest-osgi > >>> > >>> The former tests that the Camel component can be installed, it does > >>> this using pax-exam to bootup karaf and install the camel-xxx feature > >>> and find the component and then tear down the test. > >>> > >>> You can run these tests with the -Pkaraf4 profile. > >>> But mind I had trouble with these tests may stop after 20 or so tests > >>> and leave some karaf JVMs dangling. There is a kill-karaf.sh script to > >>> kill em. > >>> > >>> The camel-itest-osgi is likely in some state where some tests would > >>> fail even for older branches. But these are unit tests that run Camel > >>> apps in karaf and do some testing stuff like we do in regular > >>> component tests. This has not yet been migrated to karaf 4, and run on > >>> karaf 2.x. But you can give it a go and see how bad/good it is. It > >>> used to be able to run almost all tests and pass. And we had also had > >>> periods where everything was green. > >>> > >>> However people dont write so many osgi tests so this module do not > >>> have so much love. And there are plenty of components that are not > >>> being tested. > >>> > >>> > >>>> 4. It's necessary to configure the maven-jar-plugin to explicitly use > a > >>>> manifest file, otherwise the plugin generates its own. Originally I > tried > >>>> adding the config to the build plugins in the parent POM, but it > failed for > >>>> artifacts of type=maven-plugin. Hence I placed it in pluginManagement > and > >>>> then incorporated it in those intermediate POMs (components, examples, > >>>> etc.) whose children built normal JARs. However, I ended up tracing > the > >>>> issue to a conflict introduced by the config of the maven-jar-plugin > that > >>>> maven-plugin injects, and solved it with a couple > >>>> of combine.self="override". Hence, I think I might be able to push our > >>>> maven-jar-plugin config back to the parent POM and to the build > section. > >>>> > >>> > >>> Ah okay. I am okay with we can do "anything" we need to build all the > >>> Camel components and whatnot. > >>> > >>> Its more what the impact is for end users. If we can make out examples > >>> and maven archetypes simpler and easier that would be cool. But mind > >>> that not all examples are osgi examples. > >>> > >>> But it looks promising your work - thanks for starting this hard work. > >>> > >>> > >>> > >>>> Cheers, > >>>> > >>>> *Raúl Kripalani* > >>>> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data > and > >>>> Messaging Engineer > >>>> http://about.me/raulkripalani | > http://www.linkedin.com/in/raulkripalani > >>>> Blog: raul.io | twitter: @raulvk <https://twitter.com/raulvk> > >>>> > >>>> On Sat, Mar 26, 2016 at 10:12 AM, Claus Ibsen <claus.ib...@gmail.com> > wrote: > >>>> > >>>>> Hi > >>>>> > >>>>> Good to hear that we may drop the <bundle> extension, nice work. > >>>>> > >>>>> For camel-core I have thought of build the manifest.mf by hand so we > >>>>> can better control it. It seems the bundle plugin has its issues with > >>>>> different versions to build it, as you say. > >>>>> > >>>>> Have you tried testing any of these JARs in OSGi? It would be good if > >>>>> you try that and install some of the examples in a karaf 4 container > >>>>> and see how it runs. > >>>>> > >>>>> For lambda code in the existing source code then we cannot do that > for > >>>>> code that need to be backported to 2.17 and 2.16 branches. We can > only > >>>>> use lambdas in new code that is not to be backported. > >>>>> > >>>>> For Camel 3.0 we can then covert the code to use lambads all over > >>>>> where applicable. There is likely some tools in IDEA or Eclipse etc > >>>>> that can assist with that. > >>>>> > >>>>> > >>>>> And Camel end users can use lamdas today with their own code, also > >>>>> with older Camel versions on java 8. > >>>>> > >>>>> It may be that those people are using a new bundle plugin, or more > >>>>> likely not using OSGi at all. > >>>>> > >>>>> > >>>>> I would like to see more OSGi testing on this branch before any > merge. > >>>>> OSGi is complex and pain to maintain and build modules. So we need to > >>>>> be sure we are on a path that we can do this. There is 250+ maven > >>>>> modules that are build as OSGi so it better have to work for us. > >>>>> > >>>>> But I really like that the bundle plugin is just generating a > >>>>> MANIFEST.MF file that the JAR plugin just slurps in. Can it not be > >>>>> configured somehow so the generated MANIFEST.MF is not replace so end > >>>>> users do not need to do anything with a JAR plugin? > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Sat, Mar 26, 2016 at 1:09 AM, Raul Kripalani <ra...@apache.org> > wrote: > >>>>> > There is good news, bad news, and good news again ;-) > >>>>> > > >>>>> > *Good news:* Now that Camel 2.18 is officially on JDK8, we can use > >>>>> lambdas > >>>>> > in our code. Woohoo! > >>>>> > > >>>>> > *Bad news:* Our current Maven build breaks when doing so. > >>>>> > > >>>>> > This is what happens: > >>>>> > > >>>>> > 1. maven-bundle-plugin version 2.3.x cannot parse JDK8 lambdas. > [1] > >>>>> > > >>>>> > 2. Upgrading it to the latest version (3.0.1) makes the plugin > fail > >>>>> > miserably (at least on camel-core). > >>>>> > > >>>>> > 3. Using the latest 2.x version of the maven-bundle-plugin > (2.5.4) > >>>>> makes > >>>>> > the maven-shade-plugin fail with this error in turn: > >>>>> > > >>>>> > [ERROR] The project main artifact does not exist. This could have > the > >>>>> > following > >>>>> > [ERROR] reasons: > >>>>> > [ERROR] - You have invoked the goal directly from the command > line. This > >>>>> is > >>>>> > not > >>>>> > [ERROR] supported. Please add the goal to the default lifecycle > via an > >>>>> > [ERROR] <execution> element in your POM and use "mvn package" to > have > >>>>> it > >>>>> > run. > >>>>> > [ERROR] - You have bound the goal to a lifecycle phase before > "package". > >>>>> > Please > >>>>> > [ERROR] remove this binding from your POM such that the goal > will be > >>>>> run > >>>>> > in > >>>>> > [ERROR] the proper phase. > >>>>> > > >>>>> > 4. Upgrading the shade plugin doesn't help. > >>>>> > > >>>>> > 5. The situation is dirty :P > >>>>> > > >>>>> > The problem is the usage of the 'bundle' custom packaging type > enabled by > >>>>> > extensions='true' in the maven-bundle-plugin. > >>>>> > > >>>>> > *Good news again:* > >>>>> > > >>>>> > We can avoid the problem by reverting back to the standard 'jar' > >>>>> packaging > >>>>> > type, which even puts us in a less brittle situation wrt. to > future Maven > >>>>> > plugins we may want to add/upgrade. This implies: > >>>>> > > >>>>> > 1. Calling the maven-bundle-plugin:manifest goal (instead of the > >>>>> bundle > >>>>> > goal) to only generate the MANIFEST.MF. > >>>>> > > >>>>> > 2. Adding it to the JAR via an option in the the > maven-jar-plugin: > >>>>> > > >>>>> > <plugin> > >>>>> > <groupId>org.apache.maven.plugins</groupId> > >>>>> > <artifactId>maven-jar-plugin</artifactId> > >>>>> > <version>${maven-jar-plugin-version}</version> > >>>>> > <configuration> > >>>>> > <archive> > >>>>> > > >>>>> > > >>>>> > <manifestFile>${project.build.outputDirectory}/META-INF/MANIFEST.MF</manifestFile> > >>>>> > </archive> > >>>>> > </configuration> > >>>>> > </plugin> > >>>>> > > >>>>> > I've made the change. But since it's a big one, I didn't push to > master > >>>>> but > >>>>> > to a branch for us to review first: jdk8-lambdas. > >>>>> > > >>>>> > The build passes (yay!) but OSGi battle testing is a must to > ensure there > >>>>> > are no regressions. > >>>>> > > >>>>> > Could you have a quick look? If no feedback is received until > Monday EOD, > >>>>> > I'll merge to master. If feedback is positive, we can merge > earlier to > >>>>> > enable people to develop with lambdas finally. > >>>>> > > >>>>> > --- > >>>>> > > >>>>> > Just in case you're not aware, the OSGi legend Neil Bartlett has > just > >>>>> > released a new bnd-maven-plugin [2] that claims to overcome such > >>>>> > dysfunctions, as well as others, of the Felix plugin. He > complained about > >>>>> > how the custom packaging type breaks integration with other plugins > >>>>> > downstream (what we're experiencing). > >>>>> > > >>>>> > I see no reason to migrate to his bnd-maven-plugin, but we should > keep it > >>>>> > in mind for the future. > >>>>> > > >>>>> > [1] https://issues.apache.org/jira/browse/FELIX-4005 > >>>>> > [2] > http://njbartlett.name/2015/03/27/announcing-bnd-maven-plugin.html > >>>>> > > >>>>> > Cheers, > >>>>> > > >>>>> > *Raúl Kripalani* > >>>>> > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big > Data and > >>>>> > Messaging Engineer > >>>>> > http://about.me/raulkripalani | > http://www.linkedin.com/in/raulkripalani > >>>>> > Blog: raul.io | twitter: @raulvk <https://twitter.com/raulvk> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Claus Ibsen > >>>>> ----------------- > >>>>> http://davsclaus.com @davsclaus > >>>>> Camel in Action 2: https://www.manning.com/ibsen2 > >>>>> > >>> > >>> > >>> > >>> -- > >>> Claus Ibsen > >>> ----------------- > >>> http://davsclaus.com @davsclaus > >>> Camel in Action 2: https://www.manning.com/ibsen2 > >> > >> > >> > >> -- > >> Claus Ibsen > >> ----------------- > >> http://davsclaus.com @davsclaus > >> Camel in Action 2: https://www.manning.com/ibsen2 > > > > > > > > -- > > Claus Ibsen > > ----------------- > > http://davsclaus.com @davsclaus > > Camel in Action 2: https://www.manning.com/ibsen2 > > > > -- > Claus Ibsen > ----------------- > http://davsclaus.com @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 >
