And what integration test failures do you see when they are removed? eg maybe make sure that empty folders are removed etc, as the integration test is some voodoo magic Nicola did that creates a spring boot project per starter to see if it can install and startup spring boot, and for some it also runs additional tests. It's complex to maintain.
On Tue, Oct 19, 2021 at 11:28 AM Claus Ibsen <claus.ib...@gmail.com> wrote: > > Hi > > Yeah these starters are not in use and should be removed > > camel-debezium-common-starter > http-common-starter > jetty-common-starter > > > On Mon, Oct 18, 2021 at 8:34 PM David Jencks <david.a.jen...@gmail.com> wrote: > > > > I can easily remove the empty starter json files, but if e.g. > > camel-debezium-common-starter isn’t a starter, what is it? Removing the 3 > > starter projects from camel-spring-boot results in integration test > > failures. Does some other process install these as dependencies as > > appropriate? Are they actually unnecessary but something needs more > > tweaking? > > > > David Jencks > > > > > On Oct 18, 2021, at 12:17 AM, Claus Ibsen <claus.ib...@gmail.com> wrote: > > > > > > Hi > > > > > > The -common JARs are not starters, and they should not be listed. > > > > > > On Mon, Oct 18, 2021 at 7:45 AM David Jencks <david.a.jen...@gmail.com > > > <mailto:david.a.jen...@gmail.com>> wrote: > > >> > > >> Locally I got the ability to show unused starter json files working, and > > >> fixed the obvious problems: > > >> > > >> avro-rpc was using avro > > >> paho-mqtt5 was using paho > > >> in 3.7.x, grape and joor were set up wrong. > > >> > > >> There are still some “starters” i’m not sure about: > > >> > > >> debezium-common > > >> http-common > > >> jetty-common > > >> > > >> In a previous commit I added empty json files for these on the theory > > >> that if they were starters they should have instructions on how to use > > >> them, but they don’t have corresponding main camel doc pages. Are these > > >> starters automatically installed by the corresponding non-common starter > > >> artifacts such as camel-debezium-mongodb-starter? If so, I can remove > > >> the empty json files. If not, how should their usage be documented? > > >> > > >> I’m hoping to get my updated Antora extensions published in the next > > >> couple of days, at which point we can upgrade Camel to use them. > > >> > > >> David Jencks > > >> > > >>> On Oct 13, 2021, at 8:14 PM, David Jencks <david.a.jen...@gmail.com> > > >>> wrote: > > >>> > > >>> They are part of the camel-core-starter, although it’s hard to find the > > >>> 2 options per language among the 147 options for the starter. This > > >>> suggests that, for starters that encompass more than one component, it > > >>> would be useful to only show the options relevant to that component on > > >>> the components’ page. > > >>> > > >>> In any case I believe everything is working properly and I’ve merged > > >>> all the 9 PRs. > > >>> > > >>> The core and non-core languages are currently in different tables on > > >>> the spring-boot list page: I should be able to put them back into one > > >>> table after upgrading antora-indexer. > > >>> > > >>> David Jencks > > >>> > > >>>> On Oct 12, 2021, at 11:03 PM, David Jencks <david.a.jen...@gmail.com > > >>>> <mailto:david.a.jen...@gmail.com> <mailto:david.a.jen...@gmail.com > > >>>> <mailto:david.a.jen...@gmail.com>>> wrote: > > >>>> > > >>>> There’s at least one problem…. (also asked on zulip) > > >>>> > > >>>> Are camel-core-languages actually part of camel-spring-boot? > > >>>> > > >>>> They are listed in the table in "list of starters" but with an > > >>>> artifactid of "camel-base" which isn't a starter. The current site > > >>>> doesn't have autoconfiguration info for them either. Since they are in > > >>>> the table, I added info to them for my PRs, but it's at least partly > > >>>> wrong, since it comes out as needing camel-core-languages-starter, > > >>>> which doesn't exist. cf. > > >>>> https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages > > >>>> > > >>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages> > > >>>> > > >>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages > > >>>> > > >>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages>>, > > >>>> > > >>>> https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration > > >>>> > > >>>> <https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration> > > >>>> > > >>>> <https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration > > >>>> > > >>>> <https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration>>. > > >>>> Since they say they are part of camel-core, I've used core.json to > > >>>> construct the autoconfig options table, but that doesn't look very > > >>>> relevant to me. > > >>>> > > >>>> Is there a starter needed for them? If so, which one? > > >>>> camel-core-starter seems like a possibility, is it correct? > > >>>> > > >>>> If no starter is needed, why are they in the spring-boot starters > > >>>> table? > > >>>> > > >>>> I hope to be able to merge everything tomorrow, it got too late > > >>>> tonight. > > >>>> > > >>>> David Jencks > > >>>> > > >>>>> On Oct 12, 2021, at 7:41 PM, David Jencks <david.a.jen...@gmail.com > > >>>>> <mailto:david.a.jen...@gmail.com> <mailto:david.a.jen...@gmail.com > > >>>>> <mailto:david.a.jen...@gmail.com>>> wrote: > > >>>>> > > >>>>> The part of this work I’m ready for now is done, except for some > > >>>>> commit-squashing and dealing with check style errors :-) > > >>>>> > > >>>>> Preview at https://pr-644--camel.netlify.app > > >>>>> <https://pr-644--camel.netlify.app/> > > >>>>> <https://pr-644--camel.netlify.app/ > > >>>>> <https://pr-644--camel.netlify.app/>> > > >>>>> > > >>>>> I seem to only be able to write AsciiDoc by now…. :-) Here’s a list > > >>>>> of the differences between table rows new/old: > > >>>>> > > >>>>> —— > > >>>>> = Differences between spring-boot list-old and list > > >>>>> > > >>>>> == latest (main) > > >>>>> > > >>>>> === In list, not list-old: > > >>>>> > > >>>>> * disruptor-vm-component > > >>>>> * splunk-hec-component > > >>>>> > > >>>>> * (3 bindy dataformats collapsed to one link, corresponding to the > > >>>>> single main bindy doc page) > > >>>>> > > >>>>> * csimple-language > > >>>>> > > >>>>> == 3.12.x > > >>>>> === In list, not list-old: > > >>>>> > > >>>>> * disruptor-vm-component > > >>>>> * splunk-hec-component > > >>>>> > > >>>>> * (3 bindy dataformats collapsed to one link, corresponding to the > > >>>>> single main bindy doc page) > > >>>>> > > >>>>> * csimple-language > > >>>>> > > >>>>> == 3.11.x > > >>>>> === In list, not list-old: > > >>>>> > > >>>>> * splunk-hec-component > > >>>>> * (3 bindy dataformats collapsed to one link, corresponding to the > > >>>>> single main bindy doc page) > > >>>>> * csimple-language > > >>>>> > > >>>>> Moved to separate table: > > >>>>> > > >>>>> * camel-spring-cloud > > >>>>> * camel-spring-cloud-consul > > >>>>> * camel-spring-cloud-netflix > > >>>>> * camel-spring-cloud-zookeeper > > >>>>> > > >>>>> == 3.7.x > > >>>>> === In list, not list-old: > > >>>>> * splunk-hec-component > > >>>>> * (3 bindy dataformats collapsed to one link, corresponding to the > > >>>>> single main bindy doc page) > > >>>>> * csimple-language > > >>>>> > > >>>>> > > >>>>> Moved to separate table: > > >>>>> > > >>>>> * camel-spring-cloud > > >>>>> * camel-spring-cloud-consul > > >>>>> * camel-spring-cloud-netflix > > >>>>> * camel-spring-cloud-zookeeper > > >>>>> > > >>>>> —— > > >>>>> > > >>>>> There are also some presentation differences in the tables, the new > > >>>>> ones follow the tables in main camel components more closely, e.g. > > >>>>> > > >>>>> <td class="tableblock halign-left valign-top"><p > > >>>>> class="tableblock">Stable</p></td> > > >>>>>>> > > >>>>> <td class="tableblock halign-left valign-top"><p > > >>>>> class="tableblock">Stable-deprecated</p></td> > > >>>>> > > >>>>> as the deprecation marker. > > >>>>> > > >>>>> Also quite a few core languages now list the starter as > > >>>>> > > >>>>> <td class="tableblock halign-left valign-top"><p > > >>>>> class="tableblock">camel-core-languages-starter</p></td> > > >>>>> rather than > > >>>>> <td class="tableblock halign-left valign-top"><p > > >>>>> class="tableblock">camel-base</p></td> > > >>>>> > > >>>>> Is this correct? > > >>>>> > > >>>>> There are 8 PRs: > > >>>>> (Camel) > > >>>>> Main spring boot jsonpath (allow me to squash and merge) > > >>>>> <https://github.com/apache/camel/pull/6261 > > >>>>> <https://github.com/apache/camel/pull/6261>> > > >>>>> Camel 3.12.x spring boot jsonpath (please let me squash and merge) > > >>>>> <https://github.com/apache/camel/pull/6263 > > >>>>> <https://github.com/apache/camel/pull/6263>> > > >>>>> Camel 3.11.x spring boot jsonpath (please let me squash and merge) > > >>>>> <https://github.com/apache/camel/pull/6264 > > >>>>> <https://github.com/apache/camel/pull/6264>> > > >>>>> Camel 3.7.x spring boot jsonpath (please let me squash and merge) > > >>>>> <https://github.com/apache/camel/pull/6262 > > >>>>> <https://github.com/apache/camel/pull/6262>> > > >>>>> (Camel-spring-boot) > > >>>>> Main jsonpath (please let me squash and merge) > > >>>>> <https://github.com/apache/camel-spring-boot/pull/388 > > >>>>> <https://github.com/apache/camel-spring-boot/pull/388>> > > >>>>> Camel spring boot 3.12.x jsonpath (please let me squash and merge) > > >>>>> <https://github.com/apache/camel-spring-boot/pull/387 > > >>>>> <https://github.com/apache/camel-spring-boot/pull/387>> > > >>>>> Camel spring boot 3.11.x jsonpath (please let me squash and merge) > > >>>>> <https://github.com/apache/camel-spring-boot/pull/386 > > >>>>> <https://github.com/apache/camel-spring-boot/pull/386>> > > >>>>> Camel spring boot 3.7.x jsonpath (please let me squash and merge) > > >>>>> <https://github.com/apache/camel-spring-boot/pull/385 > > >>>>> <https://github.com/apache/camel-spring-boot/pull/385>> > > >>>>> > > >>>>> If everything looks good I plan to squash each PR to 2 or 3 commits > > >>>>> (code changes and generated changes) and rebase/merge. I don’t think > > >>>>> any playbook changes are needed. > > >>>>> > > >>>>> I’m completing a big upgrade of the antora-indexer and jsonpath > > >>>>> extensions, and plan to finish that next. It will involve changing > > >>>>> the syntax of most uses of these extensions. After that I expect to > > >>>>> be able to implement automated checking for starters that aren’t > > >>>>> linked to by main camel components. I think there are about 5, but > > >>>>> have no way to find them at this point. > > >>>>> > > >>>>> > > >>>>> David Jencks > > >>>>> > > >>>>>> On Oct 12, 2021, at 12:01 AM, Claus Ibsen <claus.ib...@gmail.com > > >>>>>> <mailto:claus.ib...@gmail.com> <mailto:claus.ib...@gmail.com > > >>>>>> <mailto:claus.ib...@gmail.com>>> wrote: > > >>>>>> > > >>>>>> On Mon, Oct 11, 2021 at 8:07 AM David Jencks > > >>>>>> <david.a.jen...@gmail.com <mailto:david.a.jen...@gmail.com> > > >>>>>> <mailto:david.a.jen...@gmail.com <mailto:david.a.jen...@gmail.com>>> > > >>>>>> wrote: > > >>>>>>> > > >>>>>>> I’ve been working on generating the spring-boot autoconfig info > > >>>>>>> from the json files generated during the spring-boot build, and the > > >>>>>>> spring boot “list.adoc” tables using the indexer extension (like > > >>>>>>> the components tables are generated). > > >>>>>>> > > >>>>>>> My understanding of the spring-boot stuff is that: > > >>>>>>> > > >>>>>>> - any Camel component can be used in Spring > > >>>>>>> > > >>>>>>> - Some camel components/dataformats… can participate in spring-boot > > >>>>>>> auto configure. These components are identified as having a > > >>>>>>> corresponding camel-*-starter project in the spring-boot repo. > > >>>>>>> > > >>>>>>> - There are 3 kinds of these: > > >>>>>>> > > >>>>>>> 1. There’s (generated) java code and a generated json file > > >>>>>>> > > >>>>>>> 2. There’s generated java code but no generated json file (e.g. > > >>>>>>> jasypt) > > >>>>>>> > > >>>>>>> 3. There’s no java code and no json file, just some properties > > >>>>>>> files etc. (e.g. aws-xray) > > >>>>>>> > > >>>>>>> I’m really not familiar with spring-boot or the philosophy behind > > >>>>>>> it, but my guess is that if there’s a spring-boot starter jar then > > >>>>>>> spring will create a singleton instance of something in the jar, > > >>>>>>> and if there’s a json file that will guide or describe configuring > > >>>>>>> this instance with data from somewhere. If this is roughly > > >>>>>>> correct, then the 2nd and 3rd kinds correspond to non-configurable > > >>>>>>> instances. > > >>>>>>> > > >>>>>>> I’ve discovered that: > > >>>>>>> > > >>>>>>> - several components have starters that are not listed in the > > >>>>>>> existing list > > >>>>>>> > > >>>>>> > > >>>>>> Oh which ones for example is that? > > >>>>>> > > >>>>>> For Camel on Spring Boot then we only support the -starter JARs. So > > >>>>>> if > > >>>>>> there is not a starter then its not supported. > > >>>>>> And yes some of the starters are just shallow and empty/almost empty. > > >>>>>> > > >>>>>> But it's our way to curate what we support and make sense to use on > > >>>>>> Spring Boot. > > >>>>>> > > >>>>>> For Camel on Karaf its the features.xml file that defines what we > > >>>>>> support there. > > >>>>>> And ditto for Quarkus with the extensions. > > >>>>>> > > >>>>>> > > >>>>>>> - quite a few components main adoc documentation do not include the > > >>>>>>> relevant autoconfig information (even if it’s “no options”) > > >>>>>>> Most of these are linked to from the “list” page, so I’d imagine it > > >>>>>>> would be pretty confusing to follow the link and find no > > >>>>>>> information about spring-boot. > > >>>>>>> > > >>>>>>> Evidently, the current process is not maintainable. > > >>>>>>> > > >>>>>>> The (new, jsonpath based) autoconfig doc generation is based on > > >>>>>>> putting the name of the appropriate json file in the main adoc file > > >>>>>>> as a header attribute. > > >>>>>>> > > >>>>>>> I think we can produce something more resilient by: > > >>>>>>> > > >>>>>>> - for kinds (2) and (3), put a no-options appropriately named json > > >>>>>>> file in the starter project under src/main/docs. This will not get > > >>>>>>> added to the starter jar, unlike the properly generated json files. > > >>>>>>> > > >>>>>>> - put the corresponding header attribute in the appropriate main > > >>>>>>> adoc file > > >>>>>>> > > >>>>>>> Now we should be able to do a basic check, that the number of > > >>>>>>> unique json file names from header attributes is equal to the > > >>>>>>> number of json files, which should be equal to the number of > > >>>>>>> starters. This should be possible to do as part of the website > > >>>>>>> build. > > >>>>>>> > > >>>>>>> This won’t check that for instance with a starter that serves many > > >>>>>>> components, all the components will refer to the starter > > >>>>>>> information, but it will detect most or all of the problems I’v > > >>>>>>> found so far. > > >>>>>>> > > >>>>>>> The preview for https://github.com/apache/camel-website/pull/644 > > >>>>>>> <https://github.com/apache/camel-website/pull/644> > > >>>>>>> <https://github.com/apache/camel-website/pull/644 > > >>>>>>> <https://github.com/apache/camel-website/pull/644>> shows how far > > >>>>>>> I’ve gotten with this: the appropriate page to look at is > > >>>>>>> https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html > > >>>>>>> > > >>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html> > > >>>>>>> > > >>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html > > >>>>>>> > > >>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html>> > > >>>>>>> This has, in addition to the tables of spring-boot > > >>>>>>> autoconfigure-enabled components, counts of json files used and > > >>>>>>> existing (there are 5 unused), and tables of > > >>>>>>> non-spring-boot-enabled components/dataformats… > > >>>>>>> All the dataformats and languages are set up properly. There are 2 > > >>>>>>> components that aren’t spring-boot-enabled and a whole lot of > > >>>>>>> “others" > > >>>>>>> > > >>>>>>> I think that actually failing the build when there are unused .json > > >>>>>>> files and also finding out which json files are unused will have to > > >>>>>>> wait for an upgrade to the antora-indexer… unfortunately this will > > >>>>>>> require some syntax changes. > > >>>>>>> > > >>>>>>> I’d like to polish these changes, replace the list page with the > > >>>>>>> new one (with the extra info removed), port this work to the other > > >>>>>>> relevant branches, and then pursue releasing the upgraded > > >>>>>>> antora-indexer extension and upgrading the syntax, and then work on > > >>>>>>> making the checks work. > > >>>>>>> > > >>>>>>> Does this seem reasonable? > > >>>>>>> > > >>>>>> > > >>>>>> Yeah sure continue your great work and effort on improving the > > >>>>>> documentation for all the Camel projects on the website. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>> —— > > >>>>>>> > > >>>>>>> Another feature of the preview is that I found out how to make the > > >>>>>>> option names in all the jsonpath generated tables link to > > >>>>>>> themselves and display a marker on hover, just like section titles, > > >>>>>>> so you can now easily copy the link to an option and paste it > > >>>>>>> somewhere. (the names have been links for a while, but there was > > >>>>>>> no practical way to find out what the link was!) > > >>>>>>> > > >>>>>>> E.g. … > > >>>>>>> https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo > > >>>>>>> > > >>>>>>> <https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo><https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo > > >>>>>>> > > >>>>>>> <https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo>> > > >>>>>>> > > >>>>>>> David Jencks > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> -- > > >>>>>> Claus Ibsen > > >>>>>> ----------------- > > >>>>>> http://davsclaus.com <http://davsclaus.com/> <http://davsclaus.com/ > > >>>>>> <http://davsclaus.com/>> @davsclaus > > >>>>>> Camel in Action 2: https://www.manning.com/ibsen2 > > >>>>>> <https://www.manning.com/ibsen2> <https://www.manning.com/ibsen2 > > >>>>>> <https://www.manning.com/ibsen2>> > > >>>> > > >>> > > >> > > > > > > > > > -- > > > Claus Ibsen > > > ----------------- > > > http://davsclaus.com <http://davsclaus.com/> @davsclaus > > > Camel in Action 2: https://www.manning.com/ibsen2 > > > <https://www.manning.com/ibsen2> > > > > -- > Claus Ibsen > ----------------- > http://davsclaus.com @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
