Re: Re: New component proposal: commons-plugins

[Peter Verhas]

I believe it would be a great proof of the whole concept if the DI system
could be plugged into the new commons plugin system using the SPI commons
plugin provides for plugins.

On Thu, Apr 14, 2022 at 8:00 PM Matt Sicker  wrote:

> Yes, at this point, it would help to see what aspects of this would be
> useful or welcomed as a library. In the meantime, I've been asked if I
> could port the log4j DI system back to 2.x, so I may end up working on
> that in the near term and defer any work to extract code to Commons
> until we have a better idea of what a Commons component would look
> like.
>
> I can say that a requirement from the Log4j side of things is having a
> dependency-free library was one important aspect. Another seemingly
> obvious yet rarely done requirement was that the DI library shouldn't
> themselves depend on Log4j or SLF4J; in log4j-plugins, I can work
> around this fairly simply by using StatusLogger, though the DI library
> I started with didn't even log anything (errors were surfaced through
> exceptions only). When a library wants to have plugins, you end up in
> a dependency loop where Spring wants to use libA, but libA wants to
> use Spring for its plugin system; how does this work? At an
> application level, though, this is less of a concern.
>
> On Thu, Apr 14, 2022 at 7:12 AM Eric Bresie  wrote:
> >
> > It sounds like embedded in this discussion are some use cases /
> requirements. Would quantifying these some with a high level list of
> features that this plugin architecture is to provide?
> >
> > What would the basic functionality and/or interfaces be…something like:
> > - Register Plugin/Services
> > - Plugin/Service Configuration
> > - Lookup Plugin/Service
> > - Invoke Plugin/Service
> > - Annotation for Plugin/Services
> > - Service/Adapter Interface for use with External module systems (i.e.
> doing generically in common-plugin to make calls to external module system)?
> >
> > What kind of test cases would be applicable to buy into acceptance?
> >
> > Would common-plugin be similar to maven where there is the basic core
> functionality with addition plugin’s providing new “goal”/functionality to
> provide additional compatibility (i.e. import plugin.osgi, import
> plugin.spring, etc.). and/or provide a wrapper/adapter between module
> systems? So assume the above would provide the core and the subsequent
> “adapter” layer or plugin for each could be implemented separately (either
> internal or external to the project).
> >
> > Eric Bresie
> > ebre...@gmail.com (mailto:ebre...@gmail.com)
> > > On April 12, 2022 at 5:10:25 PM CDT, Thomas  t...@online.de)> wrote:
> > >
> > > > > > Users should be required to do as little as possible to adapt the
> > > > > > plugin system to the environment it is running in*but plugins
> shouldn’t know or care anything about how they are located and loaded.*
> Plugins are also to the
> > > > > > application or framework that will be using them. They
> essentially define what
> > > > > > the valid plugin types are and where they can be used.
> > > > > This is impossible. The bare minimum, any system or artifact has
> to do, to be recognized as a plugin, is advertising itself as one. That
> would be true, even if your service locator is crawling through every class
> of the classloader it can get hold of to determine, whether it is a
> candidate to consider.
> > > > >
> > > > > - OSGI does this by parsing the MANIFEST and OSGI-INF
> > > > >
> > > > > - Spring uses a combination of XML and crawling annotations and
> packages
> > > > >
> > > > > Every other CDI framework does it similar to these two, most of
> them heavily reliying on XML- or property based configuration too. With one
> notable exception: the dreaded ServiceLoader mechanism, which fixes this
> during compile time in the module-info, if you use JPMS, else mapping it in
> META-INF/services with simple plain text files, that require no parsing
> other then getting the line breaks right - no reflection / premature
> loading required. (Which btw, I think is the slickest solution so far, as
> it does make service discovery very cheap for small systems, with few class
> path entries. Complication is inevitably the result of large classpaths,
> that require plugin arbitration to resolve ambiguities and filtering)
> > > > >
> > > > > So no: at the very least, a plugin has to know, by what plugin
> system it is going to be loaded. And - a different hyperbole: because of
> this, apache commons has started to introduce the attributes required by
> OSGi into the MANIFESTs of many of its components, just to bridge the gap,
> and to aleviate the need to for OSGi-Users to introduce wrappers.
> > > > >
> > > > > For me, this will be all well and fine, as long as none of the
> current or future apache commons artefacts will become unusable unless I
> also put commons-plugin or some arcane configuration parser into the class
> path.
> > > > I never said a plugin shouldn’t know it is a plugin. Providi

Re: Re: New component proposal: commons-plugins

[Matt Sicker]

Yes, at this point, it would help to see what aspects of this would be
useful or welcomed as a library. In the meantime, I've been asked if I
could port the log4j DI system back to 2.x, so I may end up working on
that in the near term and defer any work to extract code to Commons
until we have a better idea of what a Commons component would look
like.

I can say that a requirement from the Log4j side of things is having a
dependency-free library was one important aspect. Another seemingly
obvious yet rarely done requirement was that the DI library shouldn't
themselves depend on Log4j or SLF4J; in log4j-plugins, I can work
around this fairly simply by using StatusLogger, though the DI library
I started with didn't even log anything (errors were surfaced through
exceptions only). When a library wants to have plugins, you end up in
a dependency loop where Spring wants to use libA, but libA wants to
use Spring for its plugin system; how does this work? At an
application level, though, this is less of a concern.

On Thu, Apr 14, 2022 at 7:12 AM Eric Bresie  wrote:
>
> It sounds like embedded in this discussion are some use cases / requirements. 
> Would quantifying these some with a high level list of features that this 
> plugin architecture is to provide?
>
> What would the basic functionality and/or interfaces be…something like:
> - Register Plugin/Services
> - Plugin/Service Configuration
> - Lookup Plugin/Service
> - Invoke Plugin/Service
> - Annotation for Plugin/Services
> - Service/Adapter Interface for use with External module systems (i.e. doing 
> generically in common-plugin to make calls to external module system)?
>
> What kind of test cases would be applicable to buy into acceptance?
>
> Would common-plugin be similar to maven where there is the basic core 
> functionality with addition plugin’s providing new “goal”/functionality to 
> provide additional compatibility (i.e. import plugin.osgi, import 
> plugin.spring, etc.). and/or provide a wrapper/adapter between module 
> systems? So assume the above would provide the core and the subsequent 
> “adapter” layer or plugin for each could be implemented separately (either 
> internal or external to the project).
>
> Eric Bresie
> ebre...@gmail.com (mailto:ebre...@gmail.com)
> > On April 12, 2022 at 5:10:25 PM CDT, Thomas  > (mailto:t...@online.de)> wrote:
> >
> > > > > Users should be required to do as little as possible to adapt the
> > > > > plugin system to the environment it is running in*but plugins 
> > > > > shouldn’t know or care anything about how they are located and 
> > > > > loaded.* Plugins are also to the
> > > > > application or framework that will be using them. They essentially 
> > > > > define what
> > > > > the valid plugin types are and where they can be used.
> > > > This is impossible. The bare minimum, any system or artifact has to do, 
> > > > to be recognized as a plugin, is advertising itself as one. That would 
> > > > be true, even if your service locator is crawling through every class 
> > > > of the classloader it can get hold of to determine, whether it is a 
> > > > candidate to consider.
> > > >
> > > > - OSGI does this by parsing the MANIFEST and OSGI-INF
> > > >
> > > > - Spring uses a combination of XML and crawling annotations and packages
> > > >
> > > > Every other CDI framework does it similar to these two, most of them 
> > > > heavily reliying on XML- or property based configuration too. With one 
> > > > notable exception: the dreaded ServiceLoader mechanism, which fixes 
> > > > this during compile time in the module-info, if you use JPMS, else 
> > > > mapping it in META-INF/services with simple plain text files, that 
> > > > require no parsing other then getting the line breaks right - no 
> > > > reflection / premature loading required. (Which btw, I think is the 
> > > > slickest solution so far, as it does make service discovery very cheap 
> > > > for small systems, with few class path entries. Complication is 
> > > > inevitably the result of large classpaths, that require plugin 
> > > > arbitration to resolve ambiguities and filtering)
> > > >
> > > > So no: at the very least, a plugin has to know, by what plugin system 
> > > > it is going to be loaded. And - a different hyperbole: because of this, 
> > > > apache commons has started to introduce the attributes required by OSGi 
> > > > into the MANIFESTs of many of its components, just to bridge the gap, 
> > > > and to aleviate the need to for OSGi-Users to introduce wrappers.
> > > >
> > > > For me, this will be all well and fine, as long as none of the current 
> > > > or future apache commons artefacts will become unusable unless I also 
> > > > put commons-plugin or some arcane configuration parser into the class 
> > > > path.
> > > I never said a plugin shouldn’t know it is a plugin. Providing the 
> > > manifest entries required to make a component usable in an OSGi 
> > > environment doesn’t mean the Plugin has to know it is being

Re: Re: New component proposal: commons-plugins

[Eric Bresie]

It sounds like embedded in this discussion are some use cases / requirements. 
Would quantifying these some with a high level list of features that this 
plugin architecture is to provide?

What would the basic functionality and/or interfaces be…something like:
- Register Plugin/Services
- Plugin/Service Configuration
- Lookup Plugin/Service
- Invoke Plugin/Service
- Annotation for Plugin/Services
- Service/Adapter Interface for use with External module systems (i.e. doing 
generically in common-plugin to make calls to external module system)?

What kind of test cases would be applicable to buy into acceptance?

Would common-plugin be similar to maven where there is the basic core 
functionality with addition plugin’s providing new “goal”/functionality to 
provide additional compatibility (i.e. import plugin.osgi, import 
plugin.spring, etc.). and/or provide a wrapper/adapter between module systems? 
So assume the above would provide the core and the subsequent “adapter” layer 
or plugin for each could be implemented separately (either internal or external 
to the project).

Eric Bresie
ebre...@gmail.com (mailto:ebre...@gmail.com)
> On April 12, 2022 at 5:10:25 PM CDT, Thomas  (mailto:t...@online.de)> wrote:
>
> > > > Users should be required to do as little as possible to adapt the
> > > > plugin system to the environment it is running in*but plugins shouldn’t 
> > > > know or care anything about how they are located and loaded.* Plugins 
> > > > are also to the
> > > > application or framework that will be using them. They essentially 
> > > > define what
> > > > the valid plugin types are and where they can be used.
> > > This is impossible. The bare minimum, any system or artifact has to do, 
> > > to be recognized as a plugin, is advertising itself as one. That would be 
> > > true, even if your service locator is crawling through every class of the 
> > > classloader it can get hold of to determine, whether it is a candidate to 
> > > consider.
> > >
> > > - OSGI does this by parsing the MANIFEST and OSGI-INF
> > >
> > > - Spring uses a combination of XML and crawling annotations and packages
> > >
> > > Every other CDI framework does it similar to these two, most of them 
> > > heavily reliying on XML- or property based configuration too. With one 
> > > notable exception: the dreaded ServiceLoader mechanism, which fixes this 
> > > during compile time in the module-info, if you use JPMS, else mapping it 
> > > in META-INF/services with simple plain text files, that require no 
> > > parsing other then getting the line breaks right - no reflection / 
> > > premature loading required. (Which btw, I think is the slickest solution 
> > > so far, as it does make service discovery very cheap for small systems, 
> > > with few class path entries. Complication is inevitably the result of 
> > > large classpaths, that require plugin arbitration to resolve ambiguities 
> > > and filtering)
> > >
> > > So no: at the very least, a plugin has to know, by what plugin system it 
> > > is going to be loaded. And - a different hyperbole: because of this, 
> > > apache commons has started to introduce the attributes required by OSGi 
> > > into the MANIFESTs of many of its components, just to bridge the gap, and 
> > > to aleviate the need to for OSGi-Users to introduce wrappers.
> > >
> > > For me, this will be all well and fine, as long as none of the current or 
> > > future apache commons artefacts will become unusable unless I also put 
> > > commons-plugin or some arcane configuration parser into the class path.
> > I never said a plugin shouldn’t know it is a plugin. Providing the manifest 
> > entries required to make a component usable in an OSGi environment doesn’t 
> > mean the Plugin has to know it is being used in a an OSGi environment. 
> > Likewise, providing a module-info.java doesn’t mean the plugin will be used 
> > as part of a module on a module path. Using Log4j as an example once again, 
> > its Plugins know that they are a Filter, Appender, Lookup, Layout, or 
> > whatever. But they have no knowledge of how they were loaded. They DO know 
> > that they are being used by Log4j for a specific purpose and what contract 
> > they have to implement for that, but that is exactly what plugins are for.
> >
> > So I guess Log4j is doing the impossible?
> >
> ... only if your Log4J plugins would be automagically recognised as
> regular plugins by other frameworks, ranging from OSGi, to Spring, CDI,
> Jakarta, Plexus, Avalon, Netbeans, JMeter, ANT, Gradle, ... without
> adding the required plugin descriptors these systems need to find it. It
> probably wouldn't. In IT and programming, to know generally means, to
> directly refer to it. In this case: to refer to the plugin
> infrastructure supported. So your plugin is plugin, not just because it
> has the string 'plugin', somewhere in the documentation; it's not just
> any plugin, but a Log4J plugin, and knows it. No magic there.
>
> BTW.: I can see, that

Re: New component proposal: commons-plugins

[Thomas]

Users should be required to do as little as possible to adapt the
plugin system to the environment it is running in*but plugins shouldn’t know or 
care anything about how they are located and loaded.* Plugins are also to the
application or framework that will be using them. They essentially define what
the valid plugin types are and where they can be used.

This is impossible. The bare minimum, any system or artifact has to do, to be 
recognized as a plugin, is advertising itself as one. That would be true, even 
if your service locator is crawling through every class of the classloader it 
can get hold of to determine, whether it is a candidate to consider.

- OSGI does this by parsing the MANIFEST and OSGI-INF

- Spring uses a combination of XML and crawling annotations and packages

Every other CDI framework does it similar to these two, most of them heavily 
reliying on XML- or property based configuration too. With one notable 
exception: the dreaded ServiceLoader mechanism, which fixes this during compile 
time in the module-info, if you use JPMS, else mapping it in META-INF/services 
with simple plain text files, that require no parsing other then getting the 
line breaks right - no reflection / premature loading required. (Which btw, I 
think is the slickest solution so far, as it does make service discovery very 
cheap for small systems, with few class path entries. Complication is 
inevitably the result of large classpaths, that require plugin arbitration to 
resolve ambiguities and filtering)

So no: at the very least, a plugin has to know, by what plugin system it is 
going to be loaded. And - a different hyperbole: because of this, apache 
commons has started to introduce the attributes required by OSGi into the 
MANIFESTs of many of its components, just to bridge the gap, and to aleviate 
the need to for OSGi-Users to introduce wrappers.

For me, this will be all well and fine, as long as none of the current or 
future apache commons artefacts will  become unusable unless I also put 
commons-plugin or some arcane configuration parser into the class path.

I never said a plugin shouldn’t know it is a plugin. Providing the manifest 
entries required to make a component usable in an OSGi environment doesn’t mean 
the Plugin has to know it is being used in a an OSGi environment. Likewise, 
providing a module-info.java doesn’t mean the plugin will be used as part of a 
module on a module path. Using Log4j as an example once again, its Plugins know 
that they are a Filter, Appender, Lookup, Layout, or whatever. But they have no 
knowledge of how they were loaded. They DO know that they are being used by 
Log4j for a specific purpose and what contract they have to implement for that, 
but that is exactly what plugins are for.

So I guess Log4j is doing the impossible?

... only if your Log4J plugins would be automagically recognised as 
regular plugins by other frameworks, ranging from OSGi, to Spring, CDI, 
Jakarta, Plexus, Avalon, Netbeans, JMeter, ANT, Gradle, ... without 
adding the required plugin descriptors these systems need to find it. It 
probably wouldn't. In IT and programming, to know generally means, to 
directly refer to it. In this case: to refer to the plugin 
infrastructure supported. So your plugin is plugin, not just because it 
has the string 'plugin', somewhere in the documentation; it's not just 
any plugin, but a Log4J plugin, and knows it. No magic there.


BTW.: I can see, that there may be some need to add some convenience to 
the bare ServiceLoader mechanism, in order to standardise/support some 
version/feature/provider negotiation in certain corner cases. I would 
even see usecases for that in commons components like vfs, encoding, 
compress etc, that are meant to offer a facade to plugable 
implementations of a certain family.


But I don't think, there is a need to support an other full fledged 
module or plugin framework with features like dependency injection, 
component lifecycle management, interceptor chains, not to speak of 
interpolating configuration values from sources outside of Java. Even 
less do I feel the need to have an abstraction layer that enables us to 
swap out say, CDI for Spring or OSGi. Don't tempt me to cite XKCD 927 on 
you!


Cheers, Thomas


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: New component proposal: commons-plugins

[Romain Manni-Bucau]

Le mar. 12 avr. 2022 à 21:31, Ralph Goers  a
écrit :

>
>
> > On Apr 12, 2022, at 6:56 PM, Thomas  wrote:
> >
> >
> > On 11.04.2022 00:00, Ralph Goers wrote:
> >> See below
> >>
> >>> On Apr 8, 2022, at 9:23 AM, Peter Verhas wrote:
> >>>
> >>> Thanks Ralph for the detailed explanation. I appreciate it and now I
> see
> >>> the points.
> >> I’ve removed the parts that I don’t think need any more discussion.
> >>
> >>> - How will it be a “plugin" project and not another dependency
> injection
> >>> framework?
> >> This is a great question. I think the main difference is with examples
> like Log4j
> >> and Apache Flume, and even Apache Maven. All wire components together
> via
> >> user provided configuration, not code. Dependency injection could
> certainly be
> >> part of the plugin framework but that would be for implementors of
> plugins,
> >> not the users using them. Users of Maven don’t know that Plexus is used
> under
> >> the covers and neither should users of a commons-plugins implementation.
> >>
> >>> - What will distinguish it from module systems, like OSGi and what will
> >>> stop it from becoming another OSGi by the years as new features get
> added
> >>> to the library.
> >> OSGi is complicated. Implementing plugins should not be. Just as I
> described
> >> using ServiceLoader to locate plugins, plugins should also be
> accessible in
> >> OSGi bundles. Users should be required to do as little as possible to
> adapt the
> >> plugin system to the environment it is running in*but plugins shouldn’t
> know or care anything about how they are located and loaded.* Plugins are
> also to the
> >> application or framework that will be using them. They essentially
> define what
> >> the valid plugin types are and where they can be used.
> >
> > This is impossible. The bare minimum, any system or artifact has to do,
> to be recognized as a plugin, is advertising itself as one. That would be
> true, even if your service locator is crawling through every class of the
> classloader it can get hold of to determine, whether it is a candidate to
> consider.
> >
> > - OSGI does this by parsing the MANIFEST and OSGI-INF
> >
> > - Spring uses a combination of XML and crawling annotations and packages
> >
> > Every other CDI framework does it similar to these two, most of them
> heavily reliying on XML- or property based configuration too. With one
> notable exception: the dreaded ServiceLoader mechanism, which fixes this
> during compile time in the module-info, if you use JPMS, else mapping it in
> META-INF/services with simple plain text files, that require no parsing
> other then getting the line breaks right - no reflection / premature
> loading required. (Which btw, I think is the slickest solution so far, as
> it does make service discovery very cheap for small systems, with few class
> path entries. Complication is inevitably the result of large classpaths,
> that require plugin arbitration to resolve ambiguities and filtering)
> >
> > So no: at the very least, a plugin has to know, by what plugin system it
> is going to be loaded. And - a different hyperbole: because of this, apache
> commons has started to introduce the attributes required by OSGi into the
> MANIFESTs of many of its components, just to bridge the gap, and to
> aleviate the need to for OSGi-Users to introduce wrappers.
> >
> > For me, this will be all well and fine, as long as none of the current
> or future apache commons artefacts will  become unusable unless I also put
> commons-plugin or some arcane configuration parser into the class path.
>
> I never said a plugin shouldn’t know it is a plugin. Providing the
> manifest entries required to make a component usable in an OSGi environment
> doesn’t mean the Plugin has to know it is being used in a an OSGi
> environment. Likewise, providing a module-info.java doesn’t mean the plugin
> will be used as part of a module on a module path. Using Log4j as an
> example once again, its Plugins know that they are a Filter, Appender,
> Lookup, Layout, or whatever. But they have no knowledge of how they were
> loaded. They DO know that they are being used by Log4j for a specific
> purpose and what contract they have to implement for that, but that is
> exactly what plugins are for.
>
> So I guess Log4j is doing the impossible?


It would if it would be all IoC natively integrated as all plugin systems
but log4j created its own in parallel - for good and bad reasons, not
discussing it there. So overall log4j just does like other IoC as of today
;).



>
> Ralph
>
>
>

Re: New component proposal: commons-plugins

[Ralph Goers]

> On Apr 12, 2022, at 6:56 PM, Thomas  wrote:
> 
> 
> On 11.04.2022 00:00, Ralph Goers wrote:
>> See below
>> 
>>> On Apr 8, 2022, at 9:23 AM, Peter Verhas wrote:
>>> 
>>> Thanks Ralph for the detailed explanation. I appreciate it and now I see
>>> the points.
>> I’ve removed the parts that I don’t think need any more discussion.
>> 
>>> - How will it be a “plugin" project and not another dependency injection
>>> framework?
>> This is a great question. I think the main difference is with examples like 
>> Log4j
>> and Apache Flume, and even Apache Maven. All wire components together via
>> user provided configuration, not code. Dependency injection could certainly 
>> be
>> part of the plugin framework but that would be for implementors of plugins,
>> not the users using them. Users of Maven don’t know that Plexus is used under
>> the covers and neither should users of a commons-plugins implementation.
>> 
>>> - What will distinguish it from module systems, like OSGi and what will
>>> stop it from becoming another OSGi by the years as new features get added
>>> to the library.
>> OSGi is complicated. Implementing plugins should not be. Just as I described
>> using ServiceLoader to locate plugins, plugins should also be accessible in
>> OSGi bundles. Users should be required to do as little as possible to adapt 
>> the
>> plugin system to the environment it is running in*but plugins shouldn’t know 
>> or care anything about how they are located and loaded.* Plugins are also to 
>> the
>> application or framework that will be using them. They essentially define 
>> what
>> the valid plugin types are and where they can be used.
> 
> This is impossible. The bare minimum, any system or artifact has to do, to be 
> recognized as a plugin, is advertising itself as one. That would be true, 
> even if your service locator is crawling through every class of the 
> classloader it can get hold of to determine, whether it is a candidate to 
> consider.
> 
> - OSGI does this by parsing the MANIFEST and OSGI-INF
> 
> - Spring uses a combination of XML and crawling annotations and packages
> 
> Every other CDI framework does it similar to these two, most of them heavily 
> reliying on XML- or property based configuration too. With one notable 
> exception: the dreaded ServiceLoader mechanism, which fixes this during 
> compile time in the module-info, if you use JPMS, else mapping it in 
> META-INF/services with simple plain text files, that require no parsing other 
> then getting the line breaks right - no reflection / premature loading 
> required. (Which btw, I think is the slickest solution so far, as it does 
> make service discovery very cheap for small systems, with few class path 
> entries. Complication is inevitably the result of large classpaths, that 
> require plugin arbitration to resolve ambiguities and filtering)
> 
> So no: at the very least, a plugin has to know, by what plugin system it is 
> going to be loaded. And - a different hyperbole: because of this, apache 
> commons has started to introduce the attributes required by OSGi into the 
> MANIFESTs of many of its components, just to bridge the gap, and to aleviate 
> the need to for OSGi-Users to introduce wrappers.
> 
> For me, this will be all well and fine, as long as none of the current or 
> future apache commons artefacts will  become unusable unless I also put 
> commons-plugin or some arcane configuration parser into the class path.

I never said a plugin shouldn’t know it is a plugin. Providing the manifest 
entries required to make a component usable in an OSGi environment doesn’t mean 
the Plugin has to know it is being used in a an OSGi environment. Likewise, 
providing a module-info.java doesn’t mean the plugin will be used as part of a 
module on a module path. Using Log4j as an example once again, its Plugins know 
that they are a Filter, Appender, Lookup, Layout, or whatever. But they have no 
knowledge of how they were loaded. They DO know that they are being used by 
Log4j for a specific purpose and what contract they have to implement for that, 
but that is exactly what plugins are for.

So I guess Log4j is doing the impossible?  

Ralph

Re: New component proposal: commons-plugins

[Thomas]

On 11.04.2022 00:00, Ralph Goers wrote:

See below


On Apr 8, 2022, at 9:23 AM, Peter Verhas  wrote:

Thanks Ralph for the detailed explanation. I appreciate it and now I see
the points.

I’ve removed the parts that I don’t think need any more discussion.


- How will it be a “plugin" project and not another dependency injection
framework?

This is a great question. I think the main difference is with examples like 
Log4j
and Apache Flume, and even Apache Maven. All wire components together via
user provided configuration, not code. Dependency injection could certainly be
part of the plugin framework but that would be for implementors of plugins,
not the users using them. Users of Maven don’t know that Plexus is used under
the covers and neither should users of a commons-plugins implementation.


- What will distinguish it from module systems, like OSGi and what will
stop it from becoming another OSGi by the years as new features get added
to the library.

OSGi is complicated. Implementing plugins should not be. Just as I described
using ServiceLoader to locate plugins, plugins should also be accessible in
OSGi bundles. Users should be required to do as little as possible to adapt the
plugin system to the environment it is running in*but plugins shouldn’t know or care anything about how they are 
located and loaded.*  Plugins are also to the

application or framework that will be using them. They essentially define what
the valid plugin types are and where they can be used.


This is impossible. The bare minimum, any system or artifact has to do, 
to be recognized as a plugin, is advertising itself as one. That would 
be true, even if your service locator is crawling through every class of 
the classloader it can get hold of to determine, whether it is a 
candidate to consider.


- OSGI does this by parsing the MANIFEST and OSGI-INF

- Spring uses a combination of XML and crawling annotations and packages

Every other CDI framework does it similar to these two, most of them 
heavily reliying on XML- or property based configuration too. With one 
notable exception: the dreaded ServiceLoader mechanism, which fixes this 
during compile time in the module-info, if you use JPMS, else mapping it 
in META-INF/services with simple plain text files, that require no 
parsing other then getting the line breaks right - no reflection / 
premature loading required. (Which btw, I think is the slickest solution 
so far, as it does make service discovery very cheap for small systems, 
with few class path entries. Complication is inevitably the result of 
large classpaths, that require plugin arbitration to resolve ambiguities 
and filtering)


So no: at the very least, a plugin has to know, by what plugin system it 
is going to be loaded. And - a different hyperbole: because of this, 
apache commons has started to introduce the attributes required by OSGi 
into the MANIFESTs of many of its components, just to bridge the gap, 
and to aleviate the need to for OSGi-Users to introduce wrappers.


For me, this will be all well and fine, as long as none of the current 
or future apache commons artefacts will  become unusable unless I also 
put commons-plugin or some arcane configuration parser into the class path.

Re: New component proposal: commons-plugins

[Gary Gregory]

Start it here, if it fails, we'll all learn something I am sure.

Gary

On Tue, Apr 12, 2022, 08:53 Gilles Sadowski  wrote:

> Le mar. 12 avr. 2022 à 13:23, Gary Gregory  a
> écrit :
> >
> > Commons Component can and do depend on other runtime libraries, for
> > example, VFS, Configuration, JCS, and so on. There are libraries that are
> > naturally lower level where we do want to keep zero depencies like IO and
> > Lang. If an app has a plugin system it seems evident to me that it would
> be
> > the kind of app that depends on other libraries anyway.
>
> Question remains: Start something here or not (advantages vs drawbacks)?
>
> >
> > Gary
> >
> > On Tue, Apr 12, 2022, 05:57 Gilles Sadowski 
> wrote:
> >
> > > Hello.
> > >
> > > Le mar. 12 avr. 2022 à 08:58, Romain Manni-Bucau
> > >  a écrit :
> > > >
> > > > Sounds like https://svn.apache.org/repos/asf/commons/sandbox/ can
> be a
> > > > ready to start place even if I still think incubator is the real
> place
> > > for
> > > > such a project since it will quickly overpass commons standard case
> with
> > > a
> > > > lot of modules if it gets a community and adopted (for integrations).
> > > >
> > >
> > > "Commons" components are supposed to not depend on anything
> > > (except other "Commons" components and optional dependencies).
> > > If some of them need the functionality being discussed in this thread
> > > (as has been mentioned by Matt S, Matt J and Gary), but it is defined
> > > in another TLP, reuse will be "forbidden".
> > >
> > > A modular (maven) project could contain
> > > * modules that abide by the "no-dependency" policy (providing "core"
> > >   functionality that can be reused here), and
> > > * modules with external dependencies whenever required.
> > > [If the latter modules are considered out-of-scope for Commons, then
> > > the glue code would be left for the respective projects to implement.]
> > >
> > > Regards,
> > > Gilles
> > >
> > > >> [...]
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: New component proposal: commons-plugins

[Gary Gregory]

Hi Matt,
That's normal thinking IMO not devil's advocacy :-) The JDK provides a bare
bones extension mechanism in the service loader but that's not enough for a
real plugin system.

Gary

On Tue, Apr 12, 2022, 11:30 Matt Benson  wrote:

> Devil's advocate thought: given that Commons, while not officially limited
> to such, has heretofore concentrated on the Java ecosystem; as the core JDK
> evolves, matures and provides more and more functionality out of the box,
> possibly Commons components would have to increase in complexity to be able
> to provide anything worthwhile.
>
> Matt
>
> On Tue, Apr 12, 2022, 7:53 AM Gilles Sadowski 
> wrote:
>
> > Le mar. 12 avr. 2022 à 13:23, Gary Gregory  a
> > écrit :
> > >
> > > Commons Component can and do depend on other runtime libraries, for
> > > example, VFS, Configuration, JCS, and so on. There are libraries that
> are
> > > naturally lower level where we do want to keep zero depencies like IO
> and
> > > Lang. If an app has a plugin system it seems evident to me that it
> would
> > be
> > > the kind of app that depends on other libraries anyway.
> >
> > Question remains: Start something here or not (advantages vs drawbacks)?
> >
> > >
> > > Gary
> > >
> > > On Tue, Apr 12, 2022, 05:57 Gilles Sadowski 
> > wrote:
> > >
> > > > Hello.
> > > >
> > > > Le mar. 12 avr. 2022 à 08:58, Romain Manni-Bucau
> > > >  a écrit :
> > > > >
> > > > > Sounds like https://svn.apache.org/repos/asf/commons/sandbox/ can
> > be a
> > > > > ready to start place even if I still think incubator is the real
> > place
> > > > for
> > > > > such a project since it will quickly overpass commons standard case
> > with
> > > > a
> > > > > lot of modules if it gets a community and adopted (for
> integrations).
> > > > >
> > > >
> > > > "Commons" components are supposed to not depend on anything
> > > > (except other "Commons" components and optional dependencies).
> > > > If some of them need the functionality being discussed in this thread
> > > > (as has been mentioned by Matt S, Matt J and Gary), but it is defined
> > > > in another TLP, reuse will be "forbidden".
> > > >
> > > > A modular (maven) project could contain
> > > > * modules that abide by the "no-dependency" policy (providing "core"
> > > >   functionality that can be reused here), and
> > > > * modules with external dependencies whenever required.
> > > > [If the latter modules are considered out-of-scope for Commons, then
> > > > the glue code would be left for the respective projects to
> implement.]
> > > >
> > > > Regards,
> > > > Gilles
> > > >
> > > > >> [...]
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> > For additional commands, e-mail: dev-h...@commons.apache.org
> >
> >
>

Re: New component proposal: commons-plugins

[Matt Benson]

Devil's advocate thought: given that Commons, while not officially limited
to such, has heretofore concentrated on the Java ecosystem; as the core JDK
evolves, matures and provides more and more functionality out of the box,
possibly Commons components would have to increase in complexity to be able
to provide anything worthwhile.

Matt

On Tue, Apr 12, 2022, 7:53 AM Gilles Sadowski  wrote:

> Le mar. 12 avr. 2022 à 13:23, Gary Gregory  a
> écrit :
> >
> > Commons Component can and do depend on other runtime libraries, for
> > example, VFS, Configuration, JCS, and so on. There are libraries that are
> > naturally lower level where we do want to keep zero depencies like IO and
> > Lang. If an app has a plugin system it seems evident to me that it would
> be
> > the kind of app that depends on other libraries anyway.
>
> Question remains: Start something here or not (advantages vs drawbacks)?
>
> >
> > Gary
> >
> > On Tue, Apr 12, 2022, 05:57 Gilles Sadowski 
> wrote:
> >
> > > Hello.
> > >
> > > Le mar. 12 avr. 2022 à 08:58, Romain Manni-Bucau
> > >  a écrit :
> > > >
> > > > Sounds like https://svn.apache.org/repos/asf/commons/sandbox/ can
> be a
> > > > ready to start place even if I still think incubator is the real
> place
> > > for
> > > > such a project since it will quickly overpass commons standard case
> with
> > > a
> > > > lot of modules if it gets a community and adopted (for integrations).
> > > >
> > >
> > > "Commons" components are supposed to not depend on anything
> > > (except other "Commons" components and optional dependencies).
> > > If some of them need the functionality being discussed in this thread
> > > (as has been mentioned by Matt S, Matt J and Gary), but it is defined
> > > in another TLP, reuse will be "forbidden".
> > >
> > > A modular (maven) project could contain
> > > * modules that abide by the "no-dependency" policy (providing "core"
> > >   functionality that can be reused here), and
> > > * modules with external dependencies whenever required.
> > > [If the latter modules are considered out-of-scope for Commons, then
> > > the glue code would be left for the respective projects to implement.]
> > >
> > > Regards,
> > > Gilles
> > >
> > > >> [...]
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: New component proposal: commons-plugins

[Gilles Sadowski]

Le mar. 12 avr. 2022 à 13:23, Gary Gregory  a écrit :
>
> Commons Component can and do depend on other runtime libraries, for
> example, VFS, Configuration, JCS, and so on. There are libraries that are
> naturally lower level where we do want to keep zero depencies like IO and
> Lang. If an app has a plugin system it seems evident to me that it would be
> the kind of app that depends on other libraries anyway.

Question remains: Start something here or not (advantages vs drawbacks)?

>
> Gary
>
> On Tue, Apr 12, 2022, 05:57 Gilles Sadowski  wrote:
>
> > Hello.
> >
> > Le mar. 12 avr. 2022 à 08:58, Romain Manni-Bucau
> >  a écrit :
> > >
> > > Sounds like https://svn.apache.org/repos/asf/commons/sandbox/ can be a
> > > ready to start place even if I still think incubator is the real place
> > for
> > > such a project since it will quickly overpass commons standard case with
> > a
> > > lot of modules if it gets a community and adopted (for integrations).
> > >
> >
> > "Commons" components are supposed to not depend on anything
> > (except other "Commons" components and optional dependencies).
> > If some of them need the functionality being discussed in this thread
> > (as has been mentioned by Matt S, Matt J and Gary), but it is defined
> > in another TLP, reuse will be "forbidden".
> >
> > A modular (maven) project could contain
> > * modules that abide by the "no-dependency" policy (providing "core"
> >   functionality that can be reused here), and
> > * modules with external dependencies whenever required.
> > [If the latter modules are considered out-of-scope for Commons, then
> > the glue code would be left for the respective projects to implement.]
> >
> > Regards,
> > Gilles
> >
> > >> [...]

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: New component proposal: commons-plugins

[Gary Gregory]

Commons Component can and do depend on other runtime libraries, for
example, VFS, Configuration, JCS, and so on. There are libraries that are
naturally lower level where we do want to keep zero depencies like IO and
Lang. If an app has a plugin system it seems evident to me that it would be
the kind of app that depends on other libraries anyway.

Gary

On Tue, Apr 12, 2022, 05:57 Gilles Sadowski  wrote:

> Hello.
>
> Le mar. 12 avr. 2022 à 08:58, Romain Manni-Bucau
>  a écrit :
> >
> > Sounds like https://svn.apache.org/repos/asf/commons/sandbox/ can be a
> > ready to start place even if I still think incubator is the real place
> for
> > such a project since it will quickly overpass commons standard case with
> a
> > lot of modules if it gets a community and adopted (for integrations).
> >
>
> "Commons" components are supposed to not depend on anything
> (except other "Commons" components and optional dependencies).
> If some of them need the functionality being discussed in this thread
> (as has been mentioned by Matt S, Matt J and Gary), but it is defined
> in another TLP, reuse will be "forbidden".
>
> A modular (maven) project could contain
> * modules that abide by the "no-dependency" policy (providing "core"
>   functionality that can be reused here), and
> * modules with external dependencies whenever required.
> [If the latter modules are considered out-of-scope for Commons, then
> the glue code would be left for the respective projects to implement.]
>
> Regards,
> Gilles
>
> >> [...]
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: New component proposal: commons-plugins

[Gilles Sadowski]

Hello.

Le mar. 12 avr. 2022 à 08:58, Romain Manni-Bucau
 a écrit :
>
> Sounds like https://svn.apache.org/repos/asf/commons/sandbox/ can be a
> ready to start place even if I still think incubator is the real place for
> such a project since it will quickly overpass commons standard case with a
> lot of modules if it gets a community and adopted (for integrations).
>

"Commons" components are supposed to not depend on anything
(except other "Commons" components and optional dependencies).
If some of them need the functionality being discussed in this thread
(as has been mentioned by Matt S, Matt J and Gary), but it is defined
in another TLP, reuse will be "forbidden".

A modular (maven) project could contain
* modules that abide by the "no-dependency" policy (providing "core"
  functionality that can be reused here), and
* modules with external dependencies whenever required.
[If the latter modules are considered out-of-scope for Commons, then
the glue code would be left for the respective projects to implement.]

Regards,
Gilles

>> [...]

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: New component proposal: commons-plugins

[Romain Manni-Bucau]

Sounds like https://svn.apache.org/repos/asf/commons/sandbox/ can be a
ready to start place even if I still think incubator is the real place for
such a project since it will quickly overpass commons standard case with a
lot of modules if it gets a community and adopted (for integrations).

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>


Le lun. 11 avr. 2022 à 20:47, Matt Sicker  a écrit :

> At this point, I'd be most willing to start up a repo and codebase for
> this only if it would be useful for Commons, too. In this scenario, I
> can begin by porting over the relevant code from log4j to form a
> starting point for the library (mainly an API, and annotation
> processor, and default implementation for API), and then we can
> continue defining out the desired API and implement any alternative
> backends for the API. From the sounds of it, ideally, the plugins API
> would have a default dependency-free implementation (the one I'd port
> from log4j) along with options to swap that out with a more
> sophisticated DI library like Spring, Guice, Weld, whatever, which
> would really only be relevant to the application that's defining
> plugins in the first place. Similarly, the API for obtaining a tree of
> configuration nodes should be pluggable so that libraries like
> Configuration can be used as a backend or custom ones based on Jackson
> for supporting several different structured file formats. While the
> library can provide some minimal implementations for the configuration
> and DI aspects, I can appreciate why some apps would prefer to use a
> different library for that (e.g., to support aspect-oriented
> programming, interceptors, decorators, etc).
>
> On Mon, Apr 11, 2022 at 6:06 AM Gary Gregory 
> wrote:
> >
> > It certainly would be nice if a new common-plugins would be reused within
> > Commons. It seems like it would allow us to provide a nice component
> > outside the validation it would get from Log4j.
> > VFS, Configuration, JCS?
> >
> > Gary
> >
> > On Sun, Apr 10, 2022, 23:28 Bernd Eckenfels 
> wrote:
> >
> > > Hello,
> > >
> > > I don’t think plug-ins is something which fits well in the scope of
> > > commons, as it does tie into a broader ecosystem/platform normally.
> but I
> > > could be wrong.
> > >
> > > Btw: Commons VFS has a plug-in system, mostly for providers.
> > >
> > > Gruss
> > > Bernd
> > > --
> > > http://bernd.eckenfels.net
> > > 
> > > Von: Ralph Goers 
> > > Gesendet: Monday, April 11, 2022 12:00:12 AM
> > > An: Commons Developers List 
> > > Betreff: Re: New component proposal: commons-plugins
> > >
> > > See below
> > >
> > > > On Apr 8, 2022, at 9:23 AM, Peter Verhas  wrote:
> > > >
> > > > Thanks Ralph for the detailed explanation. I appreciate it and now I
> see
> > > > the points.
> > >
> > > I’ve removed the parts that I don’t think need any more discussion.
> > >
> > > > - How will it be a “plugin" project and not another dependency
> injection
> > > > framework?
> > >
> > > This is a great question. I think the main difference is with examples
> > > like Log4j
> > > and Apache Flume, and even Apache Maven. All wire components together
> via
> > > user provided configuration, not code. Dependency injection could
> > > certainly be
> > > part of the plugin framework but that would be for implementors of
> plugins,
> > > not the users using them. Users of Maven don’t know that Plexus is used
> > > under
> > > the covers and neither should users of a commons-plugins
> implementation.
> > >
> > > > - What will distinguish it from module systems, like OSGi and what
> will
> > > > stop it from becoming another OSGi by the years as new features get
> added
> > > > to the library.
> > >
> > > OSGi is complicated. Implementing plugins should not be. Just as I
> > > described
> > > using ServiceLoader to locate plugins, plugins should also be
> accessible in
> > > OSGi bundles. Users should be required to do as little as possible to
> > > adapt the
> > > plugin system to the e

Re: New component proposal: commons-plugins

[Matt Sicker]

At this point, I'd be most willing to start up a repo and codebase for
this only if it would be useful for Commons, too. In this scenario, I
can begin by porting over the relevant code from log4j to form a
starting point for the library (mainly an API, and annotation
processor, and default implementation for API), and then we can
continue defining out the desired API and implement any alternative
backends for the API. From the sounds of it, ideally, the plugins API
would have a default dependency-free implementation (the one I'd port
from log4j) along with options to swap that out with a more
sophisticated DI library like Spring, Guice, Weld, whatever, which
would really only be relevant to the application that's defining
plugins in the first place. Similarly, the API for obtaining a tree of
configuration nodes should be pluggable so that libraries like
Configuration can be used as a backend or custom ones based on Jackson
for supporting several different structured file formats. While the
library can provide some minimal implementations for the configuration
and DI aspects, I can appreciate why some apps would prefer to use a
different library for that (e.g., to support aspect-oriented
programming, interceptors, decorators, etc).

On Mon, Apr 11, 2022 at 6:06 AM Gary Gregory  wrote:
>
> It certainly would be nice if a new common-plugins would be reused within
> Commons. It seems like it would allow us to provide a nice component
> outside the validation it would get from Log4j.
> VFS, Configuration, JCS?
>
> Gary
>
> On Sun, Apr 10, 2022, 23:28 Bernd Eckenfels  wrote:
>
> > Hello,
> >
> > I don’t think plug-ins is something which fits well in the scope of
> > commons, as it does tie into a broader ecosystem/platform normally. but I
> > could be wrong.
> >
> > Btw: Commons VFS has a plug-in system, mostly for providers.
> >
> > Gruss
> > Bernd
> > --
> > http://bernd.eckenfels.net
> > ________
> > Von: Ralph Goers 
> > Gesendet: Monday, April 11, 2022 12:00:12 AM
> > An: Commons Developers List 
> > Betreff: Re: New component proposal: commons-plugins
> >
> > See below
> >
> > > On Apr 8, 2022, at 9:23 AM, Peter Verhas  wrote:
> > >
> > > Thanks Ralph for the detailed explanation. I appreciate it and now I see
> > > the points.
> >
> > I’ve removed the parts that I don’t think need any more discussion.
> >
> > > - How will it be a “plugin" project and not another dependency injection
> > > framework?
> >
> > This is a great question. I think the main difference is with examples
> > like Log4j
> > and Apache Flume, and even Apache Maven. All wire components together via
> > user provided configuration, not code. Dependency injection could
> > certainly be
> > part of the plugin framework but that would be for implementors of plugins,
> > not the users using them. Users of Maven don’t know that Plexus is used
> > under
> > the covers and neither should users of a commons-plugins implementation.
> >
> > > - What will distinguish it from module systems, like OSGi and what will
> > > stop it from becoming another OSGi by the years as new features get added
> > > to the library.
> >
> > OSGi is complicated. Implementing plugins should not be. Just as I
> > described
> > using ServiceLoader to locate plugins, plugins should also be accessible in
> > OSGi bundles. Users should be required to do as little as possible to
> > adapt the
> > plugin system to the environment it is running in but plugins shouldn’t
> > know or
> > care anything about how they are located and loaded. Plugins are also to
> > the
> > application or framework that will be using them. They essentially define
> > what
> > the valid plugin types are and where they can be used.
> >
> > > - What applications using plugins are the examples for different
> > solutions?
> > > (Log4j is a good example to show that there is a need, you also explained
> > > patiently why it is not a simple ServiceLoader, but it is only one way to
> > > solve it. Other applications may approach the issue differently. Maven,
> > > Attlassian products, other build tools, JUnit 5 and so on.)
> >
> > We’ve already mentioned Log4j. Apache Flume also sort of supports plugins,
> > which I previously mentioned. The main ones are Sinks, Channels, and
> > Sources.
> > In the Flume configuration Sinks provided by Flume can be referenced by
> > their “simple” name because they are manually registered in the code. User
> > provided components have to prov

Re: New component proposal: commons-plugins

[Gary Gregory]

It certainly would be nice if a new common-plugins would be reused within
Commons. It seems like it would allow us to provide a nice component
outside the validation it would get from Log4j.
VFS, Configuration, JCS?

Gary

On Sun, Apr 10, 2022, 23:28 Bernd Eckenfels  wrote:

> Hello,
>
> I don’t think plug-ins is something which fits well in the scope of
> commons, as it does tie into a broader ecosystem/platform normally. but I
> could be wrong.
>
> Btw: Commons VFS has a plug-in system, mostly for providers.
>
> Gruss
> Bernd
> --
> http://bernd.eckenfels.net
> 
> Von: Ralph Goers 
> Gesendet: Monday, April 11, 2022 12:00:12 AM
> An: Commons Developers List 
> Betreff: Re: New component proposal: commons-plugins
>
> See below
>
> > On Apr 8, 2022, at 9:23 AM, Peter Verhas  wrote:
> >
> > Thanks Ralph for the detailed explanation. I appreciate it and now I see
> > the points.
>
> I’ve removed the parts that I don’t think need any more discussion.
>
> > - How will it be a “plugin" project and not another dependency injection
> > framework?
>
> This is a great question. I think the main difference is with examples
> like Log4j
> and Apache Flume, and even Apache Maven. All wire components together via
> user provided configuration, not code. Dependency injection could
> certainly be
> part of the plugin framework but that would be for implementors of plugins,
> not the users using them. Users of Maven don’t know that Plexus is used
> under
> the covers and neither should users of a commons-plugins implementation.
>
> > - What will distinguish it from module systems, like OSGi and what will
> > stop it from becoming another OSGi by the years as new features get added
> > to the library.
>
> OSGi is complicated. Implementing plugins should not be. Just as I
> described
> using ServiceLoader to locate plugins, plugins should also be accessible in
> OSGi bundles. Users should be required to do as little as possible to
> adapt the
> plugin system to the environment it is running in but plugins shouldn’t
> know or
> care anything about how they are located and loaded. Plugins are also to
> the
> application or framework that will be using them. They essentially define
> what
> the valid plugin types are and where they can be used.
>
> > - What applications using plugins are the examples for different
> solutions?
> > (Log4j is a good example to show that there is a need, you also explained
> > patiently why it is not a simple ServiceLoader, but it is only one way to
> > solve it. Other applications may approach the issue differently. Maven,
> > Attlassian products, other build tools, JUnit 5 and so on.)
>
> We’ve already mentioned Log4j. Apache Flume also sort of supports plugins,
> which I previously mentioned. The main ones are Sinks, Channels, and
> Sources.
> In the Flume configuration Sinks provided by Flume can be referenced by
> their “simple” name because they are manually registered in the code. User
> provided components have to provide the whole fully qualified class name to
> be located.
>
> In addition, Flume components must locate their own configuration.
> This limits the format the configuration can take. For example, Flume
> currently
> uses properties but I would like to support JSON and YAML Due to the way
> property support was implemented it is difficult to support either of
> those without
> impacting users who have built custom components. This is where using
> injection
> does help. The plugins would be insensitive to the configuration format so
> making
> changes would just require changing how the configuration binds to the
> plugin system.
>
> > - Based on the gathered knowledge on the previous point, what is the high
> > level architecture of a plugin system the library will support and what
> > services will it provide?
>
> I think that is TBD. Although we have suggested starting from Log4j’s
> code, that
> isn’t a requirement. I would suggest that some experimental code be placed
> in a
> project and then whoever is interested can collaborate and debate the pros
> and cons.
> That is exactly how Log4j 2 got started.
>
> Ralph
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: New component proposal: commons-plugins

[Bernd Eckenfels]

Hello,

I don’t think plug-ins is something which fits well in the scope of commons, as 
it does tie into a broader ecosystem/platform normally. but I could be wrong.

Btw: Commons VFS has a plug-in system, mostly for providers.

Gruss
Bernd
--
http://bernd.eckenfels.net

Von: Ralph Goers 
Gesendet: Monday, April 11, 2022 12:00:12 AM
An: Commons Developers List 
Betreff: Re: New component proposal: commons-plugins

See below

> On Apr 8, 2022, at 9:23 AM, Peter Verhas  wrote:
>
> Thanks Ralph for the detailed explanation. I appreciate it and now I see
> the points.

I’ve removed the parts that I don’t think need any more discussion.

> - How will it be a “plugin" project and not another dependency injection
> framework?

This is a great question. I think the main difference is with examples like 
Log4j
and Apache Flume, and even Apache Maven. All wire components together via
user provided configuration, not code. Dependency injection could certainly be
part of the plugin framework but that would be for implementors of plugins,
not the users using them. Users of Maven don’t know that Plexus is used under
the covers and neither should users of a commons-plugins implementation.

> - What will distinguish it from module systems, like OSGi and what will
> stop it from becoming another OSGi by the years as new features get added
> to the library.

OSGi is complicated. Implementing plugins should not be. Just as I described
using ServiceLoader to locate plugins, plugins should also be accessible in
OSGi bundles. Users should be required to do as little as possible to adapt the
plugin system to the environment it is running in but plugins shouldn’t know or
care anything about how they are located and loaded. Plugins are also to the
application or framework that will be using them. They essentially define what
the valid plugin types are and where they can be used.

> - What applications using plugins are the examples for different solutions?
> (Log4j is a good example to show that there is a need, you also explained
> patiently why it is not a simple ServiceLoader, but it is only one way to
> solve it. Other applications may approach the issue differently. Maven,
> Attlassian products, other build tools, JUnit 5 and so on.)

We’ve already mentioned Log4j. Apache Flume also sort of supports plugins,
which I previously mentioned. The main ones are Sinks, Channels, and Sources.
In the Flume configuration Sinks provided by Flume can be referenced by
their “simple” name because they are manually registered in the code. User
provided components have to provide the whole fully qualified class name to
be located.

In addition, Flume components must locate their own configuration.
This limits the format the configuration can take. For example, Flume currently
uses properties but I would like to support JSON and YAML Due to the way
property support was implemented it is difficult to support either of those 
without
impacting users who have built custom components. This is where using injection
does help. The plugins would be insensitive to the configuration format so 
making
changes would just require changing how the configuration binds to the plugin 
system.

> - Based on the gathered knowledge on the previous point, what is the high
> level architecture of a plugin system the library will support and what
> services will it provide?

I think that is TBD. Although we have suggested starting from Log4j’s code, that
isn’t a requirement. I would suggest that some experimental code be placed in a
project and then whoever is interested can collaborate and debate the pros and 
cons.
That is exactly how Log4j 2 got started.

Ralph


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: New component proposal: commons-plugins

[Ralph Goers]

See below

> On Apr 8, 2022, at 9:23 AM, Peter Verhas  wrote:
> 
> Thanks Ralph for the detailed explanation. I appreciate it and now I see
> the points.

I’ve removed the parts that I don’t think need any more discussion.

> - How will it be a “plugin" project and not another dependency injection
> framework?

This is a great question. I think the main difference is with examples like 
Log4j 
and Apache Flume, and even Apache Maven. All wire components together via 
user provided configuration, not code. Dependency injection could certainly be 
part of the plugin framework but that would be for implementors of plugins, 
not the users using them. Users of Maven don’t know that Plexus is used under 
the covers and neither should users of a commons-plugins implementation.

> - What will distinguish it from module systems, like OSGi and what will
> stop it from becoming another OSGi by the years as new features get added
> to the library.

OSGi is complicated. Implementing plugins should not be. Just as I described 
using ServiceLoader to locate plugins, plugins should also be accessible in 
OSGi bundles. Users should be required to do as little as possible to adapt the 
plugin system to the environment it is running in but plugins shouldn’t know or 
care anything about how they are located and loaded. Plugins are also to the 
application or framework that will be using them. They essentially define what 
the valid plugin types are and where they can be used.

> - What applications using plugins are the examples for different solutions?
> (Log4j is a good example to show that there is a need, you also explained
> patiently why it is not a simple ServiceLoader, but it is only one way to
> solve it. Other applications may approach the issue differently. Maven,
> Attlassian products, other build tools, JUnit 5 and so on.)

We’ve already mentioned Log4j. Apache Flume also sort of supports plugins, 
which I previously mentioned. The main ones are Sinks, Channels, and Sources. 
In the Flume configuration Sinks provided by Flume can be referenced by 
their “simple” name because they are manually registered in the code. User 
provided components have to provide the whole fully qualified class name to 
be located. 

In addition, Flume components must locate their own configuration. 
This limits the format the configuration can take. For example, Flume currently 
uses properties but I would like to support JSON and YAML Due to the way 
property support was implemented it is difficult to support either of those 
without 
impacting users who have built custom components. This is where using injection 
does help. The plugins would be insensitive to the configuration format so 
making 
changes would just require changing how the configuration binds to the plugin 
system.

> - Based on the gathered knowledge on the previous point, what is the high
> level architecture of a plugin system the library will support and what
> services will it provide?

I think that is TBD. Although we have suggested starting from Log4j’s code, 
that 
isn’t a requirement. I would suggest that some experimental code be placed in a 
project and then whoever is interested can collaborate and debate the pros and 
cons. 
That is exactly how Log4j 2 got started.

Ralph


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: New component proposal: commons-plugins

[Romain Manni-Bucau]

Le ven. 8 avr. 2022 à 19:28, Matt Sicker  a écrit :

> One iteration of the plugin system was based on Weld and OpenWebBeans,
> but the CDI API is far too verbose for a use case where I didn't have
> a need to integrate with a decade worth of legacy Java EE APIs (no
> JNDI, no JPA, no EJB, no Resource(s) annotations, none of the
> DI-alternatives leading up to CDI).
>

Hmm, not sure what you ref to but cdi standalone is quite light and does
not depend on anything else than atinject.


> For changing annotations at runtime, the idea I had here was to be
> able to either register annotations in the API so that you can opt-in
> to which annotations to use, or we could check the name of the
> annotation class and use anything properly named (similar to how
> Lombok treats nullability annotations). So in either case, I'd follow
> an approach like that in order to support all of javax, jakarta, and
> shaded versions of the same annotations.
>

So you take cdi path somehow, this is one of my point.
Whatever plugins you need, you'll need IoC feature on one side and whatever
system you have you want it integrated with your app IoC on another side so
you dont want another mecanism IMHO.


> Yes, the current DI system from Log4j supports field, constructor, and
> method injection. I did not add interceptors and decorators as we
> didn't have a need for them (we've found it easier to create plugins
> for this purpose instead, though I can see these concepts being
> potentially useful in other apps).
>

Yet, look a cli, crest thought it was not needed but it came up quickly as
an important need, will be the same IMHO.


> Most DI systems I've ever used or tried using are optimized for the
> use case where the configurer is also the same person compiling and
> packaging the application in the first place.


Not CDI ;).

This works fairly well
> for custom applications, but it gets extremely complicated when you
> try to expose some of this configurability to the end user of the
> application (or library). I love to point at the Spinnaker project for
> this as that's what I've been working on as a day job for about a year
> now, and we have a few attempts at making this experience suck less,
> but it ended up with a barely maintained additional microservice whose
> job is to translate one input Spring config file into like 10+ config
> files, but this only works with features that explicitly add
> themselves to the configuration system (less common when working on
> Spinnaker directly as you incrementally update your Spring config
> files). While there's a fairly new plugin system available there, a
> ton of existing functionality is still integrated directly in
> Spinnaker which is an example of where you'd have far more code at
> runtime on the classpath than you actually desire. Log4j is another
> similar project with plenty of optional plugins baked in the core, but
> that's being split up more successfully than Spinnaker has been at
> doing similarly.
>
> From a CLI plugin perspective, my thoughts here would be to make it
> simpler to make an extensible CLI tool like kubectl.


Understood but still. Crest+cdi is not far.

Perhaps something
> like Quarkus would be useful here, or it would be useful as a DI
> backend (I did mention Avaje Inject which has a similar code
> generation feature to Quarkus but without the non-DI features).
>

Quarkus being an half-baked cdi a bit fatter i guess cdi is still an
option...and to be honest spring and guice reach the same goal technically
as of today, they are just more or less neat technically and more or less
fat in terms of stack.

So overall I can see the temptation but not sure what it brings technically.

That said, once again, if you have a community to get started with this is
not a blocker ;).


> On Fri, Apr 8, 2022 at 12:06 PM Romain Manni-Bucau
>  wrote:
> >
> > Le ven. 8 avr. 2022 à 18:50, Matt Sicker  a écrit :
> >
> > > I suspect at this point that most of the remaining slowness in startup
> > > on Log4j is related to code that _doesn't_ use plugins. There are some
> > > strategies that configure on startup in log4j-api based on system
> > > properties and service loaders which are provided for improved
> > > steady-state performance (or for garbage-free logging) but add
> > > complexity to startup.
> > >
> >
> > The light IoC and scanner also are not really fast for a generic
> embeddable
> > plugin system.
> >
> >
> > >
> > > From a DI point of view, I followed the standard @Inject style API
> > > that projects such as Guice already use, though I included my own
> > > copies of the javax annotations since we weren't relying on additional
> > > dependencies there. In a generic plugins framework, this could be
> > > abstracted further to support any general DI framework.
> > >
> >
> > So it means your solution can:
> >
> > 1. be modified at startup (ie I can change the annotation without any
> > bytecode rewriting/agent/classfiletransformer)?
> > 2. remove some pl

Re: New component proposal: commons-plugins

[Matt Sicker]

One iteration of the plugin system was based on Weld and OpenWebBeans,
but the CDI API is far too verbose for a use case where I didn't have
a need to integrate with a decade worth of legacy Java EE APIs (no
JNDI, no JPA, no EJB, no Resource(s) annotations, none of the
DI-alternatives leading up to CDI).

For changing annotations at runtime, the idea I had here was to be
able to either register annotations in the API so that you can opt-in
to which annotations to use, or we could check the name of the
annotation class and use anything properly named (similar to how
Lombok treats nullability annotations). So in either case, I'd follow
an approach like that in order to support all of javax, jakarta, and
shaded versions of the same annotations.

Yes, the current DI system from Log4j supports field, constructor, and
method injection. I did not add interceptors and decorators as we
didn't have a need for them (we've found it easier to create plugins
for this purpose instead, though I can see these concepts being
potentially useful in other apps).

Most DI systems I've ever used or tried using are optimized for the
use case where the configurer is also the same person compiling and
packaging the application in the first place. This works fairly well
for custom applications, but it gets extremely complicated when you
try to expose some of this configurability to the end user of the
application (or library). I love to point at the Spinnaker project for
this as that's what I've been working on as a day job for about a year
now, and we have a few attempts at making this experience suck less,
but it ended up with a barely maintained additional microservice whose
job is to translate one input Spring config file into like 10+ config
files, but this only works with features that explicitly add
themselves to the configuration system (less common when working on
Spinnaker directly as you incrementally update your Spring config
files). While there's a fairly new plugin system available there, a
ton of existing functionality is still integrated directly in
Spinnaker which is an example of where you'd have far more code at
runtime on the classpath than you actually desire. Log4j is another
similar project with plenty of optional plugins baked in the core, but
that's being split up more successfully than Spinnaker has been at
doing similarly.

>From a CLI plugin perspective, my thoughts here would be to make it
simpler to make an extensible CLI tool like kubectl. Perhaps something
like Quarkus would be useful here, or it would be useful as a DI
backend (I did mention Avaje Inject which has a similar code
generation feature to Quarkus but without the non-DI features).

On Fri, Apr 8, 2022 at 12:06 PM Romain Manni-Bucau
 wrote:
>
> Le ven. 8 avr. 2022 à 18:50, Matt Sicker  a écrit :
>
> > I suspect at this point that most of the remaining slowness in startup
> > on Log4j is related to code that _doesn't_ use plugins. There are some
> > strategies that configure on startup in log4j-api based on system
> > properties and service loaders which are provided for improved
> > steady-state performance (or for garbage-free logging) but add
> > complexity to startup.
> >
>
> The light IoC and scanner also are not really fast for a generic embeddable
> plugin system.
>
>
> >
> > From a DI point of view, I followed the standard @Inject style API
> > that projects such as Guice already use, though I included my own
> > copies of the javax annotations since we weren't relying on additional
> > dependencies there. In a generic plugins framework, this could be
> > abstracted further to support any general DI framework.
> >
>
> So it means your solution can:
>
> 1. be modified at startup (ie I can change the annotation without any
> bytecode rewriting/agent/classfiletransformer)?
> 2. remove some plugin at startup writing a simple extension outside the
> plugin code (and not just remove it after it is added)
> 3. it supports @Inject everywhere (constructor, fields, setters)
> 4. it it supports javax and jakarta (don't blame me ;))
> 5. it supports interceptors and decorators
>  etc...
>
> This is why a bridge solution is rarely a good solution and that as soon as
> you have an IoC you have a built-in plugin solution so never need another
> one.
>
>
> >
> > While the point of the plugin system is to be a much simpler approach
> > than something as complex as OSGi, it does seem like a valid concern
> > that it could become a sort of OSGi clone over time. With that in
> > mind, it becomes interesting to think what such a plugin system built
> > on top of OSGi would look like, though I haven't really explored that
> > possibility much (it was something I suggested long ago in the Jenkins
> > project as a way to improve and standardize their plugin system,
> > though there seems to be too much tech debt there to successfully do
> > so).
> >
>
> I was not even at OSGi, just plain CDI/Spring case when I said it wouldn't
> be that helping ot write app

Re: New component proposal: commons-plugins

[Romain Manni-Bucau]

Le ven. 8 avr. 2022 à 18:50, Matt Sicker  a écrit :

> I suspect at this point that most of the remaining slowness in startup
> on Log4j is related to code that _doesn't_ use plugins. There are some
> strategies that configure on startup in log4j-api based on system
> properties and service loaders which are provided for improved
> steady-state performance (or for garbage-free logging) but add
> complexity to startup.
>

The light IoC and scanner also are not really fast for a generic embeddable
plugin system.


>
> From a DI point of view, I followed the standard @Inject style API
> that projects such as Guice already use, though I included my own
> copies of the javax annotations since we weren't relying on additional
> dependencies there. In a generic plugins framework, this could be
> abstracted further to support any general DI framework.
>

So it means your solution can:

1. be modified at startup (ie I can change the annotation without any
bytecode rewriting/agent/classfiletransformer)?
2. remove some plugin at startup writing a simple extension outside the
plugin code (and not just remove it after it is added)
3. it supports @Inject everywhere (constructor, fields, setters)
4. it it supports javax and jakarta (don't blame me ;))
5. it supports interceptors and decorators
 etc...

This is why a bridge solution is rarely a good solution and that as soon as
you have an IoC you have a built-in plugin solution so never need another
one.


>
> While the point of the plugin system is to be a much simpler approach
> than something as complex as OSGi, it does seem like a valid concern
> that it could become a sort of OSGi clone over time. With that in
> mind, it becomes interesting to think what such a plugin system built
> on top of OSGi would look like, though I haven't really explored that
> possibility much (it was something I suggested long ago in the Jenkins
> project as a way to improve and standardize their plugin system,
> though there seems to be too much tech debt there to successfully do
> so).
>

I was not even at OSGi, just plain CDI/Spring case when I said it wouldn't
be that helping ot write apps, even graalvm ones.


>
> A potential design goal that could avoid OSGi-ifying the library would
> be to aim to be a superfast, lightweight plugin system that can work
> well with libraries, CLI apps (particularly interesting when using
> GraalVM for AOT compilation or when using jlink/jpackage/etc. to
> create a standalone application), and server software. If the DI
> aspect is sufficiently pluggable, then that would allow for using
> different DI backends appropriate to the target environment (e.g.,
> using Spring in a long-running server app, or Avaje for a CLI app with
> its code generation feature, or even the built-in DI impl for a
> minimal-dependency library).
>

CDI fits that description AFAIK (I'm using openwebbeans and tomitribe crest
quite often for CLI implementations but guess quarkus can be an option too
if you don't care to be fatter)


>
> On Fri, Apr 8, 2022 at 2:38 AM Romain Manni-Bucau 
> wrote:
> >
> > Guess that theorically there is room for a new generic plugin system but
> I
> > would mention a few points:
> >
> > 1. log4j one is not yet a good *generic* one for multiple reasons but the
> > biggest blocker for me is that it is slow (slower than a plain IoC as of
> > today, even dropping some legacy parts) - there is a ticket about it but
> we
> > are still slower than a plain CDI container even if it is way better than
> > 2.x on 3.x branch
> > 2. Most naturally you want your plugin system to integrate your IoC
> > (spring, CDI, guice, ...) - to get injections, same bean pattern than
> your
> > IoC/app, extensions,  If we go with log4j system then we must do
> > bridges which are quite useless in practise so it is an abstraction you
> > will fight against instead of leveraging.
> > 3. If there is a community around this project then it fits apache so it
> > can go but I think it belongs better to incubator and not commons
> >
> > Hope it makes sense
> >
> > Romain Manni-Bucau
> > @rmannibucau  |  Blog
> >  | Old Blog
> >  | Github <
> https://github.com/rmannibucau> |
> > LinkedIn  | Book
> > <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> >
> >
> > Le ven. 8 avr. 2022 à 09:23, Peter Verhas  a écrit :
> >
> > > Thanks Ralph for the detailed explanation. I appreciate it and now I
> see
> > > the points.
> > >
> > > I have never experienced an application where class loading time was an
> > > issue, and I understand that it can really be.
> > > I have never experienced a setup where there were a lot of "plugin"
> classes
> > > on the classpath or on the modulepath that were never loaded by the
> > > application.
> > >
> > > We still are not on the same page about module systems. I am happily
> u

Re: New component proposal: commons-plugins

[Matt Sicker]

I suspect at this point that most of the remaining slowness in startup
on Log4j is related to code that _doesn't_ use plugins. There are some
strategies that configure on startup in log4j-api based on system
properties and service loaders which are provided for improved
steady-state performance (or for garbage-free logging) but add
complexity to startup.

>From a DI point of view, I followed the standard @Inject style API
that projects such as Guice already use, though I included my own
copies of the javax annotations since we weren't relying on additional
dependencies there. In a generic plugins framework, this could be
abstracted further to support any general DI framework.

While the point of the plugin system is to be a much simpler approach
than something as complex as OSGi, it does seem like a valid concern
that it could become a sort of OSGi clone over time. With that in
mind, it becomes interesting to think what such a plugin system built
on top of OSGi would look like, though I haven't really explored that
possibility much (it was something I suggested long ago in the Jenkins
project as a way to improve and standardize their plugin system,
though there seems to be too much tech debt there to successfully do
so).

A potential design goal that could avoid OSGi-ifying the library would
be to aim to be a superfast, lightweight plugin system that can work
well with libraries, CLI apps (particularly interesting when using
GraalVM for AOT compilation or when using jlink/jpackage/etc. to
create a standalone application), and server software. If the DI
aspect is sufficiently pluggable, then that would allow for using
different DI backends appropriate to the target environment (e.g.,
using Spring in a long-running server app, or Avaje for a CLI app with
its code generation feature, or even the built-in DI impl for a
minimal-dependency library).

On Fri, Apr 8, 2022 at 2:38 AM Romain Manni-Bucau  wrote:
>
> Guess that theorically there is room for a new generic plugin system but I
> would mention a few points:
>
> 1. log4j one is not yet a good *generic* one for multiple reasons but the
> biggest blocker for me is that it is slow (slower than a plain IoC as of
> today, even dropping some legacy parts) - there is a ticket about it but we
> are still slower than a plain CDI container even if it is way better than
> 2.x on 3.x branch
> 2. Most naturally you want your plugin system to integrate your IoC
> (spring, CDI, guice, ...) - to get injections, same bean pattern than your
> IoC/app, extensions,  If we go with log4j system then we must do
> bridges which are quite useless in practise so it is an abstraction you
> will fight against instead of leveraging.
> 3. If there is a community around this project then it fits apache so it
> can go but I think it belongs better to incubator and not commons
>
> Hope it makes sense
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github  |
> LinkedIn  | Book
> 
>
>
> Le ven. 8 avr. 2022 à 09:23, Peter Verhas  a écrit :
>
> > Thanks Ralph for the detailed explanation. I appreciate it and now I see
> > the points.
> >
> > I have never experienced an application where class loading time was an
> > issue, and I understand that it can really be.
> > I have never experienced a setup where there were a lot of "plugin" classes
> > on the classpath or on the modulepath that were never loaded by the
> > application.
> >
> > We still are not on the same page about module systems. I am happily using
> > it in all of my new projects and my experience is mixed. I accept the
> > struggle that it takes to make it properly, like opening the packaged via
> > command line options for testing purposes. On the other hand the structure
> > and the encapsulation is one step better than without JPMS.
> >
> > >But since you don’t want to look at he Log4j plugin system
> >
> > I did have a look at it to some level, as I also mentioned it  in my
> > previous mail. I am not an expert in that field as you are and I never
> > will, and that is what I do not want to be.
> >
> > I accept that there is room for a commons-plugin project. The project has
> > to address these questions to be on the right track (not to me, to the
> > project itself):
> >
> > - How will it be a "plugin" project and not another dependency injection
> > framework?
> > - What will distinguish it from module systems, like OSGi and what will
> > stop it from becoming another OSGi by the years as new features get added
> > to the library.
> > - What applications using plugins are the examples for different solutions?
> > (Log4j is a good example to show that there is a need, you also explained
> > patiently why it is not a simple ServiceLoader, but it is only on

Re: New component proposal: commons-plugins

[Romain Manni-Bucau]

Guess that theorically there is room for a new generic plugin system but I
would mention a few points:

1. log4j one is not yet a good *generic* one for multiple reasons but the
biggest blocker for me is that it is slow (slower than a plain IoC as of
today, even dropping some legacy parts) - there is a ticket about it but we
are still slower than a plain CDI container even if it is way better than
2.x on 3.x branch
2. Most naturally you want your plugin system to integrate your IoC
(spring, CDI, guice, ...) - to get injections, same bean pattern than your
IoC/app, extensions,  If we go with log4j system then we must do
bridges which are quite useless in practise so it is an abstraction you
will fight against instead of leveraging.
3. If there is a community around this project then it fits apache so it
can go but I think it belongs better to incubator and not commons

Hope it makes sense

Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book



Le ven. 8 avr. 2022 à 09:23, Peter Verhas  a écrit :

> Thanks Ralph for the detailed explanation. I appreciate it and now I see
> the points.
>
> I have never experienced an application where class loading time was an
> issue, and I understand that it can really be.
> I have never experienced a setup where there were a lot of "plugin" classes
> on the classpath or on the modulepath that were never loaded by the
> application.
>
> We still are not on the same page about module systems. I am happily using
> it in all of my new projects and my experience is mixed. I accept the
> struggle that it takes to make it properly, like opening the packaged via
> command line options for testing purposes. On the other hand the structure
> and the encapsulation is one step better than without JPMS.
>
> >But since you don’t want to look at he Log4j plugin system
>
> I did have a look at it to some level, as I also mentioned it  in my
> previous mail. I am not an expert in that field as you are and I never
> will, and that is what I do not want to be.
>
> I accept that there is room for a commons-plugin project. The project has
> to address these questions to be on the right track (not to me, to the
> project itself):
>
> - How will it be a "plugin" project and not another dependency injection
> framework?
> - What will distinguish it from module systems, like OSGi and what will
> stop it from becoming another OSGi by the years as new features get added
> to the library.
> - What applications using plugins are the examples for different solutions?
> (Log4j is a good example to show that there is a need, you also explained
> patiently why it is not a simple ServiceLoader, but it is only one way to
> solve it. Other applications may approach the issue differently. Maven,
> Attlassian products, other build tools, JUnit 5 and so on.)
> - Based on the gathered knowledge on the previous point, what is the high
> level architecture of a plugin system the library will support and what
> services will it provide?
>
>
>
>
>
> On Thu, Apr 7, 2022 at 7:57 PM Ralph Goers 
> wrote:
>
> >
> >
> > > On Apr 7, 2022, at 2:52 AM, Peter Verhas  wrote:
> > >
> > >
> > >> I would suggest that before responding to this email that
> > >> you go look at how log4j-plugins is implemented in the master branch.
> > >>
> > >
> > > Sure, if you propose to create a commons-plugin library that is an
> > extract
> > > of the plugin handling of log4j so that log4j next releases do not need
> > to
> > > keep the functionality inside log4j but rather as a dependency on this
> > new
> > > library. On the other hand, if the goal is to create something that is
> > > useful generally then we should look at other solutions as well.
> >
> > It remains to be seen whether Log4j would replace log4j-plugins with
> > Commons-plugins. The point of looking at Log4j was to understand the
> > problem it is solving and how it solves it.
> >
> >
> > >
> > >
> > >> From this it is clear that you completely misunderstand what the
> plugin
> > >> system is doing.
> > >>
> > >
> > > That may really be very much true.
> > >
> > >
> > >> First, the plugin system DOES use ServiceLoader. In fact, Log4j uses
> > >> ServiceLoader
> > >> in at least 7 different places. For one, it uses ServiceLoader to
> locate
> > >> the Log4j
> > >> implementation similar to how SLF4J does (although Log4j implemented
> it
> > >> prior to SLF4J).
> > >>
> > >> Consider that you have 150 or so plugins, of which maybe only 10 will
> > get
> > >> used. But
> > >> you don’t know beforehand which 10. ServiceLoader would have to load
> all
> > >> 150 classes.
> > >
> > >
> > > Not. It does not need to  create instances of all of them. This is
> what I
> > > explained, and this is the re

Re: New component proposal: commons-plugins

[Peter Verhas]

Thanks Ralph for the detailed explanation. I appreciate it and now I see
the points.

I have never experienced an application where class loading time was an
issue, and I understand that it can really be.
I have never experienced a setup where there were a lot of "plugin" classes
on the classpath or on the modulepath that were never loaded by the
application.

We still are not on the same page about module systems. I am happily using
it in all of my new projects and my experience is mixed. I accept the
struggle that it takes to make it properly, like opening the packaged via
command line options for testing purposes. On the other hand the structure
and the encapsulation is one step better than without JPMS.

>But since you don’t want to look at he Log4j plugin system

I did have a look at it to some level, as I also mentioned it  in my
previous mail. I am not an expert in that field as you are and I never
will, and that is what I do not want to be.

I accept that there is room for a commons-plugin project. The project has
to address these questions to be on the right track (not to me, to the
project itself):

- How will it be a "plugin" project and not another dependency injection
framework?
- What will distinguish it from module systems, like OSGi and what will
stop it from becoming another OSGi by the years as new features get added
to the library.
- What applications using plugins are the examples for different solutions?
(Log4j is a good example to show that there is a need, you also explained
patiently why it is not a simple ServiceLoader, but it is only one way to
solve it. Other applications may approach the issue differently. Maven,
Attlassian products, other build tools, JUnit 5 and so on.)
- Based on the gathered knowledge on the previous point, what is the high
level architecture of a plugin system the library will support and what
services will it provide?





On Thu, Apr 7, 2022 at 7:57 PM Ralph Goers 
wrote:

>
>
> > On Apr 7, 2022, at 2:52 AM, Peter Verhas  wrote:
> >
> >
> >> I would suggest that before responding to this email that
> >> you go look at how log4j-plugins is implemented in the master branch.
> >>
> >
> > Sure, if you propose to create a commons-plugin library that is an
> extract
> > of the plugin handling of log4j so that log4j next releases do not need
> to
> > keep the functionality inside log4j but rather as a dependency on this
> new
> > library. On the other hand, if the goal is to create something that is
> > useful generally then we should look at other solutions as well.
>
> It remains to be seen whether Log4j would replace log4j-plugins with
> Commons-plugins. The point of looking at Log4j was to understand the
> problem it is solving and how it solves it.
>
>
> >
> >
> >> From this it is clear that you completely misunderstand what the plugin
> >> system is doing.
> >>
> >
> > That may really be very much true.
> >
> >
> >> First, the plugin system DOES use ServiceLoader. In fact, Log4j uses
> >> ServiceLoader
> >> in at least 7 different places. For one, it uses ServiceLoader to locate
> >> the Log4j
> >> implementation similar to how SLF4J does (although Log4j implemented it
> >> prior to SLF4J).
> >>
> >> Consider that you have 150 or so plugins, of which maybe only 10 will
> get
> >> used. But
> >> you don’t know beforehand which 10. ServiceLoader would have to load all
> >> 150 classes.
> >
> >
> > Not. It does not need to  create instances of all of them. This is what I
> > explained, and this is the reason why I explained it. JPMS or not, the
> new
> > ServiceLoader used by Java 9 and later will call the static factory
> method
> > of the class if that exists. Then the plugin can decide if it wants
> itself
> > to be instantiated or not. Somewhere there should be some logic that
> > filters 10 from the 150. If that is in a central place implementing some
> > industry-best-practice configuration based algorithm, or outsources the
> > decision to the plugins themselves should not make a big difference in
> > performance.
> >
> > On  the second though, instantiating 150 lightweight classes must not be
> a
> > big burden. The ServiceProvider JavaDoc documentation suggest that
> services
> > should be implemented as proxies or factories instantiating their real
> > working class lazily in case the instantiation is costly.
>
> Class loading in general is slow. Perhaps not to you but we have customers
> who
> complain about all the things we do during startup and locating and
> instantiating
> plugins is one of them. When we first switched to use the ServiceLoader we
> attempted to load the plugin classes simply by having the plugin
> definition include
> the Class.Performance was very bad. We now only include the class name and
> only access the Class when absolutely necessary.
>
> Again, if you think about what the Plugin Manager is doing, it is using
> ServiceLoader
> to find all Plugin Service instances. Each of these contains the
> definitions of all the
> pl

Re: New component proposal: commons-plugins

[Matt Sicker]

One of the issues I've found with loading classes eagerly is that many
ClassLoader implementations rely on fairly broad locks. Deferred class
loading can avoid some of this lock contention.

On Thu, Apr 7, 2022 at 12:57 PM Ralph Goers  wrote:
>
>
>
> > On Apr 7, 2022, at 2:52 AM, Peter Verhas  wrote:
> >
> >
> >> I would suggest that before responding to this email that
> >> you go look at how log4j-plugins is implemented in the master branch.
> >>
> >
> > Sure, if you propose to create a commons-plugin library that is an extract
> > of the plugin handling of log4j so that log4j next releases do not need to
> > keep the functionality inside log4j but rather as a dependency on this new
> > library. On the other hand, if the goal is to create something that is
> > useful generally then we should look at other solutions as well.
>
> It remains to be seen whether Log4j would replace log4j-plugins with
> Commons-plugins. The point of looking at Log4j was to understand the
> problem it is solving and how it solves it.
>
>
> >
> >
> >> From this it is clear that you completely misunderstand what the plugin
> >> system is doing.
> >>
> >
> > That may really be very much true.
> >
> >
> >> First, the plugin system DOES use ServiceLoader. In fact, Log4j uses
> >> ServiceLoader
> >> in at least 7 different places. For one, it uses ServiceLoader to locate
> >> the Log4j
> >> implementation similar to how SLF4J does (although Log4j implemented it
> >> prior to SLF4J).
> >>
> >> Consider that you have 150 or so plugins, of which maybe only 10 will get
> >> used. But
> >> you don’t know beforehand which 10. ServiceLoader would have to load all
> >> 150 classes.
> >
> >
> > Not. It does not need to  create instances of all of them. This is what I
> > explained, and this is the reason why I explained it. JPMS or not, the new
> > ServiceLoader used by Java 9 and later will call the static factory method
> > of the class if that exists. Then the plugin can decide if it wants itself
> > to be instantiated or not. Somewhere there should be some logic that
> > filters 10 from the 150. If that is in a central place implementing some
> > industry-best-practice configuration based algorithm, or outsources the
> > decision to the plugins themselves should not make a big difference in
> > performance.
> >
> > On  the second though, instantiating 150 lightweight classes must not be a
> > big burden. The ServiceProvider JavaDoc documentation suggest that services
> > should be implemented as proxies or factories instantiating their real
> > working class lazily in case the instantiation is costly.
>
> Class loading in general is slow. Perhaps not to you but we have customers who
> complain about all the things we do during startup and locating and 
> instantiating
> plugins is one of them. When we first switched to use the ServiceLoader we
> attempted to load the plugin classes simply by having the plugin definition 
> include
> the Class.Performance was very bad. We now only include the class name and
> only access the Class when absolutely necessary.
>
> Again, if you think about what the Plugin Manager is doing, it is using 
> ServiceLoader
> to find all Plugin Service instances. Each of these contains the definitions 
> of all the
> plugins. The Plugin Manager then uses these to create the actual plugins. So 
> in
> essence, the services are used as part of the factory. Note that decisions 
> about
> whether to load a plugin or not do not have to be made when the plugin service
> s accessed. That is deferred until the Plugin actually needs to be wired.
>
> >
> >
> >> This is indeed very slow. Yes we have benchmarked it. But the Plugin
> >> system IS a
> >> Java Service. So instead, you need to only load 1 class for every jar that
> >> contains plugins.
> >> The result is that you only need to load 4 or 5 classes. This is
> >> considerably faster.
> >>
> >
> > Yes, this is the approach where you insert another layer above the service
> > loader that does part of the job that the service loader can also do in a
> > different way. Essentially you let the plugin libraries, each containing
> > several plugin classes to implement their own service loaders.
>
> Sort of. With Log4j’s plugin system Plugins are not aware of ServiceLoader
> at all. Other than that, it behaves the way you are describing.
>
> >
> >
> >> In addition, services loaded by ServiceLoader are singletons.
> >
> >
> > No, they are not. Every ServiceLoader instance will create new instances,
> > and even a single ServiceLoader can be asked calling the `reload()` method
> > to evict the instances from its cache and create new instances.
>
> Umm. Even calling reload() doesn’t change it from being a singleton. You can
> only have a single instance at a time.
>
> Yes, I suppose if you use multiple ServiceLoader instances then you can I
> nstantiate multiple instances of the service. However, as you noted that 
> wouldn’t
> be the best practice since the ServiceLoader really

Re: New component proposal: commons-plugins

[Ralph Goers]

> On Apr 7, 2022, at 2:52 AM, Peter Verhas  wrote:
> 
> 
>> I would suggest that before responding to this email that
>> you go look at how log4j-plugins is implemented in the master branch.
>> 
> 
> Sure, if you propose to create a commons-plugin library that is an extract
> of the plugin handling of log4j so that log4j next releases do not need to
> keep the functionality inside log4j but rather as a dependency on this new
> library. On the other hand, if the goal is to create something that is
> useful generally then we should look at other solutions as well.

It remains to be seen whether Log4j would replace log4j-plugins with 
Commons-plugins. The point of looking at Log4j was to understand the 
problem it is solving and how it solves it.


> 
> 
>> From this it is clear that you completely misunderstand what the plugin
>> system is doing.
>> 
> 
> That may really be very much true.
> 
> 
>> First, the plugin system DOES use ServiceLoader. In fact, Log4j uses
>> ServiceLoader
>> in at least 7 different places. For one, it uses ServiceLoader to locate
>> the Log4j
>> implementation similar to how SLF4J does (although Log4j implemented it
>> prior to SLF4J).
>> 
>> Consider that you have 150 or so plugins, of which maybe only 10 will get
>> used. But
>> you don’t know beforehand which 10. ServiceLoader would have to load all
>> 150 classes.
> 
> 
> Not. It does not need to  create instances of all of them. This is what I
> explained, and this is the reason why I explained it. JPMS or not, the new
> ServiceLoader used by Java 9 and later will call the static factory method
> of the class if that exists. Then the plugin can decide if it wants itself
> to be instantiated or not. Somewhere there should be some logic that
> filters 10 from the 150. If that is in a central place implementing some
> industry-best-practice configuration based algorithm, or outsources the
> decision to the plugins themselves should not make a big difference in
> performance.
> 
> On  the second though, instantiating 150 lightweight classes must not be a
> big burden. The ServiceProvider JavaDoc documentation suggest that services
> should be implemented as proxies or factories instantiating their real
> working class lazily in case the instantiation is costly.

Class loading in general is slow. Perhaps not to you but we have customers who 
complain about all the things we do during startup and locating and 
instantiating
plugins is one of them. When we first switched to use the ServiceLoader we 
attempted to load the plugin classes simply by having the plugin definition 
include 
the Class.Performance was very bad. We now only include the class name and 
only access the Class when absolutely necessary.

Again, if you think about what the Plugin Manager is doing, it is using 
ServiceLoader 
to find all Plugin Service instances. Each of these contains the definitions of 
all the 
plugins. The Plugin Manager then uses these to create the actual plugins. So in 
essence, the services are used as part of the factory. Note that decisions 
about 
whether to load a plugin or not do not have to be made when the plugin service 
s accessed. That is deferred until the Plugin actually needs to be wired.

> 
> 
>> This is indeed very slow. Yes we have benchmarked it. But the Plugin
>> system IS a
>> Java Service. So instead, you need to only load 1 class for every jar that
>> contains plugins.
>> The result is that you only need to load 4 or 5 classes. This is
>> considerably faster.
>> 
> 
> Yes, this is the approach where you insert another layer above the service
> loader that does part of the job that the service loader can also do in a
> different way. Essentially you let the plugin libraries, each containing
> several plugin classes to implement their own service loaders.

Sort of. With Log4j’s plugin system Plugins are not aware of ServiceLoader 
at all. Other than that, it behaves the way you are describing.

> 
> 
>> In addition, services loaded by ServiceLoader are singletons.
> 
> 
> No, they are not. Every ServiceLoader instance will create new instances,
> and even a single ServiceLoader can be asked calling the `reload()` method
> to evict the instances from its cache and create new instances.

Umm. Even calling reload() doesn’t change it from being a singleton. You can 
only have a single instance at a time.

Yes, I suppose if you use multiple ServiceLoader instances then you can I
nstantiate multiple instances of the service. However, as you noted that 
wouldn’t 
be the best practice since the ServiceLoader really should be instantiating a 
factory that create as many instances as you want.

> 
> Perhaps developing a commons-plugin system is a good point to require
> application developers and plugin developers to use the module system.

Please no. I would never encourage anyone to use it. It has caused nothing 
but pain. Yes, that is opinion, but opinion based on experience.

> 
> 
>> Again, I would suggest you look at the 

Re: New component proposal: commons-plugins

[Gilles Sadowski]

Le jeu. 7 avr. 2022 à 14:34, Gary Gregory  a écrit :
>
> A slight tangent: a smaller simpler component idea? The log4j variable
> interpolation, the ${lookup:variable} type of logic is in many places:
> deprecated in Commons Lang, now in Commons Text, also implemented in
> Commons Configuration. We could bring in the Log4j version, now safer than
> other implementations into Commons Text or a new component and everyone
> depends on this new version.

+1

TBD in another thread (name, etc.) ?

Regards,
Gilles

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: New component proposal: commons-plugins

[Gary Gregory]

A slight tangent: a smaller simpler component idea? The log4j variable
interpolation, the ${lookup:variable} type of logic is in many places:
deprecated in Commons Lang, now in Commons Text, also implemented in
Commons Configuration. We could bring in the Log4j version, now safer than
other implementations into Commons Text or a new component and everyone
depends on this new version.

Gary

On Thu, Apr 7, 2022, 05:55 Peter Verhas  wrote:

> >
> > > The plugin system is design to convert configuration data into
> > instantiated objects.
> >
>
> For me that very much reads a dependency injection framework.
>
> --
> Peter Verhas
> pe...@verhas.com
> t: +41791542095
> skype: verhas
>

Re: New component proposal: commons-plugins

[Peter Verhas]

>
> > The plugin system is design to convert configuration data into
> instantiated objects.
>

For me that very much reads a dependency injection framework.

-- 
Peter Verhas
pe...@verhas.com
t: +41791542095
skype: verhas

Re: New component proposal: commons-plugins

[Peter Verhas]

> If I had known this was going to turn into a debate I wouldn’t have
> responded to the
> question. But it seems either my response wasn’t clear, you did not
> understand
> what I wrote, or you have some preconceived notion as to why anything
> beyond
> ServiceLoader is needed.


I believe that a debate is something that helps clear  the details and is a
positive thing. I am not asking questions l'art-pour-l'art. I still think
that some of your arguments and reasonings could be reconsidered. By doing
so, I can help raising my concern, you can think about these and
answer those and the end result may be something better than a single sided
approach.

Contemplating about the reason why I was asking some of the questions is
counter productive and drives to a bad, feeling influenced direction. Do
not.


> I would suggest that before responding to this email that
> you go look at how log4j-plugins is implemented in the master branch.
>

Sure, if you propose to create a commons-plugin library that is an extract
of the plugin handling of log4j so that log4j next releases do not need to
keep the functionality inside log4j but rather as a dependency on this new
library. On the other hand, if the goal is to create something that is
useful generally then we should look at other solutions as well.


> From this it is clear that you completely misunderstand what the plugin
> system is doing.
>

That may really be very much true.


> First, the plugin system DOES use ServiceLoader. In fact, Log4j uses
> ServiceLoader
> in at least 7 different places. For one, it uses ServiceLoader to locate
> the Log4j
> implementation similar to how SLF4J does (although Log4j implemented it
> prior to SLF4J).
>
> Consider that you have 150 or so plugins, of which maybe only 10 will get
> used. But
> you don’t know beforehand which 10. ServiceLoader would have to load all
> 150 classes.


Not. It does not need to  create instances of all of them. This is what I
explained, and this is the reason why I explained it. JPMS or not, the new
ServiceLoader used by Java 9 and later will call the static factory method
of the class if that exists. Then the plugin can decide if it wants itself
to be instantiated or not. Somewhere there should be some logic that
filters 10 from the 150. If that is in a central place implementing some
industry-best-practice configuration based algorithm, or outsources the
decision to the plugins themselves should not make a big difference in
performance.

On  the second though, instantiating 150 lightweight classes must not be a
big burden. The ServiceProvider JavaDoc documentation suggest that services
should be implemented as proxies or factories instantiating their real
working class lazily in case the instantiation is costly.


> This is indeed very slow. Yes we have benchmarked it. But the Plugin
> system IS a
> Java Service. So instead, you need to only load 1 class for every jar that
> contains plugins.
> The result is that you only need to load 4 or 5 classes. This is
> considerably faster.
>

Yes, this is the approach where you insert another layer above the service
loader that does part of the job that the service loader can also do in a
different way. Essentially you let the plugin libraries, each containing
several plugin classes to implement their own service loaders.


> In addition, services loaded by ServiceLoader are singletons.


No, they are not. Every ServiceLoader instance will create new instances,
and even a single ServiceLoader can be asked calling the `reload()` method
to evict the instances from its cache and create new instances.


> It is not unusual to require
> multiple instances of.a Plugin so this would be difficult to do with the
> ServiceLoader.


The practice in that case is that the plugins implement a `List`
interface and the instantiation or provider method returns the list of the
instances. I understand that this may not be something that one likes, but
it is not too complex and can be done.


> I’m not sure why you are explaining how ServiceLoader works. As I said, I
> am quite
> familiar with it.
>

Because you said things which are factually wrong, like services being
singletons.


> Nothing that helps. What ServiceLoader really does best is instantiating
> factories.
> You then use that factory to instantiate what you really want. Most uses
> of ServiceLoader
> in Log4j do that.
>

I get that, and I agree with that. This is how a plugin system is to be
designed.


> This is not opinion. This is a fact. Adding 50 entries to the service
> loader file is much
> more painful than adding just 1. There is no way you can argue against
> that.
>

If you say that adding 50 entries takes ten more minutes than adding one,
that is factual. "Painful" as an expression is opinion. It hurts you, may
not hurt others.

For example, when I develop a plugin I have to create a class, unit tests,
mocking the environment. documentation and so on. Adding an extra line to
the META-INF/services comp

Re: New component proposal: commons-plugins

[Matt Benson]

Generally speaking it sounds like there's at least enough interest to
create a repo and start playing. Only recently I could personally have used
a working version of something like this. No guarantees but I'd like to
help if it works out that way, for whatever that's worth.

Matt

On Sat, Apr 2, 2022, 2:33 PM Matt Sicker  wrote:

> Hi all, I’ve got a proposal for a new Commons component that I’d like to
> get feedback on. Essentially, I’d like to propose the creation of a Commons
> Plugins component inspired by the plugin system developed for Log4j 3.x
> [0]. This library would be a lightweight dependency injection and
> configuration library where developers create pluggable classes that can be
> referenced through plugin names via the configuration file (or
> configuration source in general). In contrast with more typical dependency
> injection frameworks like Spring and Guice, this library is for
> applications where pluggable implementations of things is desired.
> Developing a plugin system on top of DI frameworks is not a very
> standardized domain, and each project ends up reinventing this from scratch
> over time.
>
> Some existing material on how the Log4j plugin and configuration system
> works that I’d adapt from to form the basis for this component include:
> -
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
> <
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
> >
> -
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/plugins.adoc
> <
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/plugins.adoc
> >
>
> The general goal of this library is to make it so that applications can
> provide better configuration DSLs for their plugin components. As both a
> developer and user, I absolutely despise configuring complex applications
> with properties files, and YAML variants of properties aren’t that much
> better. If there was a common plugin library we could reuse, then perhaps
> more applications would support a better configuration system. This could
> also provide a nice place for tooling integration similar to how JUnit is
> supported by IDEs and other tools.
>
> What do you think? Should this start in the Sandbox? Is anyone interested
> in working on or using this?
>
> [0]: https://github.com/apache/logging-log4j2/tree/master/log4j-plugins <
> https://github.com/apache/logging-log4j2/tree/master/log4j-plugins>
>
> —
> Matt Sicker
>
>

Re: New component proposal: commons-plugins

[Emmanuel Bourg]

Le 04/04/2022 à 02:27, Ralph Goers a écrit :

I don’t think so.  Commons Configuration (which I was working actively on when
I started writing Log4j 2) is designed to provide access to the configuration 
data.
The plugin system is design to convert configuration data into instantiated 
objects.
I could easily see Commons Configuration being used as the input to the plugin
system, but I think it would be a mistake to tightly couple them.


+1 I agree, the goal of Commons Configuration is to parse dumb 
configuration files in various formats and with some extra convenient 
features. Building complex object structures is out of its scope.


Emmanuel Bourg

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: New component proposal: commons-plugins

[Ralph Goers]

If I had known this was going to turn into a debate I wouldn’t have responded 
to the 
question. But it seems either my response wasn’t clear, you did not understand 
what I wrote, or you have some preconceived notion as to why anything beyond 
ServiceLoader is needed. I would suggest that before responding to this email 
that 
you go look at how log4j-plugins is implemented in the master branch.

> On Apr 4, 2022, at 8:13 PM, Peter Verhas  wrote:
> 
> Thank Ralph for the answer related ServiceLoader. I understand your
> concerns. A new commons plugin module is in place if we identify the needs
> unmet with modern Java. However, the shortcomings you listed are mainly
> related to Java 8 and older versions. Thus I would appreciate it if you
> could clarify the followings:
> 
>> Loading all of Log4j’s plugins via ServiceLoader would be very slow
> 
> What kind of measurements and benchmarks show that? I have never had any
> issue with speed using ServiceLoader, as it is a standard way to load
> classes and get their objects which may not be there during the compile
> time. For example, my Jamal macro language loads all the macros and
> debuggers as services.
> SLF4J relies on ServiceLoader.

From this it is clear that you completely misunderstand what the plugin system 
is doing.
First, the plugin system DOES use ServiceLoader. In fact, Log4j uses 
ServiceLoader 
in at least 7 different places. For one, it uses ServiceLoader to locate the 
Log4j 
implementation similar to how SLF4J does (although Log4j implemented it prior 
to SLF4J).

Consider that you have 150 or so plugins, of which maybe only 10 will get used. 
But 
you don’t know beforehand which 10. ServiceLoader would have to load all 150 
classes. 
This is indeed very slow. Yes we have benchmarked it. But the Plugin system IS 
a 
Java Service. So instead, you need to only load 1 class for every jar that 
contains plugins. 
The result is that you only need to load 4 or 5 classes. This is considerably 
faster.

In addition, services loaded by ServiceLoader are singletons. It is not unusual 
to require 
multiple instances of.a Plugin so this would be difficult to do with the 
ServiceLoader. 
But since the service loaded for the plugin system only contains the plugin 
definition 
this is not an issue.

> 
>> and would require every
>> plugin be listed in the service configuration file.
> 
> I do not understand what configuration file you are talking about.

META-INF/services/service.fully.qualified.class.name. Every plugin would need 
an 
entry in this file if every Plugin is a raw service. In addition, for JPMS they 
would need 
an additional entry in module-info.java. The Log4j annotation processor 
generates the 
META-INF/services file for you. 

> 
> You implement an interface when you develop a service to be loaded by the
> JDK ServiceLoader. In addition to that, you either list the full name of
> the class in a text file inside your library, and the name of the file is
> the fully qualified name of the interface (this is the old style,
> pre-Java-9, and Java module systems), or you declare your service in the
> module-info.java file.

I’m not sure why you are explaining how ServiceLoader works. As I said, I am 
quite 
familiar with it.


> 
>> This has several problems:
>> 1. A service must have a constructor that accepts no parameters.
> 
> You can also have a provider method in the service implementing class.
> There is some flexibility introduced in Java 9.

Nothing that helps. What ServiceLoader really does best is instantiating 
factories. 
You then use that factory to instantiate what you really want. Most uses of 
ServiceLoader 
in Log4j do that.

> 
>> 2. Manually configuring every plugin in the META-INF/services file would
> be painful.
> 
> This is opinionated. There is no point debating opinions.

This is not opinion. This is a fact. Adding 50 entries to the service loader 
file is much 
more painful than adding just 1. There is no way you can argue against that.

> Editing a text
> file and inserting the name of the fully qualified name of the class never
> felt painful significantly since editors like IntelliJ help you copy/paste
> the fully qualified names. If you find it cumbersome to maintain these
> files manually, you can also use annotation processors to generate them. I
> have never used it, but Spring @AutoService does something like that.
> 
> In addition to that, you do not need to have the old-style
> META-INF/services files if you use Java 9 or newer.

That is not correct. That may be true when using the module path but the 
javadoc 
for ServiceLoader is quite clear that the file is still required for jars on 
the classpath. 
Since the vast majority of Java users do not use JPMS this isn’t really 
helpful. I 
should also note that in fact the opposite is true. If your jar is an automatic 
module 
then JPMS will create the provides statement for you based on the presence of 
the 
META-INF/services file.

> 
>> 3. Every plugi

Re: New component proposal: commons-plugins

[Peter Verhas]

Thank Ralph for the answer related ServiceLoader. I understand your
concerns. A new commons plugin module is in place if we identify the needs
unmet with modern Java. However, the shortcomings you listed are mainly
related to Java 8 and older versions. Thus I would appreciate it if you
could clarify the followings:

>Loading all of Log4j’s plugins via ServiceLoader would be very slow

What kind of measurements and benchmarks show that? I have never had any
issue with speed using ServiceLoader, as it is a standard way to load
classes and get their objects which may not be there during the compile
time. For example, my Jamal macro language loads all the macros and
debuggers as services.
SLF4J relies on ServiceLoader.

> and would require every
>plugin be listed in the service configuration file.

I do not understand what configuration file you are talking about.

You implement an interface when you develop a service to be loaded by the
JDK ServiceLoader. In addition to that, you either list the full name of
the class in a text file inside your library, and the name of the file is
the fully qualified name of the interface (this is the old style,
pre-Java-9, and Java module systems), or you declare your service in the
module-info.java file.

>This has several problems:
>1. A service must have a constructor that accepts no parameters.

You can also have a provider method in the service implementing class.
There is some flexibility introduced in Java 9.

>2. Manually configuring every plugin in the META-INF/services file would
be painful.

This is opinionated. There is no point debating opinions. Editing a text
file and inserting the name of the fully qualified name of the class never
felt painful significantly since editors like IntelliJ help you copy/paste
the fully qualified names. If you find it cumbersome to maintain these
files manually, you can also use annotation processors to generate them. I
have never used it, but Spring @AutoService does something like that.

In addition to that, you do not need to have the old-style
META-INF/services files if you use Java 9 or newer.

>3. Every plugin might have its own unique configuration. ServiceLoader
provides no mechanism
>for configuration.

This is true, but other modules, like commons configuration, do. This is
also discussed by others, so let's not focus on this one.

I know that there are areas for plugins, which are not supported by the JDK
service loader mechanism, but I am skeptical that these can be implemented
in a library in a general way. One example is the version negotiation,
which is very much application-specific. Another one is suitability
negotiation, when your application may face several implementations of the
service and there is room only for one: the best fitting for the purpose.
These are very much application-specific and there are no widely accepted
industry best practice standards.





On Mon, 4 Apr 2022 at 04:51, Ralph Goers  wrote:

> It has been too long since I looked at Commons Configuration for me to
> answer that.
>
> Ralph
>
> > On Apr 3, 2022, at 7:25 PM, Gary Gregory  wrote:
> >
> > So in a Commons centric fantasy, Log4j Nodes could be Commons
> Configuration
> > objects?
> >
> > G
> >
> > On Sun, Apr 3, 2022, 22:18 Ralph Goers 
> wrote:
> >
> >> If you look at Log4j, we parse the configuration and convert it to a
> Node
> >> hierarchy.
> >> Variable interpolation occurs as the Nodes are constructed.
> >>
> >> But again, Nothing says a Commons Plugins implementation has to work the
> >> same way.
> >>
> >> Ralph
> >>
> >>> On Apr 3, 2022, at 7:12 PM, Gary Gregory 
> wrote:
> >>>
> >>> Nice thread :-)
> >>>
> >>> Where does the parsing of configuration files mix in such a stack?
> >>>
> >>> Where does variable interpolation come into play?
> >>>
> >>> Gary
> >>>
> >>> On Sun, Apr 3, 2022, 20:50 Ralph Goers 
> >> wrote:
> >>>
>  Matt J,
> 
>  I fully expect that if commons-plugins came into fruition it would
> bear
> >> a
>  resemblance
>  to the Log4j plugin system but there would be differences. For
> example,
>  while Log4j
>  integrates with Spring we don’t currently support having the logging
>  configuration in
>  application.yml. I also suspect it would an almost impossible
> >> abstraction
>  to have the
>  DI system be pluggable, but I could be wrong about that.
> 
>  Of the bulleted items you listed Log4j’s plugin system handles all of
> >> them
>  except that
>  it doesn’t deal with dependencies. Instead, if your plugin is running
> in
>  an OSGi
>  environment it would be expected that the dependent modules would be
>  available as
>  described by the manifest for the module the plugin is included in.
> 
>  The goals for the log4j-plugins are really pretty simple.
>  * Allow applications to provide flexibility by exposing sets of
> >> pluggable
>  component types.
>  * Allow users to provide their own implementations of

Re: New component proposal: commons-plugins

[Matt Sicker]

Right, Provider is a ServiceLoader class, but the same concept can
apply within plugin categories regarding version compatibility or
capability negotiation. Most plugins (besides the "Core" category in
Log4j) are for creating components that can be referenced by name, and
some plugins have their own syntax for usage (like the pattern
conversion keys using a printf-style syntax). Plugins that take
configuration data to instantiate would require a configuration tree.

I just looked at commons-configuration briefly, and while it sounds
fairly useful in theory, I'm not so sure about some of the APIs in
use. Maybe it's adaptable, though I'd have to look more closely. I see
that its expression language uses commons-jexl, for example, while we
have a fairly minimal string substitution engine (via the StrLookup
plugins) which can be used here without depending on JavaEE APIs.
Similarly, the bean-related config stuff uses commons-beanutils,
something we didn't really need to use here (though some of that is
thanks to lambdas making Java less verbose these days). For JSON
configuration, we can support that without a dependency by including a
minimal JSON parser (like in Log4j), though different formats could
easily use different libraries for the implementation. The commons-vfs
integration looks interesting (we have something supported for
JBoss-style VFS URLs at least).

On Mon, Apr 4, 2022 at 10:51 AM Ralph Goers  wrote:
>
> Matt,
>
> The Logging Provider is not a plugin as it is required way before plugins are 
> available.
>
> That said, the plugin system certainly could have a way of performing version 
> checking.
> But keep in mind, the versions would be defined by the “owner” of each plugin 
> category
> and not tied to the plugin system.
>
> Ralph
>
> > On Apr 4, 2022, at 8:39 AM, Matt Sicker  wrote:
> >
> > I used to work on the Jenkins project for a few years, so yes, I'm
> > fairly familiar with those difficulties. I've also used OSGi in the
> > past, and I like their approach to semantic versioning and dynamic
> > "plugin" (well, bundle/component/etc.) lifecycles. I currently work on
> > the Spinnaker project where we have a less sophisticated plugin system
> > based on pf4j, and we've already found that to be incredibly
> > burdensome to work with so far.
> >
> > For version compatibility, I'd like to maintain the usual Commons
> > standards of strict compatibility along semantic versioning. The
> > existing Log4j plugin system already has some basic version check
> > support (at least for the logging provider), and a more generic plugin
> > system would definitely need the more sophisticated version check
> > support. I'd also like a way for plugins to declare data compatibility
> > related metadata for easier upgrades and such, but those are merely
> > details.
> >
> > On Mon, Apr 4, 2022 at 10:33 AM Xeno Amess  wrote:
> >>
> >> If you really want to start this, it would be useful.
> >> However it would be complicated, very very complicated.
> >> For example, version management.
> >> The plugins have their versions.
> >> And the base program has its version.
> >> And the plugin registration mechanism has its own version.
> >> How to make it work would be a big challenge.
> >> If we wanna make a useful plugin management system, maybe a good way is
> >> first to learn about some existing usage of these things(although developed
> >> in different groups/repos), like eclipse, jetbrains idea, even OSGI...
> >>
> >> Gary Gregory  于2022年4月4日周一 21:11写道：
> >>
> >>> I am trying to figure out the input and output of a Commons Plugins
> >>> that Log4j would use.
> >>>
> >>> Gary
> >>>
> >>> On Sun, Apr 3, 2022 at 10:51 PM Ralph Goers 
> >>> wrote:
> 
>  It has been too long since I looked at Commons Configuration for me to
> >>> answer that.
> 
>  Ralph
> 
> > On Apr 3, 2022, at 7:25 PM, Gary Gregory 
> >>> wrote:
> >
> > So in a Commons centric fantasy, Log4j Nodes could be Commons
> >>> Configuration
> > objects?
> >
> > G
> >
> > On Sun, Apr 3, 2022, 22:18 Ralph Goers 
> >>> wrote:
> >
> >> If you look at Log4j, we parse the configuration and convert it to a
> >>> Node
> >> hierarchy.
> >> Variable interpolation occurs as the Nodes are constructed.
> >>
> >> But again, Nothing says a Commons Plugins implementation has to work
> >>> the
> >> same way.
> >>
> >> Ralph
> >>
> >>> On Apr 3, 2022, at 7:12 PM, Gary Gregory 
> >>> wrote:
> >>>
> >>> Nice thread :-)
> >>>
> >>> Where does the parsing of configuration files mix in such a stack?
> >>>
> >>> Where does variable interpolation come into play?
> >>>
> >>> Gary
> >>>
> >>> On Sun, Apr 3, 2022, 20:50 Ralph Goers 
> >> wrote:
> >>>
>  Matt J,
> 
>  I fully expect that if commons-plugins came into fruition it would
> >>> bear
> >> a
>  resemblance
>  to the Log4j plugin syste

Re: New component proposal: commons-plugins

[Ralph Goers]

Matt,

The Logging Provider is not a plugin as it is required way before plugins are 
available.

That said, the plugin system certainly could have a way of performing version 
checking. 
But keep in mind, the versions would be defined by the “owner” of each plugin 
category 
and not tied to the plugin system.

Ralph

> On Apr 4, 2022, at 8:39 AM, Matt Sicker  wrote:
> 
> I used to work on the Jenkins project for a few years, so yes, I'm
> fairly familiar with those difficulties. I've also used OSGi in the
> past, and I like their approach to semantic versioning and dynamic
> "plugin" (well, bundle/component/etc.) lifecycles. I currently work on
> the Spinnaker project where we have a less sophisticated plugin system
> based on pf4j, and we've already found that to be incredibly
> burdensome to work with so far.
> 
> For version compatibility, I'd like to maintain the usual Commons
> standards of strict compatibility along semantic versioning. The
> existing Log4j plugin system already has some basic version check
> support (at least for the logging provider), and a more generic plugin
> system would definitely need the more sophisticated version check
> support. I'd also like a way for plugins to declare data compatibility
> related metadata for easier upgrades and such, but those are merely
> details.
> 
> On Mon, Apr 4, 2022 at 10:33 AM Xeno Amess  wrote:
>> 
>> If you really want to start this, it would be useful.
>> However it would be complicated, very very complicated.
>> For example, version management.
>> The plugins have their versions.
>> And the base program has its version.
>> And the plugin registration mechanism has its own version.
>> How to make it work would be a big challenge.
>> If we wanna make a useful plugin management system, maybe a good way is
>> first to learn about some existing usage of these things(although developed
>> in different groups/repos), like eclipse, jetbrains idea, even OSGI...
>> 
>> Gary Gregory  于2022年4月4日周一 21:11写道：
>> 
>>> I am trying to figure out the input and output of a Commons Plugins
>>> that Log4j would use.
>>> 
>>> Gary
>>> 
>>> On Sun, Apr 3, 2022 at 10:51 PM Ralph Goers 
>>> wrote:
 
 It has been too long since I looked at Commons Configuration for me to
>>> answer that.
 
 Ralph
 
> On Apr 3, 2022, at 7:25 PM, Gary Gregory 
>>> wrote:
> 
> So in a Commons centric fantasy, Log4j Nodes could be Commons
>>> Configuration
> objects?
> 
> G
> 
> On Sun, Apr 3, 2022, 22:18 Ralph Goers 
>>> wrote:
> 
>> If you look at Log4j, we parse the configuration and convert it to a
>>> Node
>> hierarchy.
>> Variable interpolation occurs as the Nodes are constructed.
>> 
>> But again, Nothing says a Commons Plugins implementation has to work
>>> the
>> same way.
>> 
>> Ralph
>> 
>>> On Apr 3, 2022, at 7:12 PM, Gary Gregory 
>>> wrote:
>>> 
>>> Nice thread :-)
>>> 
>>> Where does the parsing of configuration files mix in such a stack?
>>> 
>>> Where does variable interpolation come into play?
>>> 
>>> Gary
>>> 
>>> On Sun, Apr 3, 2022, 20:50 Ralph Goers 
>> wrote:
>>> 
 Matt J,
 
 I fully expect that if commons-plugins came into fruition it would
>>> bear
>> a
 resemblance
 to the Log4j plugin system but there would be differences. For
>>> example,
 while Log4j
 integrates with Spring we don’t currently support having the logging
 configuration in
 application.yml. I also suspect it would an almost impossible
>> abstraction
 to have the
 DI system be pluggable, but I could be wrong about that.
 
 Of the bulleted items you listed Log4j’s plugin system handles all
>>> of
>> them
 except that
 it doesn’t deal with dependencies. Instead, if your plugin is
>>> running in
 an OSGi
 environment it would be expected that the dependent modules would be
 available as
 described by the manifest for the module the plugin is included in.
 
 The goals for the log4j-plugins are really pretty simple.
 * Allow applications to provide flexibility by exposing sets of
>> pluggable
 component types.
 * Allow users to provide their own implementations of the pluggable
 components that
 are on an equal footing with the “standard” components.
 
 Just by way of example, Apache Flume allows pluggable components
>>> such as
 Sources,
 Sinks and Channels. Writing a new Sink is fairly straightforward.
>>> But it
 will not be on
 an equal footing with the Sinks provided by Flume. This is because
>>> there
 is code in
 Flume that provides an alias (such as “avro” for the AvroSink) so
>>> that
>> the
 plugin class
 can be named without having to specify th

Re: New component proposal: commons-plugins

[Ralph Goers]

Matt,

I don’t see commons-plugins needing a configuration. Remember, there are 
several components 
in Log4j that don’t use a configuration file. Namely, Lookups, 
PatternConverters, KeyProviders, 
and Watchers, I view the plugin system as an API that a configuration system 
would use to 
instantiate the objects referenced in the configuration.

Ralph

> On Apr 4, 2022, at 8:30 AM, Matt Sicker  wrote:
> 
> From the Log4j side of things, I didn't exactly overhaul the
> configuration parsing or representation of it, so it would depend on
> how different that looks in this proposed component. Ideally, though,
> a commons-plugins component would replace log4j-plugins and
> log4j-plugin-processor along with some of the plugin types we have
> like type converters and config variable substitution.
> 
> From a high level point of view, the inputs would be a configuration
> source (e.g., a reloadable file or HTTPS link or whatever), and the
> output would be a graph of instantiated plugins from the
> configuration.
> 
> On Mon, Apr 4, 2022 at 8:12 AM Gary Gregory  wrote:
>> 
>> I am trying to figure out the input and output of a Commons Plugins
>> that Log4j would use.
>> 
>> Gary
>> 
>> On Sun, Apr 3, 2022 at 10:51 PM Ralph Goers  
>> wrote:
>>> 
>>> It has been too long since I looked at Commons Configuration for me to 
>>> answer that.
>>> 
>>> Ralph
>>> 
 On Apr 3, 2022, at 7:25 PM, Gary Gregory  wrote:
 
 So in a Commons centric fantasy, Log4j Nodes could be Commons Configuration
 objects?
 
 G
 
 On Sun, Apr 3, 2022, 22:18 Ralph Goers  wrote:
 
> If you look at Log4j, we parse the configuration and convert it to a Node
> hierarchy.
> Variable interpolation occurs as the Nodes are constructed.
> 
> But again, Nothing says a Commons Plugins implementation has to work the
> same way.
> 
> Ralph
> 
>> On Apr 3, 2022, at 7:12 PM, Gary Gregory  wrote:
>> 
>> Nice thread :-)
>> 
>> Where does the parsing of configuration files mix in such a stack?
>> 
>> Where does variable interpolation come into play?
>> 
>> Gary
>> 
>> On Sun, Apr 3, 2022, 20:50 Ralph Goers 
> wrote:
>> 
>>> Matt J,
>>> 
>>> I fully expect that if commons-plugins came into fruition it would bear
> a
>>> resemblance
>>> to the Log4j plugin system but there would be differences. For example,
>>> while Log4j
>>> integrates with Spring we don’t currently support having the logging
>>> configuration in
>>> application.yml. I also suspect it would an almost impossible
> abstraction
>>> to have the
>>> DI system be pluggable, but I could be wrong about that.
>>> 
>>> Of the bulleted items you listed Log4j’s plugin system handles all of
> them
>>> except that
>>> it doesn’t deal with dependencies. Instead, if your plugin is running in
>>> an OSGi
>>> environment it would be expected that the dependent modules would be
>>> available as
>>> described by the manifest for the module the plugin is included in.
>>> 
>>> The goals for the log4j-plugins are really pretty simple.
>>> * Allow applications to provide flexibility by exposing sets of
> pluggable
>>> component types.
>>> * Allow users to provide their own implementations of the pluggable
>>> components that
>>> are on an equal footing with the “standard” components.
>>> 
>>> Just by way of example, Apache Flume allows pluggable components such as
>>> Sources,
>>> Sinks and Channels. Writing a new Sink is fairly straightforward. But it
>>> will not be on
>>> an equal footing with the Sinks provided by Flume. This is because there
>>> is code in
>>> Flume that provides an alias (such as “avro” for the AvroSink) so that
> the
>>> plugin class
>>> can be named without having to specify the fully qualified class name.
>>> Custom
>>> components must be configured using the FQCN.  The Log4j plugin system
>>> solves
>>> this by requiring every plugin to have a “name” which is then used as
> the
>>> mechanism
>>> to locate it from the configuration.
>>> 
>>> Non-goals would be
>>> * It is meant to load plugins, not be a full DI system for every
> component.
>>> * It leverages standard class loading methodologies, not a new one. i.e.
>>> it works with
>>> the standard Java class path, module path, or OSGi.
>>> 
>>> Ralph
>>> 
>>> 
 On Apr 3, 2022, at 8:49 AM, Matt Juntunen 
>>> wrote:
 
 Hi Matt,
 
 This is quite timely since I've spent the past week researching
 frameworks to modularize a monolithic application at my day job into
 separate components/plugins. Everything I've looked at so far is
 larger and more complicated than I need (e.g. OSGi, Spring, etc) so I
 was seriously co

Re: New component proposal: commons-plugins

[Matt Sicker]

I used to work on the Jenkins project for a few years, so yes, I'm
fairly familiar with those difficulties. I've also used OSGi in the
past, and I like their approach to semantic versioning and dynamic
"plugin" (well, bundle/component/etc.) lifecycles. I currently work on
the Spinnaker project where we have a less sophisticated plugin system
based on pf4j, and we've already found that to be incredibly
burdensome to work with so far.

For version compatibility, I'd like to maintain the usual Commons
standards of strict compatibility along semantic versioning. The
existing Log4j plugin system already has some basic version check
support (at least for the logging provider), and a more generic plugin
system would definitely need the more sophisticated version check
support. I'd also like a way for plugins to declare data compatibility
related metadata for easier upgrades and such, but those are merely
details.

On Mon, Apr 4, 2022 at 10:33 AM Xeno Amess  wrote:
>
> If you really want to start this, it would be useful.
> However it would be complicated, very very complicated.
> For example, version management.
> The plugins have their versions.
> And the base program has its version.
> And the plugin registration mechanism has its own version.
> How to make it work would be a big challenge.
> If we wanna make a useful plugin management system, maybe a good way is
> first to learn about some existing usage of these things(although developed
> in different groups/repos), like eclipse, jetbrains idea, even OSGI...
>
> Gary Gregory  于2022年4月4日周一 21:11写道：
>
> > I am trying to figure out the input and output of a Commons Plugins
> > that Log4j would use.
> >
> > Gary
> >
> > On Sun, Apr 3, 2022 at 10:51 PM Ralph Goers 
> > wrote:
> > >
> > > It has been too long since I looked at Commons Configuration for me to
> > answer that.
> > >
> > > Ralph
> > >
> > > > On Apr 3, 2022, at 7:25 PM, Gary Gregory 
> > wrote:
> > > >
> > > > So in a Commons centric fantasy, Log4j Nodes could be Commons
> > Configuration
> > > > objects?
> > > >
> > > > G
> > > >
> > > > On Sun, Apr 3, 2022, 22:18 Ralph Goers 
> > wrote:
> > > >
> > > >> If you look at Log4j, we parse the configuration and convert it to a
> > Node
> > > >> hierarchy.
> > > >> Variable interpolation occurs as the Nodes are constructed.
> > > >>
> > > >> But again, Nothing says a Commons Plugins implementation has to work
> > the
> > > >> same way.
> > > >>
> > > >> Ralph
> > > >>
> > > >>> On Apr 3, 2022, at 7:12 PM, Gary Gregory 
> > wrote:
> > > >>>
> > > >>> Nice thread :-)
> > > >>>
> > > >>> Where does the parsing of configuration files mix in such a stack?
> > > >>>
> > > >>> Where does variable interpolation come into play?
> > > >>>
> > > >>> Gary
> > > >>>
> > > >>> On Sun, Apr 3, 2022, 20:50 Ralph Goers 
> > > >> wrote:
> > > >>>
> > >  Matt J,
> > > 
> > >  I fully expect that if commons-plugins came into fruition it would
> > bear
> > > >> a
> > >  resemblance
> > >  to the Log4j plugin system but there would be differences. For
> > example,
> > >  while Log4j
> > >  integrates with Spring we don’t currently support having the logging
> > >  configuration in
> > >  application.yml. I also suspect it would an almost impossible
> > > >> abstraction
> > >  to have the
> > >  DI system be pluggable, but I could be wrong about that.
> > > 
> > >  Of the bulleted items you listed Log4j’s plugin system handles all
> > of
> > > >> them
> > >  except that
> > >  it doesn’t deal with dependencies. Instead, if your plugin is
> > running in
> > >  an OSGi
> > >  environment it would be expected that the dependent modules would be
> > >  available as
> > >  described by the manifest for the module the plugin is included in.
> > > 
> > >  The goals for the log4j-plugins are really pretty simple.
> > >  * Allow applications to provide flexibility by exposing sets of
> > > >> pluggable
> > >  component types.
> > >  * Allow users to provide their own implementations of the pluggable
> > >  components that
> > >  are on an equal footing with the “standard” components.
> > > 
> > >  Just by way of example, Apache Flume allows pluggable components
> > such as
> > >  Sources,
> > >  Sinks and Channels. Writing a new Sink is fairly straightforward.
> > But it
> > >  will not be on
> > >  an equal footing with the Sinks provided by Flume. This is because
> > there
> > >  is code in
> > >  Flume that provides an alias (such as “avro” for the AvroSink) so
> > that
> > > >> the
> > >  plugin class
> > >  can be named without having to specify the fully qualified class
> > name.
> > >  Custom
> > >  components must be configured using the FQCN.  The Log4j plugin
> > system
> > >  solves
> > >  this by requiring every plugin to have a “name” which is then used
> > as
> > > >> the
> > >  mechanism
> > >  to locate it 

Re: New component proposal: commons-plugins

[Xeno Amess]

No, I think it is a little different, but commons-configuration can be its
dependency.

Gary Gregory  于2022年4月3日周日 18:24写道：

> Should this be in Commons Configuration?
>
> Gary
>
> On Sat, Apr 2, 2022, 15:33 Matt Sicker  wrote:
>
> > Hi all, I’ve got a proposal for a new Commons component that I’d like to
> > get feedback on. Essentially, I’d like to propose the creation of a
> Commons
> > Plugins component inspired by the plugin system developed for Log4j 3.x
> > [0]. This library would be a lightweight dependency injection and
> > configuration library where developers create pluggable classes that can
> be
> > referenced through plugin names via the configuration file (or
> > configuration source in general). In contrast with more typical
> dependency
> > injection frameworks like Spring and Guice, this library is for
> > applications where pluggable implementations of things is desired.
> > Developing a plugin system on top of DI frameworks is not a very
> > standardized domain, and each project ends up reinventing this from
> scratch
> > over time.
> >
> > Some existing material on how the Log4j plugin and configuration system
> > works that I’d adapt from to form the basis for this component include:
> > -
> >
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
> > <
> >
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
> > >
> > -
> >
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/plugins.adoc
> > <
> >
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/plugins.adoc
> > >
> >
> > The general goal of this library is to make it so that applications can
> > provide better configuration DSLs for their plugin components. As both a
> > developer and user, I absolutely despise configuring complex applications
> > with properties files, and YAML variants of properties aren’t that much
> > better. If there was a common plugin library we could reuse, then perhaps
> > more applications would support a better configuration system. This could
> > also provide a nice place for tooling integration similar to how JUnit is
> > supported by IDEs and other tools.
> >
> > What do you think? Should this start in the Sandbox? Is anyone interested
> > in working on or using this?
> >
> > [0]: https://github.com/apache/logging-log4j2/tree/master/log4j-plugins
> <
> > https://github.com/apache/logging-log4j2/tree/master/log4j-plugins>
> >
> > —
> > Matt Sicker
> >
> >
>

Re: New component proposal: commons-plugins

[Xeno Amess]

If you really want to start this, it would be useful.
However it would be complicated, very very complicated.
For example, version management.
The plugins have their versions.
And the base program has its version.
And the plugin registration mechanism has its own version.
How to make it work would be a big challenge.
If we wanna make a useful plugin management system, maybe a good way is
first to learn about some existing usage of these things(although developed
in different groups/repos), like eclipse, jetbrains idea, even OSGI...

Gary Gregory  于2022年4月4日周一 21:11写道：

> I am trying to figure out the input and output of a Commons Plugins
> that Log4j would use.
>
> Gary
>
> On Sun, Apr 3, 2022 at 10:51 PM Ralph Goers 
> wrote:
> >
> > It has been too long since I looked at Commons Configuration for me to
> answer that.
> >
> > Ralph
> >
> > > On Apr 3, 2022, at 7:25 PM, Gary Gregory 
> wrote:
> > >
> > > So in a Commons centric fantasy, Log4j Nodes could be Commons
> Configuration
> > > objects?
> > >
> > > G
> > >
> > > On Sun, Apr 3, 2022, 22:18 Ralph Goers 
> wrote:
> > >
> > >> If you look at Log4j, we parse the configuration and convert it to a
> Node
> > >> hierarchy.
> > >> Variable interpolation occurs as the Nodes are constructed.
> > >>
> > >> But again, Nothing says a Commons Plugins implementation has to work
> the
> > >> same way.
> > >>
> > >> Ralph
> > >>
> > >>> On Apr 3, 2022, at 7:12 PM, Gary Gregory 
> wrote:
> > >>>
> > >>> Nice thread :-)
> > >>>
> > >>> Where does the parsing of configuration files mix in such a stack?
> > >>>
> > >>> Where does variable interpolation come into play?
> > >>>
> > >>> Gary
> > >>>
> > >>> On Sun, Apr 3, 2022, 20:50 Ralph Goers 
> > >> wrote:
> > >>>
> >  Matt J,
> > 
> >  I fully expect that if commons-plugins came into fruition it would
> bear
> > >> a
> >  resemblance
> >  to the Log4j plugin system but there would be differences. For
> example,
> >  while Log4j
> >  integrates with Spring we don’t currently support having the logging
> >  configuration in
> >  application.yml. I also suspect it would an almost impossible
> > >> abstraction
> >  to have the
> >  DI system be pluggable, but I could be wrong about that.
> > 
> >  Of the bulleted items you listed Log4j’s plugin system handles all
> of
> > >> them
> >  except that
> >  it doesn’t deal with dependencies. Instead, if your plugin is
> running in
> >  an OSGi
> >  environment it would be expected that the dependent modules would be
> >  available as
> >  described by the manifest for the module the plugin is included in.
> > 
> >  The goals for the log4j-plugins are really pretty simple.
> >  * Allow applications to provide flexibility by exposing sets of
> > >> pluggable
> >  component types.
> >  * Allow users to provide their own implementations of the pluggable
> >  components that
> >  are on an equal footing with the “standard” components.
> > 
> >  Just by way of example, Apache Flume allows pluggable components
> such as
> >  Sources,
> >  Sinks and Channels. Writing a new Sink is fairly straightforward.
> But it
> >  will not be on
> >  an equal footing with the Sinks provided by Flume. This is because
> there
> >  is code in
> >  Flume that provides an alias (such as “avro” for the AvroSink) so
> that
> > >> the
> >  plugin class
> >  can be named without having to specify the fully qualified class
> name.
> >  Custom
> >  components must be configured using the FQCN.  The Log4j plugin
> system
> >  solves
> >  this by requiring every plugin to have a “name” which is then used
> as
> > >> the
> >  mechanism
> >  to locate it from the configuration.
> > 
> >  Non-goals would be
> >  * It is meant to load plugins, not be a full DI system for every
> > >> component.
> >  * It leverages standard class loading methodologies, not a new one.
> i.e.
> >  it works with
> >  the standard Java class path, module path, or OSGi.
> > 
> >  Ralph
> > 
> > 
> > > On Apr 3, 2022, at 8:49 AM, Matt Juntunen <
> matt.a.juntu...@gmail.com>
> >  wrote:
> > >
> > > Hi Matt,
> > >
> > > This is quite timely since I've spent the past week researching
> > > frameworks to modularize a monolithic application at my day job
> into
> > > separate components/plugins. Everything I've looked at so far is
> > > larger and more complicated than I need (e.g. OSGi, Spring, etc)
> so I
> > > was seriously considering writing my own, perhaps based on select
> > > components from the Plexus project [1]. I would be interested in
> this
> > > project if it could do the following:
> > > - provide a standardized plugin packaging format;
> > > - provide standardized configuration mechanisms;
> > > - handle plugin discovery and enumeration;
> > > - handle servic

Re: New component proposal: commons-plugins

[Matt Sicker]

>From the Log4j side of things, I didn't exactly overhaul the
configuration parsing or representation of it, so it would depend on
how different that looks in this proposed component. Ideally, though,
a commons-plugins component would replace log4j-plugins and
log4j-plugin-processor along with some of the plugin types we have
like type converters and config variable substitution.

>From a high level point of view, the inputs would be a configuration
source (e.g., a reloadable file or HTTPS link or whatever), and the
output would be a graph of instantiated plugins from the
configuration.

On Mon, Apr 4, 2022 at 8:12 AM Gary Gregory  wrote:
>
> I am trying to figure out the input and output of a Commons Plugins
> that Log4j would use.
>
> Gary
>
> On Sun, Apr 3, 2022 at 10:51 PM Ralph Goers  
> wrote:
> >
> > It has been too long since I looked at Commons Configuration for me to 
> > answer that.
> >
> > Ralph
> >
> > > On Apr 3, 2022, at 7:25 PM, Gary Gregory  wrote:
> > >
> > > So in a Commons centric fantasy, Log4j Nodes could be Commons 
> > > Configuration
> > > objects?
> > >
> > > G
> > >
> > > On Sun, Apr 3, 2022, 22:18 Ralph Goers  wrote:
> > >
> > >> If you look at Log4j, we parse the configuration and convert it to a Node
> > >> hierarchy.
> > >> Variable interpolation occurs as the Nodes are constructed.
> > >>
> > >> But again, Nothing says a Commons Plugins implementation has to work the
> > >> same way.
> > >>
> > >> Ralph
> > >>
> > >>> On Apr 3, 2022, at 7:12 PM, Gary Gregory  wrote:
> > >>>
> > >>> Nice thread :-)
> > >>>
> > >>> Where does the parsing of configuration files mix in such a stack?
> > >>>
> > >>> Where does variable interpolation come into play?
> > >>>
> > >>> Gary
> > >>>
> > >>> On Sun, Apr 3, 2022, 20:50 Ralph Goers 
> > >> wrote:
> > >>>
> >  Matt J,
> > 
> >  I fully expect that if commons-plugins came into fruition it would bear
> > >> a
> >  resemblance
> >  to the Log4j plugin system but there would be differences. For example,
> >  while Log4j
> >  integrates with Spring we don’t currently support having the logging
> >  configuration in
> >  application.yml. I also suspect it would an almost impossible
> > >> abstraction
> >  to have the
> >  DI system be pluggable, but I could be wrong about that.
> > 
> >  Of the bulleted items you listed Log4j’s plugin system handles all of
> > >> them
> >  except that
> >  it doesn’t deal with dependencies. Instead, if your plugin is running 
> >  in
> >  an OSGi
> >  environment it would be expected that the dependent modules would be
> >  available as
> >  described by the manifest for the module the plugin is included in.
> > 
> >  The goals for the log4j-plugins are really pretty simple.
> >  * Allow applications to provide flexibility by exposing sets of
> > >> pluggable
> >  component types.
> >  * Allow users to provide their own implementations of the pluggable
> >  components that
> >  are on an equal footing with the “standard” components.
> > 
> >  Just by way of example, Apache Flume allows pluggable components such 
> >  as
> >  Sources,
> >  Sinks and Channels. Writing a new Sink is fairly straightforward. But 
> >  it
> >  will not be on
> >  an equal footing with the Sinks provided by Flume. This is because 
> >  there
> >  is code in
> >  Flume that provides an alias (such as “avro” for the AvroSink) so that
> > >> the
> >  plugin class
> >  can be named without having to specify the fully qualified class name.
> >  Custom
> >  components must be configured using the FQCN.  The Log4j plugin system
> >  solves
> >  this by requiring every plugin to have a “name” which is then used as
> > >> the
> >  mechanism
> >  to locate it from the configuration.
> > 
> >  Non-goals would be
> >  * It is meant to load plugins, not be a full DI system for every
> > >> component.
> >  * It leverages standard class loading methodologies, not a new one. 
> >  i.e.
> >  it works with
> >  the standard Java class path, module path, or OSGi.
> > 
> >  Ralph
> > 
> > 
> > > On Apr 3, 2022, at 8:49 AM, Matt Juntunen 
> >  wrote:
> > >
> > > Hi Matt,
> > >
> > > This is quite timely since I've spent the past week researching
> > > frameworks to modularize a monolithic application at my day job into
> > > separate components/plugins. Everything I've looked at so far is
> > > larger and more complicated than I need (e.g. OSGi, Spring, etc) so I
> > > was seriously considering writing my own, perhaps based on select
> > > components from the Plexus project [1]. I would be interested in this
> > > project if it could do the following:
> > > - provide a standardized plugin packaging format;
> > > - provide standardized configuration mechanisms;
> > > - hand

Re: New component proposal: commons-plugins

[Gary Gregory]

I am trying to figure out the input and output of a Commons Plugins
that Log4j would use.

Gary

On Sun, Apr 3, 2022 at 10:51 PM Ralph Goers  wrote:
>
> It has been too long since I looked at Commons Configuration for me to answer 
> that.
>
> Ralph
>
> > On Apr 3, 2022, at 7:25 PM, Gary Gregory  wrote:
> >
> > So in a Commons centric fantasy, Log4j Nodes could be Commons Configuration
> > objects?
> >
> > G
> >
> > On Sun, Apr 3, 2022, 22:18 Ralph Goers  wrote:
> >
> >> If you look at Log4j, we parse the configuration and convert it to a Node
> >> hierarchy.
> >> Variable interpolation occurs as the Nodes are constructed.
> >>
> >> But again, Nothing says a Commons Plugins implementation has to work the
> >> same way.
> >>
> >> Ralph
> >>
> >>> On Apr 3, 2022, at 7:12 PM, Gary Gregory  wrote:
> >>>
> >>> Nice thread :-)
> >>>
> >>> Where does the parsing of configuration files mix in such a stack?
> >>>
> >>> Where does variable interpolation come into play?
> >>>
> >>> Gary
> >>>
> >>> On Sun, Apr 3, 2022, 20:50 Ralph Goers 
> >> wrote:
> >>>
>  Matt J,
> 
>  I fully expect that if commons-plugins came into fruition it would bear
> >> a
>  resemblance
>  to the Log4j plugin system but there would be differences. For example,
>  while Log4j
>  integrates with Spring we don’t currently support having the logging
>  configuration in
>  application.yml. I also suspect it would an almost impossible
> >> abstraction
>  to have the
>  DI system be pluggable, but I could be wrong about that.
> 
>  Of the bulleted items you listed Log4j’s plugin system handles all of
> >> them
>  except that
>  it doesn’t deal with dependencies. Instead, if your plugin is running in
>  an OSGi
>  environment it would be expected that the dependent modules would be
>  available as
>  described by the manifest for the module the plugin is included in.
> 
>  The goals for the log4j-plugins are really pretty simple.
>  * Allow applications to provide flexibility by exposing sets of
> >> pluggable
>  component types.
>  * Allow users to provide their own implementations of the pluggable
>  components that
>  are on an equal footing with the “standard” components.
> 
>  Just by way of example, Apache Flume allows pluggable components such as
>  Sources,
>  Sinks and Channels. Writing a new Sink is fairly straightforward. But it
>  will not be on
>  an equal footing with the Sinks provided by Flume. This is because there
>  is code in
>  Flume that provides an alias (such as “avro” for the AvroSink) so that
> >> the
>  plugin class
>  can be named without having to specify the fully qualified class name.
>  Custom
>  components must be configured using the FQCN.  The Log4j plugin system
>  solves
>  this by requiring every plugin to have a “name” which is then used as
> >> the
>  mechanism
>  to locate it from the configuration.
> 
>  Non-goals would be
>  * It is meant to load plugins, not be a full DI system for every
> >> component.
>  * It leverages standard class loading methodologies, not a new one. i.e.
>  it works with
>  the standard Java class path, module path, or OSGi.
> 
>  Ralph
> 
> 
> > On Apr 3, 2022, at 8:49 AM, Matt Juntunen 
>  wrote:
> >
> > Hi Matt,
> >
> > This is quite timely since I've spent the past week researching
> > frameworks to modularize a monolithic application at my day job into
> > separate components/plugins. Everything I've looked at so far is
> > larger and more complicated than I need (e.g. OSGi, Spring, etc) so I
> > was seriously considering writing my own, perhaps based on select
> > components from the Plexus project [1]. I would be interested in this
> > project if it could do the following:
> > - provide a standardized plugin packaging format;
> > - provide standardized configuration mechanisms;
> > - handle plugin discovery and enumeration;
> > - handle service discovery, enumeration, and dependency injection;
> > - handle class loading and resolution of plugin dependencies (e.g. a
> > plugin that depends on a different version of commons-lang than
> > another plugin); and
> > - provide plugin lifecycle methods.
> >
> > It would also be great if the project was compatible with different
> > frameworks. For example, if the dependency injection functionality
> > could be swapped out for Spring if needed.
> >
> > I haven't totally completed my research so I'm not sure if there is
> > actually an existing framework out there that satisfies the above
> > requirements. If not, I would be willing to help out to get this
> > implemented, regardless of whether it ends up in commons or not.
> >
> > One more thought: I think it would be equally important (if not more
> > 

Re: New component proposal: commons-plugins

[Ralph Goers]

It has been too long since I looked at Commons Configuration for me to answer 
that.

Ralph

> On Apr 3, 2022, at 7:25 PM, Gary Gregory  wrote:
> 
> So in a Commons centric fantasy, Log4j Nodes could be Commons Configuration
> objects?
> 
> G
> 
> On Sun, Apr 3, 2022, 22:18 Ralph Goers  wrote:
> 
>> If you look at Log4j, we parse the configuration and convert it to a Node
>> hierarchy.
>> Variable interpolation occurs as the Nodes are constructed.
>> 
>> But again, Nothing says a Commons Plugins implementation has to work the
>> same way.
>> 
>> Ralph
>> 
>>> On Apr 3, 2022, at 7:12 PM, Gary Gregory  wrote:
>>> 
>>> Nice thread :-)
>>> 
>>> Where does the parsing of configuration files mix in such a stack?
>>> 
>>> Where does variable interpolation come into play?
>>> 
>>> Gary
>>> 
>>> On Sun, Apr 3, 2022, 20:50 Ralph Goers 
>> wrote:
>>> 
 Matt J,
 
 I fully expect that if commons-plugins came into fruition it would bear
>> a
 resemblance
 to the Log4j plugin system but there would be differences. For example,
 while Log4j
 integrates with Spring we don’t currently support having the logging
 configuration in
 application.yml. I also suspect it would an almost impossible
>> abstraction
 to have the
 DI system be pluggable, but I could be wrong about that.
 
 Of the bulleted items you listed Log4j’s plugin system handles all of
>> them
 except that
 it doesn’t deal with dependencies. Instead, if your plugin is running in
 an OSGi
 environment it would be expected that the dependent modules would be
 available as
 described by the manifest for the module the plugin is included in.
 
 The goals for the log4j-plugins are really pretty simple.
 * Allow applications to provide flexibility by exposing sets of
>> pluggable
 component types.
 * Allow users to provide their own implementations of the pluggable
 components that
 are on an equal footing with the “standard” components.
 
 Just by way of example, Apache Flume allows pluggable components such as
 Sources,
 Sinks and Channels. Writing a new Sink is fairly straightforward. But it
 will not be on
 an equal footing with the Sinks provided by Flume. This is because there
 is code in
 Flume that provides an alias (such as “avro” for the AvroSink) so that
>> the
 plugin class
 can be named without having to specify the fully qualified class name.
 Custom
 components must be configured using the FQCN.  The Log4j plugin system
 solves
 this by requiring every plugin to have a “name” which is then used as
>> the
 mechanism
 to locate it from the configuration.
 
 Non-goals would be
 * It is meant to load plugins, not be a full DI system for every
>> component.
 * It leverages standard class loading methodologies, not a new one. i.e.
 it works with
 the standard Java class path, module path, or OSGi.
 
 Ralph
 
 
> On Apr 3, 2022, at 8:49 AM, Matt Juntunen 
 wrote:
> 
> Hi Matt,
> 
> This is quite timely since I've spent the past week researching
> frameworks to modularize a monolithic application at my day job into
> separate components/plugins. Everything I've looked at so far is
> larger and more complicated than I need (e.g. OSGi, Spring, etc) so I
> was seriously considering writing my own, perhaps based on select
> components from the Plexus project [1]. I would be interested in this
> project if it could do the following:
> - provide a standardized plugin packaging format;
> - provide standardized configuration mechanisms;
> - handle plugin discovery and enumeration;
> - handle service discovery, enumeration, and dependency injection;
> - handle class loading and resolution of plugin dependencies (e.g. a
> plugin that depends on a different version of commons-lang than
> another plugin); and
> - provide plugin lifecycle methods.
> 
> It would also be great if the project was compatible with different
> frameworks. For example, if the dependency injection functionality
> could be swapped out for Spring if needed.
> 
> I haven't totally completed my research so I'm not sure if there is
> actually an existing framework out there that satisfies the above
> requirements. If not, I would be willing to help out to get this
> implemented, regardless of whether it ends up in commons or not.
> 
> One more thought: I think it would be equally important (if not more
> so) to define the non-goals of this potential project as the goals. Do
> you have an idea of what those would be?
> 
> Regards,
> Matt J
> 
> [1] https://codehaus-plexus.github.io/
> 
> On Sun, Apr 3, 2022 at 6:24 AM Gary Gregory 
 wrote:
>> 
>> Should this be in Commons Configuration?
>> 
>> Gary
>> 
>> On Sat, Apr

Re: New component proposal: commons-plugins

[Matt Sicker]

Perhaps? I’d have to investigate how commons configuration works to be sure. 
And thanks, Ralph, for answering the questions here. I’ll write up a more 
detailed proposal we can discuss.

—
Matt Sicker

> On Apr 3, 2022, at 21:25, Gary Gregory  wrote:
> 
> So in a Commons centric fantasy, Log4j Nodes could be Commons Configuration
> objects?
> 
> G
> 
>> On Sun, Apr 3, 2022, 22:18 Ralph Goers  wrote:
>> 
>> If you look at Log4j, we parse the configuration and convert it to a Node
>> hierarchy.
>> Variable interpolation occurs as the Nodes are constructed.
>> 
>> But again, Nothing says a Commons Plugins implementation has to work the
>> same way.
>> 
>> Ralph
>> 
 On Apr 3, 2022, at 7:12 PM, Gary Gregory  wrote:
>>> 
>>> Nice thread :-)
>>> 
>>> Where does the parsing of configuration files mix in such a stack?
>>> 
>>> Where does variable interpolation come into play?
>>> 
>>> Gary
>>> 
>>> On Sun, Apr 3, 2022, 20:50 Ralph Goers 
>> wrote:
>>> 
 Matt J,
 
 I fully expect that if commons-plugins came into fruition it would bear
>> a
 resemblance
 to the Log4j plugin system but there would be differences. For example,
 while Log4j
 integrates with Spring we don’t currently support having the logging
 configuration in
 application.yml. I also suspect it would an almost impossible
>> abstraction
 to have the
 DI system be pluggable, but I could be wrong about that.
 
 Of the bulleted items you listed Log4j’s plugin system handles all of
>> them
 except that
 it doesn’t deal with dependencies. Instead, if your plugin is running in
 an OSGi
 environment it would be expected that the dependent modules would be
 available as
 described by the manifest for the module the plugin is included in.
 
 The goals for the log4j-plugins are really pretty simple.
 * Allow applications to provide flexibility by exposing sets of
>> pluggable
 component types.
 * Allow users to provide their own implementations of the pluggable
 components that
 are on an equal footing with the “standard” components.
 
 Just by way of example, Apache Flume allows pluggable components such as
 Sources,
 Sinks and Channels. Writing a new Sink is fairly straightforward. But it
 will not be on
 an equal footing with the Sinks provided by Flume. This is because there
 is code in
 Flume that provides an alias (such as “avro” for the AvroSink) so that
>> the
 plugin class
 can be named without having to specify the fully qualified class name.
 Custom
 components must be configured using the FQCN.  The Log4j plugin system
 solves
 this by requiring every plugin to have a “name” which is then used as
>> the
 mechanism
 to locate it from the configuration.
 
 Non-goals would be
 * It is meant to load plugins, not be a full DI system for every
>> component.
 * It leverages standard class loading methodologies, not a new one. i.e.
 it works with
 the standard Java class path, module path, or OSGi.
 
 Ralph
 
 
> On Apr 3, 2022, at 8:49 AM, Matt Juntunen 
 wrote:
> 
> Hi Matt,
> 
> This is quite timely since I've spent the past week researching
> frameworks to modularize a monolithic application at my day job into
> separate components/plugins. Everything I've looked at so far is
> larger and more complicated than I need (e.g. OSGi, Spring, etc) so I
> was seriously considering writing my own, perhaps based on select
> components from the Plexus project [1]. I would be interested in this
> project if it could do the following:
> - provide a standardized plugin packaging format;
> - provide standardized configuration mechanisms;
> - handle plugin discovery and enumeration;
> - handle service discovery, enumeration, and dependency injection;
> - handle class loading and resolution of plugin dependencies (e.g. a
> plugin that depends on a different version of commons-lang than
> another plugin); and
> - provide plugin lifecycle methods.
> 
> It would also be great if the project was compatible with different
> frameworks. For example, if the dependency injection functionality
> could be swapped out for Spring if needed.
> 
> I haven't totally completed my research so I'm not sure if there is
> actually an existing framework out there that satisfies the above
> requirements. If not, I would be willing to help out to get this
> implemented, regardless of whether it ends up in commons or not.
> 
> One more thought: I think it would be equally important (if not more
> so) to define the non-goals of this potential project as the goals. Do
> you have an idea of what those would be?
> 
> Regards,
> Matt J
> 
> [1] https://codehaus-plexus.github.io/
> 
> On Sun, Apr 3, 2022 at 6:24 AM Gary Gregory 
>>

Re: New component proposal: commons-plugins

[Gary Gregory]

So in a Commons centric fantasy, Log4j Nodes could be Commons Configuration
objects?

G

On Sun, Apr 3, 2022, 22:18 Ralph Goers  wrote:

> If you look at Log4j, we parse the configuration and convert it to a Node
> hierarchy.
> Variable interpolation occurs as the Nodes are constructed.
>
> But again, Nothing says a Commons Plugins implementation has to work the
> same way.
>
> Ralph
>
> > On Apr 3, 2022, at 7:12 PM, Gary Gregory  wrote:
> >
> > Nice thread :-)
> >
> > Where does the parsing of configuration files mix in such a stack?
> >
> > Where does variable interpolation come into play?
> >
> > Gary
> >
> > On Sun, Apr 3, 2022, 20:50 Ralph Goers 
> wrote:
> >
> >> Matt J,
> >>
> >> I fully expect that if commons-plugins came into fruition it would bear
> a
> >> resemblance
> >> to the Log4j plugin system but there would be differences. For example,
> >> while Log4j
> >> integrates with Spring we don’t currently support having the logging
> >> configuration in
> >> application.yml. I also suspect it would an almost impossible
> abstraction
> >> to have the
> >> DI system be pluggable, but I could be wrong about that.
> >>
> >> Of the bulleted items you listed Log4j’s plugin system handles all of
> them
> >> except that
> >> it doesn’t deal with dependencies. Instead, if your plugin is running in
> >> an OSGi
> >> environment it would be expected that the dependent modules would be
> >> available as
> >> described by the manifest for the module the plugin is included in.
> >>
> >> The goals for the log4j-plugins are really pretty simple.
> >> * Allow applications to provide flexibility by exposing sets of
> pluggable
> >> component types.
> >> * Allow users to provide their own implementations of the pluggable
> >> components that
> >> are on an equal footing with the “standard” components.
> >>
> >> Just by way of example, Apache Flume allows pluggable components such as
> >> Sources,
> >> Sinks and Channels. Writing a new Sink is fairly straightforward. But it
> >> will not be on
> >> an equal footing with the Sinks provided by Flume. This is because there
> >> is code in
> >> Flume that provides an alias (such as “avro” for the AvroSink) so that
> the
> >> plugin class
> >> can be named without having to specify the fully qualified class name.
> >> Custom
> >> components must be configured using the FQCN.  The Log4j plugin system
> >> solves
> >> this by requiring every plugin to have a “name” which is then used as
> the
> >> mechanism
> >> to locate it from the configuration.
> >>
> >> Non-goals would be
> >> * It is meant to load plugins, not be a full DI system for every
> component.
> >> * It leverages standard class loading methodologies, not a new one. i.e.
> >> it works with
> >> the standard Java class path, module path, or OSGi.
> >>
> >> Ralph
> >>
> >>
> >>> On Apr 3, 2022, at 8:49 AM, Matt Juntunen 
> >> wrote:
> >>>
> >>> Hi Matt,
> >>>
> >>> This is quite timely since I've spent the past week researching
> >>> frameworks to modularize a monolithic application at my day job into
> >>> separate components/plugins. Everything I've looked at so far is
> >>> larger and more complicated than I need (e.g. OSGi, Spring, etc) so I
> >>> was seriously considering writing my own, perhaps based on select
> >>> components from the Plexus project [1]. I would be interested in this
> >>> project if it could do the following:
> >>> - provide a standardized plugin packaging format;
> >>> - provide standardized configuration mechanisms;
> >>> - handle plugin discovery and enumeration;
> >>> - handle service discovery, enumeration, and dependency injection;
> >>> - handle class loading and resolution of plugin dependencies (e.g. a
> >>> plugin that depends on a different version of commons-lang than
> >>> another plugin); and
> >>> - provide plugin lifecycle methods.
> >>>
> >>> It would also be great if the project was compatible with different
> >>> frameworks. For example, if the dependency injection functionality
> >>> could be swapped out for Spring if needed.
> >>>
> >>> I haven't totally completed my research so I'm not sure if there is
> >>> actually an existing framework out there that satisfies the above
> >>> requirements. If not, I would be willing to help out to get this
> >>> implemented, regardless of whether it ends up in commons or not.
> >>>
> >>> One more thought: I think it would be equally important (if not more
> >>> so) to define the non-goals of this potential project as the goals. Do
> >>> you have an idea of what those would be?
> >>>
> >>> Regards,
> >>> Matt J
> >>>
> >>> [1] https://codehaus-plexus.github.io/
> >>>
> >>> On Sun, Apr 3, 2022 at 6:24 AM Gary Gregory 
> >> wrote:
> 
>  Should this be in Commons Configuration?
> 
>  Gary
> 
>  On Sat, Apr 2, 2022, 15:33 Matt Sicker  wrote:
> 
> > Hi all, I’ve got a proposal for a new Commons component that I’d like
> >> to
> > get feedback on. Essentially, I’d like to propose the creation of a
>

Re: New component proposal: commons-plugins

[Ralph Goers]

If you look at Log4j, we parse the configuration and convert it to a Node 
hierarchy. 
Variable interpolation occurs as the Nodes are constructed.

But again, Nothing says a Commons Plugins implementation has to work the same 
way.

Ralph

> On Apr 3, 2022, at 7:12 PM, Gary Gregory  wrote:
> 
> Nice thread :-)
> 
> Where does the parsing of configuration files mix in such a stack?
> 
> Where does variable interpolation come into play?
> 
> Gary
> 
> On Sun, Apr 3, 2022, 20:50 Ralph Goers  wrote:
> 
>> Matt J,
>> 
>> I fully expect that if commons-plugins came into fruition it would bear a
>> resemblance
>> to the Log4j plugin system but there would be differences. For example,
>> while Log4j
>> integrates with Spring we don’t currently support having the logging
>> configuration in
>> application.yml. I also suspect it would an almost impossible abstraction
>> to have the
>> DI system be pluggable, but I could be wrong about that.
>> 
>> Of the bulleted items you listed Log4j’s plugin system handles all of them
>> except that
>> it doesn’t deal with dependencies. Instead, if your plugin is running in
>> an OSGi
>> environment it would be expected that the dependent modules would be
>> available as
>> described by the manifest for the module the plugin is included in.
>> 
>> The goals for the log4j-plugins are really pretty simple.
>> * Allow applications to provide flexibility by exposing sets of pluggable
>> component types.
>> * Allow users to provide their own implementations of the pluggable
>> components that
>> are on an equal footing with the “standard” components.
>> 
>> Just by way of example, Apache Flume allows pluggable components such as
>> Sources,
>> Sinks and Channels. Writing a new Sink is fairly straightforward. But it
>> will not be on
>> an equal footing with the Sinks provided by Flume. This is because there
>> is code in
>> Flume that provides an alias (such as “avro” for the AvroSink) so that the
>> plugin class
>> can be named without having to specify the fully qualified class name.
>> Custom
>> components must be configured using the FQCN.  The Log4j plugin system
>> solves
>> this by requiring every plugin to have a “name” which is then used as the
>> mechanism
>> to locate it from the configuration.
>> 
>> Non-goals would be
>> * It is meant to load plugins, not be a full DI system for every component.
>> * It leverages standard class loading methodologies, not a new one. i.e.
>> it works with
>> the standard Java class path, module path, or OSGi.
>> 
>> Ralph
>> 
>> 
>>> On Apr 3, 2022, at 8:49 AM, Matt Juntunen 
>> wrote:
>>> 
>>> Hi Matt,
>>> 
>>> This is quite timely since I've spent the past week researching
>>> frameworks to modularize a monolithic application at my day job into
>>> separate components/plugins. Everything I've looked at so far is
>>> larger and more complicated than I need (e.g. OSGi, Spring, etc) so I
>>> was seriously considering writing my own, perhaps based on select
>>> components from the Plexus project [1]. I would be interested in this
>>> project if it could do the following:
>>> - provide a standardized plugin packaging format;
>>> - provide standardized configuration mechanisms;
>>> - handle plugin discovery and enumeration;
>>> - handle service discovery, enumeration, and dependency injection;
>>> - handle class loading and resolution of plugin dependencies (e.g. a
>>> plugin that depends on a different version of commons-lang than
>>> another plugin); and
>>> - provide plugin lifecycle methods.
>>> 
>>> It would also be great if the project was compatible with different
>>> frameworks. For example, if the dependency injection functionality
>>> could be swapped out for Spring if needed.
>>> 
>>> I haven't totally completed my research so I'm not sure if there is
>>> actually an existing framework out there that satisfies the above
>>> requirements. If not, I would be willing to help out to get this
>>> implemented, regardless of whether it ends up in commons or not.
>>> 
>>> One more thought: I think it would be equally important (if not more
>>> so) to define the non-goals of this potential project as the goals. Do
>>> you have an idea of what those would be?
>>> 
>>> Regards,
>>> Matt J
>>> 
>>> [1] https://codehaus-plexus.github.io/
>>> 
>>> On Sun, Apr 3, 2022 at 6:24 AM Gary Gregory 
>> wrote:
 
 Should this be in Commons Configuration?
 
 Gary
 
 On Sat, Apr 2, 2022, 15:33 Matt Sicker  wrote:
 
> Hi all, I’ve got a proposal for a new Commons component that I’d like
>> to
> get feedback on. Essentially, I’d like to propose the creation of a
>> Commons
> Plugins component inspired by the plugin system developed for Log4j 3.x
> [0]. This library would be a lightweight dependency injection and
> configuration library where developers create pluggable classes that
>> can be
> referenced through plugin names via the configuration file (or
> configuration source in general). In contrast wit

Re: New component proposal: commons-plugins

[Gary Gregory]

Nice thread :-)

Where does the parsing of configuration files mix in such a stack?

Where does variable interpolation come into play?

Gary

On Sun, Apr 3, 2022, 20:50 Ralph Goers  wrote:

> Matt J,
>
> I fully expect that if commons-plugins came into fruition it would bear a
> resemblance
> to the Log4j plugin system but there would be differences. For example,
> while Log4j
> integrates with Spring we don’t currently support having the logging
> configuration in
> application.yml. I also suspect it would an almost impossible abstraction
> to have the
> DI system be pluggable, but I could be wrong about that.
>
> Of the bulleted items you listed Log4j’s plugin system handles all of them
> except that
> it doesn’t deal with dependencies. Instead, if your plugin is running in
> an OSGi
> environment it would be expected that the dependent modules would be
> available as
> described by the manifest for the module the plugin is included in.
>
> The goals for the log4j-plugins are really pretty simple.
> * Allow applications to provide flexibility by exposing sets of pluggable
> component types.
> * Allow users to provide their own implementations of the pluggable
> components that
> are on an equal footing with the “standard” components.
>
> Just by way of example, Apache Flume allows pluggable components such as
> Sources,
> Sinks and Channels. Writing a new Sink is fairly straightforward. But it
> will not be on
> an equal footing with the Sinks provided by Flume. This is because there
> is code in
> Flume that provides an alias (such as “avro” for the AvroSink) so that the
> plugin class
> can be named without having to specify the fully qualified class name.
> Custom
> components must be configured using the FQCN.  The Log4j plugin system
> solves
> this by requiring every plugin to have a “name” which is then used as the
> mechanism
> to locate it from the configuration.
>
> Non-goals would be
> * It is meant to load plugins, not be a full DI system for every component.
> * It leverages standard class loading methodologies, not a new one. i.e.
> it works with
> the standard Java class path, module path, or OSGi.
>
> Ralph
>
>
> > On Apr 3, 2022, at 8:49 AM, Matt Juntunen 
> wrote:
> >
> > Hi Matt,
> >
> > This is quite timely since I've spent the past week researching
> > frameworks to modularize a monolithic application at my day job into
> > separate components/plugins. Everything I've looked at so far is
> > larger and more complicated than I need (e.g. OSGi, Spring, etc) so I
> > was seriously considering writing my own, perhaps based on select
> > components from the Plexus project [1]. I would be interested in this
> > project if it could do the following:
> > - provide a standardized plugin packaging format;
> > - provide standardized configuration mechanisms;
> > - handle plugin discovery and enumeration;
> > - handle service discovery, enumeration, and dependency injection;
> > - handle class loading and resolution of plugin dependencies (e.g. a
> > plugin that depends on a different version of commons-lang than
> > another plugin); and
> > - provide plugin lifecycle methods.
> >
> > It would also be great if the project was compatible with different
> > frameworks. For example, if the dependency injection functionality
> > could be swapped out for Spring if needed.
> >
> > I haven't totally completed my research so I'm not sure if there is
> > actually an existing framework out there that satisfies the above
> > requirements. If not, I would be willing to help out to get this
> > implemented, regardless of whether it ends up in commons or not.
> >
> > One more thought: I think it would be equally important (if not more
> > so) to define the non-goals of this potential project as the goals. Do
> > you have an idea of what those would be?
> >
> > Regards,
> > Matt J
> >
> > [1] https://codehaus-plexus.github.io/
> >
> > On Sun, Apr 3, 2022 at 6:24 AM Gary Gregory 
> wrote:
> >>
> >> Should this be in Commons Configuration?
> >>
> >> Gary
> >>
> >> On Sat, Apr 2, 2022, 15:33 Matt Sicker  wrote:
> >>
> >>> Hi all, I’ve got a proposal for a new Commons component that I’d like
> to
> >>> get feedback on. Essentially, I’d like to propose the creation of a
> Commons
> >>> Plugins component inspired by the plugin system developed for Log4j 3.x
> >>> [0]. This library would be a lightweight dependency injection and
> >>> configuration library where developers create pluggable classes that
> can be
> >>> referenced through plugin names via the configuration file (or
> >>> configuration source in general). In contrast with more typical
> dependency
> >>> injection frameworks like Spring and Guice, this library is for
> >>> applications where pluggable implementations of things is desired.
> >>> Developing a plugin system on top of DI frameworks is not a very
> >>> standardized domain, and each project ends up reinventing this from
> scratch
> >>> over time.
> >>>
> >>> Some existing material on how the Lo

Re: New component proposal: commons-plugins

[Ralph Goers]

Matt J,

I fully expect that if commons-plugins came into fruition it would bear a 
resemblance 
to the Log4j plugin system but there would be differences. For example, while 
Log4j 
integrates with Spring we don’t currently support having the logging 
configuration in 
application.yml. I also suspect it would an almost impossible abstraction to 
have the 
DI system be pluggable, but I could be wrong about that.

Of the bulleted items you listed Log4j’s plugin system handles all of them 
except that 
it doesn’t deal with dependencies. Instead, if your plugin is running in an 
OSGi 
environment it would be expected that the dependent modules would be available 
as 
described by the manifest for the module the plugin is included in.

The goals for the log4j-plugins are really pretty simple. 
* Allow applications to provide flexibility by exposing sets of pluggable 
component types. 
* Allow users to provide their own implementations of the pluggable components 
that 
are on an equal footing with the “standard” components.

Just by way of example, Apache Flume allows pluggable components such as 
Sources, 
Sinks and Channels. Writing a new Sink is fairly straightforward. But it will 
not be on 
an equal footing with the Sinks provided by Flume. This is because there is 
code in 
Flume that provides an alias (such as “avro” for the AvroSink) so that the 
plugin class 
can be named without having to specify the fully qualified class name. Custom 
components must be configured using the FQCN.  The Log4j plugin system solves 
this by requiring every plugin to have a “name” which is then used as the 
mechanism 
to locate it from the configuration.

Non-goals would be 
* It is meant to load plugins, not be a full DI system for every component.
* It leverages standard class loading methodologies, not a new one. i.e. it 
works with 
the standard Java class path, module path, or OSGi.

Ralph


> On Apr 3, 2022, at 8:49 AM, Matt Juntunen  wrote:
> 
> Hi Matt,
> 
> This is quite timely since I've spent the past week researching
> frameworks to modularize a monolithic application at my day job into
> separate components/plugins. Everything I've looked at so far is
> larger and more complicated than I need (e.g. OSGi, Spring, etc) so I
> was seriously considering writing my own, perhaps based on select
> components from the Plexus project [1]. I would be interested in this
> project if it could do the following:
> - provide a standardized plugin packaging format;
> - provide standardized configuration mechanisms;
> - handle plugin discovery and enumeration;
> - handle service discovery, enumeration, and dependency injection;
> - handle class loading and resolution of plugin dependencies (e.g. a
> plugin that depends on a different version of commons-lang than
> another plugin); and
> - provide plugin lifecycle methods.
> 
> It would also be great if the project was compatible with different
> frameworks. For example, if the dependency injection functionality
> could be swapped out for Spring if needed.
> 
> I haven't totally completed my research so I'm not sure if there is
> actually an existing framework out there that satisfies the above
> requirements. If not, I would be willing to help out to get this
> implemented, regardless of whether it ends up in commons or not.
> 
> One more thought: I think it would be equally important (if not more
> so) to define the non-goals of this potential project as the goals. Do
> you have an idea of what those would be?
> 
> Regards,
> Matt J
> 
> [1] https://codehaus-plexus.github.io/
> 
> On Sun, Apr 3, 2022 at 6:24 AM Gary Gregory  wrote:
>> 
>> Should this be in Commons Configuration?
>> 
>> Gary
>> 
>> On Sat, Apr 2, 2022, 15:33 Matt Sicker  wrote:
>> 
>>> Hi all, I’ve got a proposal for a new Commons component that I’d like to
>>> get feedback on. Essentially, I’d like to propose the creation of a Commons
>>> Plugins component inspired by the plugin system developed for Log4j 3.x
>>> [0]. This library would be a lightweight dependency injection and
>>> configuration library where developers create pluggable classes that can be
>>> referenced through plugin names via the configuration file (or
>>> configuration source in general). In contrast with more typical dependency
>>> injection frameworks like Spring and Guice, this library is for
>>> applications where pluggable implementations of things is desired.
>>> Developing a plugin system on top of DI frameworks is not a very
>>> standardized domain, and each project ends up reinventing this from scratch
>>> over time.
>>> 
>>> Some existing material on how the Log4j plugin and configuration system
>>> works that I’d adapt from to form the basis for this component include:
>>> -
>>> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
>>> <
>>> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
 
>>> -
>>> https://github.com/

Re: New component proposal: commons-plugins

[Ralph Goers]

I don’t think so.  Commons Configuration (which I was working actively on when 
I started writing Log4j 2) is designed to provide access to the configuration 
data. 
The plugin system is design to convert configuration data into instantiated 
objects. 
I could easily see Commons Configuration being used as the input to the plugin 
system, but I think it would be a mistake to tightly couple them.

Ralph

> On Apr 3, 2022, at 3:23 AM, Gary Gregory  wrote:
> 
> Should this be in Commons Configuration?
> 
> Gary
> 
> On Sat, Apr 2, 2022, 15:33 Matt Sicker  wrote:
> 
>> Hi all, I’ve got a proposal for a new Commons component that I’d like to
>> get feedback on. Essentially, I’d like to propose the creation of a Commons
>> Plugins component inspired by the plugin system developed for Log4j 3.x
>> [0]. This library would be a lightweight dependency injection and
>> configuration library where developers create pluggable classes that can be
>> referenced through plugin names via the configuration file (or
>> configuration source in general). In contrast with more typical dependency
>> injection frameworks like Spring and Guice, this library is for
>> applications where pluggable implementations of things is desired.
>> Developing a plugin system on top of DI frameworks is not a very
>> standardized domain, and each project ends up reinventing this from scratch
>> over time.
>> 
>> Some existing material on how the Log4j plugin and configuration system
>> works that I’d adapt from to form the basis for this component include:
>> -
>> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
>> <
>> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
>>> 
>> -
>> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/plugins.adoc
>> <
>> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/plugins.adoc
>>> 
>> 
>> The general goal of this library is to make it so that applications can
>> provide better configuration DSLs for their plugin components. As both a
>> developer and user, I absolutely despise configuring complex applications
>> with properties files, and YAML variants of properties aren’t that much
>> better. If there was a common plugin library we could reuse, then perhaps
>> more applications would support a better configuration system. This could
>> also provide a nice place for tooling integration similar to how JUnit is
>> supported by IDEs and other tools.
>> 
>> What do you think? Should this start in the Sandbox? Is anyone interested
>> in working on or using this?
>> 
>> [0]: https://github.com/apache/logging-log4j2/tree/master/log4j-plugins <
>> https://github.com/apache/logging-log4j2/tree/master/log4j-plugins>
>> 
>> —
>> Matt Sicker
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: New component proposal: commons-plugins

[Ralph Goers]

ServliceLoader is used to locate plugins but what is loaded by ServiceLoader 
are not plugins.

Log4j Plugins are categorized into different groupings, primarily by where and 
how the plugins
 are used. For example, PatternConverters used by the PatternLayout are 
separate from 
Appenders, Filters and other things that can appear in a Log4j configuration 
file.

Loading all of Log4j’s plugins via ServiceLoader would be very slow and would 
require every 
plugin be listed in the service configuration file. This has several problems:
1. A service must have a constructor that accepts no parameters.
2. Manually configuring every plugin in the META-INF/services file would be 
painful.
3. Every plugin might have its own unique configuration. ServiceLoader provides 
no mechanism 
for configuration.

Log4j solves these problems by using the Java annotation processor to detect 
all the plugins 
defined in the compilation and generates the service instance ServiceLoader 
will load. This 
service instance maintains the list of all the plugins that were defined along 
with other information 
such as their category and type. The annotation processor also generates the 
META-INF/services 
file automatically.

In addition, every plugin has a Builder with annotated parameters. This allows 
a configuration 
system to easily construct all its components. For example, Log4j allows JSON, 
YAML, XML, 
and properties. All of these use the exact same code to instantiate the Log4j 
configuration.

I hope that answers your question.

Ralph

> On Apr 2, 2022, at 9:57 PM, Peter Verhas  wrote:
> 
> What is the feature not provided by Service Loader?
> 
> On Sat 2. Apr 2022 at 21:33, Matt Sicker  wrote:
> 
>> Hi all, I’ve got a proposal for a new Commons component that I’d like to
>> get feedback on. Essentially, I’d like to propose the creation of a Commons
>> Plugins component inspired by the plugin system developed for Log4j 3.x
>> [0]. This library would be a lightweight dependency injection and
>> configuration library where developers create pluggable classes that can be
>> referenced through plugin names via the configuration file (or
>> configuration source in general). In contrast with more typical dependency
>> injection frameworks like Spring and Guice, this library is for
>> applications where pluggable implementations of things is desired.
>> Developing a plugin system on top of DI frameworks is not a very
>> standardized domain, and each project ends up reinventing this from scratch
>> over time.
>> 
>> Some existing material on how the Log4j plugin and configuration system
>> works that I’d adapt from to form the basis for this component include:
>> -
>> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
>> <
>> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
>>> 
>> -
>> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/plugins.adoc
>> <
>> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/plugins.adoc
>>> 
>> 
>> The general goal of this library is to make it so that applications can
>> provide better configuration DSLs for their plugin components. As both a
>> developer and user, I absolutely despise configuring complex applications
>> with properties files, and YAML variants of properties aren’t that much
>> better. If there was a common plugin library we could reuse, then perhaps
>> more applications would support a better configuration system. This could
>> also provide a nice place for tooling integration similar to how JUnit is
>> supported by IDEs and other tools.
>> 
>> What do you think? Should this start in the Sandbox? Is anyone interested
>> in working on or using this?
>> 
>> [0]: https://github.com/apache/logging-log4j2/tree/master/log4j-plugins <
>> https://github.com/apache/logging-log4j2/tree/master/log4j-plugins>
>> 
>> —
>> Matt Sicker
>> 
>> --
> Peter Verhas
> pe...@verhas.com
> t: +41791542095
> skype: verhas


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: New component proposal: commons-plugins

[Matt Juntunen]

Hi Matt,

This is quite timely since I've spent the past week researching
frameworks to modularize a monolithic application at my day job into
separate components/plugins. Everything I've looked at so far is
larger and more complicated than I need (e.g. OSGi, Spring, etc) so I
was seriously considering writing my own, perhaps based on select
components from the Plexus project [1]. I would be interested in this
project if it could do the following:
- provide a standardized plugin packaging format;
- provide standardized configuration mechanisms;
- handle plugin discovery and enumeration;
- handle service discovery, enumeration, and dependency injection;
- handle class loading and resolution of plugin dependencies (e.g. a
plugin that depends on a different version of commons-lang than
another plugin); and
- provide plugin lifecycle methods.

It would also be great if the project was compatible with different
frameworks. For example, if the dependency injection functionality
could be swapped out for Spring if needed.

I haven't totally completed my research so I'm not sure if there is
actually an existing framework out there that satisfies the above
requirements. If not, I would be willing to help out to get this
implemented, regardless of whether it ends up in commons or not.

One more thought: I think it would be equally important (if not more
so) to define the non-goals of this potential project as the goals. Do
you have an idea of what those would be?

Regards,
Matt J

[1] https://codehaus-plexus.github.io/

On Sun, Apr 3, 2022 at 6:24 AM Gary Gregory  wrote:
>
> Should this be in Commons Configuration?
>
> Gary
>
> On Sat, Apr 2, 2022, 15:33 Matt Sicker  wrote:
>
> > Hi all, I’ve got a proposal for a new Commons component that I’d like to
> > get feedback on. Essentially, I’d like to propose the creation of a Commons
> > Plugins component inspired by the plugin system developed for Log4j 3.x
> > [0]. This library would be a lightweight dependency injection and
> > configuration library where developers create pluggable classes that can be
> > referenced through plugin names via the configuration file (or
> > configuration source in general). In contrast with more typical dependency
> > injection frameworks like Spring and Guice, this library is for
> > applications where pluggable implementations of things is desired.
> > Developing a plugin system on top of DI frameworks is not a very
> > standardized domain, and each project ends up reinventing this from scratch
> > over time.
> >
> > Some existing material on how the Log4j plugin and configuration system
> > works that I’d adapt from to form the basis for this component include:
> > -
> > https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
> > <
> > https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
> > >
> > -
> > https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/plugins.adoc
> > <
> > https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/plugins.adoc
> > >
> >
> > The general goal of this library is to make it so that applications can
> > provide better configuration DSLs for their plugin components. As both a
> > developer and user, I absolutely despise configuring complex applications
> > with properties files, and YAML variants of properties aren’t that much
> > better. If there was a common plugin library we could reuse, then perhaps
> > more applications would support a better configuration system. This could
> > also provide a nice place for tooling integration similar to how JUnit is
> > supported by IDEs and other tools.
> >
> > What do you think? Should this start in the Sandbox? Is anyone interested
> > in working on or using this?
> >
> > [0]: https://github.com/apache/logging-log4j2/tree/master/log4j-plugins <
> > https://github.com/apache/logging-log4j2/tree/master/log4j-plugins>
> >
> > —
> > Matt Sicker
> >
> >

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: New component proposal: commons-plugins

[Gary Gregory]

Should this be in Commons Configuration?

Gary

On Sat, Apr 2, 2022, 15:33 Matt Sicker  wrote:

> Hi all, I’ve got a proposal for a new Commons component that I’d like to
> get feedback on. Essentially, I’d like to propose the creation of a Commons
> Plugins component inspired by the plugin system developed for Log4j 3.x
> [0]. This library would be a lightweight dependency injection and
> configuration library where developers create pluggable classes that can be
> referenced through plugin names via the configuration file (or
> configuration source in general). In contrast with more typical dependency
> injection frameworks like Spring and Guice, this library is for
> applications where pluggable implementations of things is desired.
> Developing a plugin system on top of DI frameworks is not a very
> standardized domain, and each project ends up reinventing this from scratch
> over time.
>
> Some existing material on how the Log4j plugin and configuration system
> works that I’d adapt from to form the basis for this component include:
> -
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
> <
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
> >
> -
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/plugins.adoc
> <
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/plugins.adoc
> >
>
> The general goal of this library is to make it so that applications can
> provide better configuration DSLs for their plugin components. As both a
> developer and user, I absolutely despise configuring complex applications
> with properties files, and YAML variants of properties aren’t that much
> better. If there was a common plugin library we could reuse, then perhaps
> more applications would support a better configuration system. This could
> also provide a nice place for tooling integration similar to how JUnit is
> supported by IDEs and other tools.
>
> What do you think? Should this start in the Sandbox? Is anyone interested
> in working on or using this?
>
> [0]: https://github.com/apache/logging-log4j2/tree/master/log4j-plugins <
> https://github.com/apache/logging-log4j2/tree/master/log4j-plugins>
>
> —
> Matt Sicker
>
>

Re: New component proposal: commons-plugins

[Peter Verhas]

What is the feature not provided by Service Loader?

On Sat 2. Apr 2022 at 21:33, Matt Sicker  wrote:

> Hi all, I’ve got a proposal for a new Commons component that I’d like to
> get feedback on. Essentially, I’d like to propose the creation of a Commons
> Plugins component inspired by the plugin system developed for Log4j 3.x
> [0]. This library would be a lightweight dependency injection and
> configuration library where developers create pluggable classes that can be
> referenced through plugin names via the configuration file (or
> configuration source in general). In contrast with more typical dependency
> injection frameworks like Spring and Guice, this library is for
> applications where pluggable implementations of things is desired.
> Developing a plugin system on top of DI frameworks is not a very
> standardized domain, and each project ends up reinventing this from scratch
> over time.
>
> Some existing material on how the Log4j plugin and configuration system
> works that I’d adapt from to form the basis for this component include:
> -
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
> <
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/dependencyinjection.adoc
> >
> -
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/plugins.adoc
> <
> https://github.com/apache/logging-log4j2/blob/master/src/site/asciidoc/manual/plugins.adoc
> >
>
> The general goal of this library is to make it so that applications can
> provide better configuration DSLs for their plugin components. As both a
> developer and user, I absolutely despise configuring complex applications
> with properties files, and YAML variants of properties aren’t that much
> better. If there was a common plugin library we could reuse, then perhaps
> more applications would support a better configuration system. This could
> also provide a nice place for tooling integration similar to how JUnit is
> supported by IDEs and other tools.
>
> What do you think? Should this start in the Sandbox? Is anyone interested
> in working on or using this?
>
> [0]: https://github.com/apache/logging-log4j2/tree/master/log4j-plugins <
> https://github.com/apache/logging-log4j2/tree/master/log4j-plugins>
>
> —
> Matt Sicker
>
> --
Peter Verhas
pe...@verhas.com
t: +41791542095
skype: verhas