Re: [VOTE] Release Log4j 2.14.1-rc1
Thanks so much Ralph, I can indeed view https://logging.staged.apache.org/log4j/2.x/manual/json-template-layout.html https://logging.staged.apache.org/log4j/2.x/manual/layouts.html#JsonTemplateLayout still misses the correction I have shared, but not a big issue. I can clean it up later on. On Sun, Mar 7, 2021 at 11:39 PM Ralph Goers wrote: > Json-Template-layout is available in the staging site now. > > Ralph > > > On Mar 7, 2021, at 2:16 PM, Ralph Goers > wrote: > > > > I will rebuild the site and update that one file. > > > > Ralph > > > >> On Mar 7, 2021, at 2:13 PM, Ralph Goers > wrote: > >> > >> The generated site contains json-template-layout.vm.html in the manual > directory. I see json-template-layout.vm.adoc in the source, which I > suspect is the source of the problem. I believe the file needs to be named > json-template-layout.adoc.vm to be processed by velocity and end up with an > adoc file. > >> > >> Ralph > >> > >>> On Mar 7, 2021, at 2:08 PM, Volkan Yazıcı > wrote: > >>> > >>> Ah! If that is the case, definitely +1! (Sorry for the inconvenience.) > >>> > >>> On Sun, 7 Mar 2021, 22:06 Gary Gregory wrote: > >>> > I don't think we need to fail the RC if the site generation is the > only > item impacted as opposed to the running software since its relatively > easy > to fix the site after the release. Up to the RM of course ;-) > > Gary > > > On Sun, Mar 7, 2021, 15:58 Ralph Goers > wrote: > > > See - https://issues.apache.org/jira/projects/MDOAP/issues/MDOAP-61. > This > > is documented as part of the release process. > > > > I don’t understand the impact of the missing > json-template-layout.html. > > Is that from the web site? Was it present in 2.14.0 and why is it > missing > > now? > > > > Ralph > > > > > > > >> On Mar 7, 2021, at 1:37 PM, Volkan Yazıcı > > wrote: > >> > >> -1, due to missing `json-template-layout.html` file! > >> > >> `./mvnw site` fails for me thanks to maven-doap-plugin: > >> > >> [ERROR] Failed to execute goal > >> org.apache.maven.plugins:maven-doap-plugin:1.2:generate (site) on > project > >> log4j: Execution site of goal > >> org.apache.maven.plugins:maven-doap-plugin:1.2:generate failed: An > API > >> incompatibility was encountered while executing > >> org.apache.maven.plugins:maven-doap-plugin:1.2:generate: > >> java.lang.ExceptionInInitializerError: null > >> [ERROR] - > >> [ERROR] realm = > plugin>org.apache.maven.plugins:maven-doap-plugin:1.2 > >> [ERROR] strategy = > >> org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy > >> [ERROR] urls[0] = > >> > > > > file:/home/vy/.m2/repository/org/apache/maven/plugins/maven-doap-plugin/1.2/maven-doap-plugin-1.2.jar > >> [ERROR] urls[1] = > >> > > > > file:/home/vy/.m2/repository/backport-util-concurrent/backport-util-concurrent/3.1/backport-util-concurrent-3.1.jar > >> [ERROR] urls[2] = > >> file:/home/vy/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar > >> [ERROR] urls[3] = > >> > > > > file:/home/vy/.m2/repository/org/apache/maven/scm/maven-scm-api/1.9.2/maven-scm-api-1.9.2.jar > >> [ERROR] urls[4] = > >> > > > > file:/home/vy/.m2/repository/org/apache/maven/scm/maven-scm-manager-plexus/1.9.2/maven-scm-manager-plexus-1.9.2.jar > >> [ERROR] urls[5] = > >> > > > > file:/home/vy/.m2/repository/org/apache/maven/scm/maven-scm-provider-svn-commons/1.9.2/maven-scm-provider-svn-commons-1.9.2.jar > >> [ERROR] urls[6] = > >> > > > > file:/home/vy/.m2/repository/org/apache/maven/scm/maven-scm-provider-svnexe/1.9.2/maven-scm-provider-svnexe-1.9.2.jar > >> [ERROR] urls[7] = > >> > > > > file:/home/vy/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar > >> [ERROR] urls[8] = > >> > > > > file:/home/vy/.m2/repository/org/apache/maven/scm/maven-scm-provider-cvs-commons/1.9.2/maven-scm-provider-cvs-commons-1.9.2.jar > >> [ERROR] urls[9] = > >> > > > > file:/home/vy/.m2/repository/org/apache/maven/scm/maven-scm-provider-cvsexe/1.9.2/maven-scm-provider-cvsexe-1.9.2.jar > >> [ERROR] urls[10] = > >> > > > > file:/home/vy/.m2/repository/org/apache/maven/scm/maven-scm-provider-cvsjava/1.9.2/maven-scm-provider-cvsjava-1.9.2.jar > >> [ERROR] urls[11] = > >> > > > > file:/home/vy/.m2/repository/org/netbeans/lib/cvsclient/20060125/cvsclient-20060125.jar > >> [ERROR] urls[12] = > >> > > > > file:/home/vy/.m2/repository/ch/ethz/ganymed/ganymed-ssh2/build210/ganymed-ssh2-build210.jar > >> [ERROR] urls[13] = > >> > > > > file:/home/vy/.m2/repository/org/codehaus/plexus/plexus-utils/3.0.20/plexus-utils-3.0.20.jar > >> [ERROR] urls[14] = > >> > > > > f
Re: [VOTE] Release Log4j 2.14.1-rc1
+1 Verified the snapshots in a few internal projects, as well as code review and tests on the tag. Test pass on ubuntu: $ mvn clean && mvn install /usr/lib/jvm/java-11-openjdk-amd64/bin/java -version openjdk version "11.0.10" 2021-01-19 OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.20.04) OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, sharing) mvn apache-rat:check passes On Sun, Mar 7, 2021, at 01:46, Ralph Goers wrote: > This is a vote to release Log4j 2.14.1, the next version of the Log4j 2 > project. > > Please download, test, and cast your votes on the log4j developers list. > [] +1, release the artifacts > [] -1, don't release because... > > The vote will remain open for 72 hours (or more if required). All votes are > welcome and we encourage everyone to test the release, but only Logging PMC > votes are “officially” counted. As always, at least 3 +1 votes and more > positive than negative votes are required. > > Changes in this release include: > > New Features > > • LOG4J2-2962: Enrich "map" resolver by unifying its backend with "mdc" > resolver. > • LOG4J2-2999: Replace JsonTemplateLayout resolver configurations table in > docs with sections. > • LOG4J2-2993: Support stack trace truncation in JsonTemplateLayout. > Fixed Bugs > > • LOG4J2-3033: Add log method with no parameters - i.e. it has an empty > message. > • LOG4J2-2947: Document that LogBuilder default methods do nothing. > • LOG4J2-2948: Replace HashSet with IdentityHashMap in ParameterFormatter to > detect cycles. > • LOG4J2-3028: OutputStreamManager.flushBuffer always resets the buffer, > previously the buffer was not reset after an exception. Thanks to Jakub > Kozlowski. > • LOG4J2-2981: OnStartupTriggeringPolicy would fail to cause the file to roll > over with DirectWriteTriggeringPolicy unless minSize was set to 0. > • LOG4J2-2990: Reduce garbage by using putAll when copying the ThreadContext > for SLF4J. Thanks to Diogo Monteiro. > • LOG4J2-3006: Directly create a thread instead of using the common ForkJoin > pool when initializing ThreadContextDataInjector" > • LOG4J2-2624: Allow auto-shutdown of log4j in log4j-web to be turned off and > provide a ServletContextListener "Log4jShutdownOnContextDestroyedListener" to > stop log4j. Register the listener at the top of web.xml to ensure the > shutdown happens last. Thanks to Tim Perry. > • LOG4J2-1606: Allow auto-shutdown of log4j in log4j-web to be turned off and > provide a ServletContextListener "Log4jShutdownOnContextDestroyedListener" to > stop log4j. Register the listener at the top of web.xml to ensure the > shutdown happens last. Thanks to Tim Perry. > • LOG4J2-2998: Fix truncation of excessive strings ending with a high > surrogate in JsonWriter. > • LOG4J2-2973: Rename EventTemplateAdditionalField#type (conflicting with > properties file parser) to "format". Thanks to Fabio Ricchiuti. > • LOG4J2-2972: Refactor AsyncAppender and AppenderControl for handling of > Throwables. > • LOG4J2-2985: Add eventTemplateRootObjectKey parameter to JsonTemplateLayout. > • LOG4J2-2974: Log4j would fail to initialize in Java 8 with > log4j-spring-boot. > • LOG4J2-2964: Merge packages from several Configurations in Composite > Configuration. Thanks to Valery Yatsynovich. > • LOG4J2-2961: Fix reading of JsonTemplateLayout event additional fields from > config. > • LOG4J2-2916: Avoid redundant Kafka producer instantiation causing thread > leaks. Thanks to wuqian0808. > • LOG4J2-2967: Fix JsonTemplateLayout index based parameter resolution when > messages contain too few parameters. > • LOG4J2-2976: JdbcAppender composes an incorrect INSERT statement without a > ColumnMapping element. > • LOG4J2-3014: Log4j1ConfigurationConverter on Windows produces " " at end of > every line. Thanks to Lee Breisacher, Gary Gregory. > Changes > > • LOG4J2-2893: Allow reconfiguration when Log4j 1 configuration files are > updated. > • : Update Spring dependencies to 5.3.2, Spring Boot to 2.3.6, and Spring > Cloud to Hoxton.SR9 > • : Update org.fusesource.jansi:jansi 1.17.1 -> 2.0.1. > • : Update commons-codec:commons-codec 1.14 -> 1.15. > • : Update org.apache.commons:commons-lang3 3.10 -> 3.11. > • : Update org.apache.commons:commons-pool2 2.8.1 -> 2.9.0. > • : Update org.apache.commons:commons-dbcp2 2.4.0 -> 2.8.0. > • : Update commons-io:commons-io 2.7 -> 2.8.0. > • : Update org.codehaus.groovy:* 3.0.5 -> 3.0.6. > • : Update com.fasterxml.jackson.: 2.11.2 - 2.11.3. > • : Update org.springframework:* 5.2.8.RELEASE -> 5.3.1. > • : Update junit:junit 4.13 -> 4.13.1. > • : Update org.xmlunit:* 2.7.0 -> 2.8.0. > • : Update org.assertj:assertj-core 3.14.0 -> 3.18.1. > • : Update org.awaitility:awaitility 4.0.2 -> 4.0.3. > • : Update org.codehaus.plexus:plexus-utils 3.2.0 -> 3.3.0. > • : Update MongoDB 3 plugin: org.mongodb:mongodb-driver 3.12.6 -> 3.12.7. > • : Update MongoDB 4 plugin: org.mongodb:* 4.1.0 -> 4.1.1. > • : Update org
Parallelization of PluginManager#collectPlugins()
PluginManager#collectPlugins() performs quite some package scanning
sequentially. I have the impression that this operation can simply be
parallelized as follows:
Stream
.of(inputs)
.flatMap(input -> Stream
.of(ops)
.map(op -> new Object[]{op, input}))
.parallel()
.map(opAndInput -> {
final Function op = opAndInput[0];
final Input input = opAndInput[1];
return op.accept(input);
})
.reduce(this::merge);
Here input denotes the packages and ops denote the independent sequential
steps performed in collectPlugins(). I don't know about the overhead of
this call, but the above simple effort might be worth a shot. What do you
think?
Re: Parallelization of PluginManager#collectPlugins()
Please don't use stream.parallel! It executes tasks on the ForkJoin pool which
implements work-stealing, and can result in expensive long-running operations
from other components in the system that we don't expect blocking our call
(anything using the fork-join pool, all stream.parallel callers). This is a
good read: https://stackoverflow.com/a/54581148/7182570
On Mon, Mar 8, 2021, at 16:06, Volkan Yazıcı wrote:
> PluginManager#collectPlugins() performs quite some package scanning
> sequentially. I have the impression that this operation can simply be
> parallelized as follows:
>
> Stream
> .of(inputs)
> .flatMap(input -> Stream
> .of(ops)
> .map(op -> new Object[]{op, input}))
> .parallel()
> .map(opAndInput -> {
> final Function op = opAndInput[0];
> final Input input = opAndInput[1];
> return op.accept(input);
> })
> .reduce(this::merge);
>
> Here input denotes the packages and ops denote the independent sequential
> steps performed in collectPlugins(). I don't know about the overhead of
> this call, but the above simple effort might be worth a shot. What do you
> think?
>
New plugin annotation: @PluginOrder
I am working on migrating statically bound JsonTemplateLayout resolvers to plugins. There certain types of resolvers need to have a single implementation. When a user provides a custom plugin which implements the same footprint as the singleton one, I need a way to prefer one over another. I am not sure if this is the right way of thinking, but... if I'd be doing this in Spring, I'd probably use @Primary or @Order. I can introduce my own @PluginOrder annotation, but I had the impression that this might have a wider audience. What do you think?
Re: Parallelization of PluginManager#collectPlugins()
Do we have a Log4j-specific ExecutorService somewhere we can hook into
PluginManager?
On Mon, Mar 8, 2021 at 10:24 PM Carter Kozak wrote:
> Please don't use stream.parallel! It executes tasks on the ForkJoin pool
> which implements work-stealing, and can result in expensive long-running
> operations from other components in the system that we don't expect
> blocking our call (anything using the fork-join pool, all stream.parallel
> callers). This is a good read:
> https://stackoverflow.com/a/54581148/7182570
>
> On Mon, Mar 8, 2021, at 16:06, Volkan Yazıcı wrote:
> > PluginManager#collectPlugins() performs quite some package scanning
> > sequentially. I have the impression that this operation can simply be
> > parallelized as follows:
> >
> > Stream
> > .of(inputs)
> > .flatMap(input -> Stream
> > .of(ops)
> > .map(op -> new Object[]{op, input}))
> > .parallel()
> > .map(opAndInput -> {
> > final Function op = opAndInput[0];
> > final Input input = opAndInput[1];
> > return op.accept(input);
> > })
> > .reduce(this::merge);
> >
> > Here input denotes the packages and ops denote the independent sequential
> > steps performed in collectPlugins(). I don't know about the overhead of
> > this call, but the above simple effort might be worth a shot. What do you
> > think?
> >
>
Re: New plugin annotation: @PluginOrder
There's an @Order annotation that's only supported for ConfigurationFactory plugins that you could extend for arbitrary plugins perhaps? On Mon, 8 Mar 2021 at 15:24, Volkan Yazıcı wrote: > > I am working on migrating statically bound JsonTemplateLayout resolvers to > plugins. There certain types of resolvers need to have a single > implementation. When a user provides a custom plugin which implements the > same footprint as the singleton one, I need a way to prefer one over > another. I am not sure if this is the right way of thinking, but... if I'd > be doing this in Spring, I'd probably use @Primary or @Order. I can > introduce my own @PluginOrder annotation, but I had the impression that > this might have a wider audience. What do you think?
Re: Parallelization of PluginManager#collectPlugins()
We do have a thread pool factory that decorates thread names with
"log4j" IIRC...
On Mon, Mar 8, 2021 at 4:29 PM Volkan Yazıcı wrote:
>
> Do we have a Log4j-specific ExecutorService somewhere we can hook into
> PluginManager?
>
> On Mon, Mar 8, 2021 at 10:24 PM Carter Kozak wrote:
>
> > Please don't use stream.parallel! It executes tasks on the ForkJoin pool
> > which implements work-stealing, and can result in expensive long-running
> > operations from other components in the system that we don't expect
> > blocking our call (anything using the fork-join pool, all stream.parallel
> > callers). This is a good read:
> > https://stackoverflow.com/a/54581148/7182570
> >
> > On Mon, Mar 8, 2021, at 16:06, Volkan Yazıcı wrote:
> > > PluginManager#collectPlugins() performs quite some package scanning
> > > sequentially. I have the impression that this operation can simply be
> > > parallelized as follows:
> > >
> > > Stream
> > > .of(inputs)
> > > .flatMap(input -> Stream
> > > .of(ops)
> > > .map(op -> new Object[]{op, input}))
> > > .parallel()
> > > .map(opAndInput -> {
> > > final Function op = opAndInput[0];
> > > final Input input = opAndInput[1];
> > > return op.accept(input);
> > > })
> > > .reduce(this::merge);
> > >
> > > Here input denotes the packages and ops denote the independent sequential
> > > steps performed in collectPlugins(). I don't know about the overhead of
> > > this call, but the above simple effort might be worth a shot. What do you
> > > think?
> > >
> >
Re: New plugin annotation: @PluginOrder
Thanks Matt! I have indeed spotted the following line: // TODO: 2.2+ use this annotation for all @Plugin classes Right now I am sitting on the fence whether I shall address my issue with extending the plugin system or rather implement a MySingletonBeanFactory interface extending from Comparable and @Plugin-annotate those factories. Food for thought... It is 22:50 here, time to sleep. On Mon, Mar 8, 2021 at 10:42 PM Matt Sicker wrote: > There's an @Order annotation that's only supported for > ConfigurationFactory plugins that you could extend for arbitrary > plugins perhaps? > > On Mon, 8 Mar 2021 at 15:24, Volkan Yazıcı > wrote: > > > > I am working on migrating statically bound JsonTemplateLayout resolvers > to > > plugins. There certain types of resolvers need to have a single > > implementation. When a user provides a custom plugin which implements the > > same footprint as the singleton one, I need a way to prefer one over > > another. I am not sure if this is the right way of thinking, but... if > I'd > > be doing this in Spring, I'd probably use @Primary or @Order. I can > > introduce my own @PluginOrder annotation, but I had the impression that > > this might have a wider audience. What do you think? >
Re: Parallelization of PluginManager#collectPlugins()
I am not sure if spawning up and tearing down a new thread pool for each
call would be a good idea.
On Mon, Mar 8, 2021 at 10:46 PM Gary Gregory wrote:
> We do have a thread pool factory that decorates thread names with
> "log4j" IIRC...
>
> On Mon, Mar 8, 2021 at 4:29 PM Volkan Yazıcı
> wrote:
> >
> > Do we have a Log4j-specific ExecutorService somewhere we can hook into
> > PluginManager?
> >
> > On Mon, Mar 8, 2021 at 10:24 PM Carter Kozak wrote:
> >
> > > Please don't use stream.parallel! It executes tasks on the ForkJoin
> pool
> > > which implements work-stealing, and can result in expensive
> long-running
> > > operations from other components in the system that we don't expect
> > > blocking our call (anything using the fork-join pool, all
> stream.parallel
> > > callers). This is a good read:
> > > https://stackoverflow.com/a/54581148/7182570
> > >
> > > On Mon, Mar 8, 2021, at 16:06, Volkan Yazıcı wrote:
> > > > PluginManager#collectPlugins() performs quite some package scanning
> > > > sequentially. I have the impression that this operation can simply be
> > > > parallelized as follows:
> > > >
> > > > Stream
> > > > .of(inputs)
> > > > .flatMap(input -> Stream
> > > > .of(ops)
> > > > .map(op -> new Object[]{op, input}))
> > > > .parallel()
> > > > .map(opAndInput -> {
> > > > final Function op = opAndInput[0];
> > > > final Input input = opAndInput[1];
> > > > return op.accept(input);
> > > > })
> > > > .reduce(this::merge);
> > > >
> > > > Here input denotes the packages and ops denote the independent
> sequential
> > > > steps performed in collectPlugins(). I don't know about the overhead
> of
> > > > this call, but the above simple effort might be worth a shot. What
> do you
> > > > think?
> > > >
> > >
>
Re: New plugin annotation: @PluginOrder
> On Mar 8, 2021, at 2:52 PM, Volkan Yazıcı wrote: > > Thanks Matt! I have indeed spotted the following line: > > // TODO: 2.2+ use this annotation for all @Plugin classes > > Right now I am sitting on the fence whether I shall address my issue with > extending the plugin system or rather implement a MySingletonBeanFactory > interface extending from Comparable and > @Plugin-annotate those factories. Food for thought... It is 22:50 here, > time to sleep. That is the time I do my best work. ;-) I’m not sure why you would want to annotate the factory. Generally the actual plugin class is annotated. The component that uses plugins finds the plugins for its category. Each Plugin category may use a different mechanism to load the plugin. For example, Lookups are created just by invoking new on the class, PatternConverters require a static newInstance method in the class, while core plugins require a PluginBuilder or static PluginFactory method. Basically, the plugin system just provides you the information about the plugin. It never actually creates one. Ralph
Re: Parallelization of PluginManager#collectPlugins()
Notice that class path scanning only occurs when the following evaluates to
true:
if (builtInPlugins.isEmpty()) {
We encourage the value of this to always be false. I think it would be good to
log an info message indicating that package scanning is taking place and the
application should be corrected.
Ralph
> On Mar 8, 2021, at 2:06 PM, Volkan Yazıcı wrote:
>
> PluginManager#collectPlugins() performs quite some package scanning
> sequentially. I have the impression that this operation can simply be
> parallelized as follows:
>
> Stream
>.of(inputs)
>.flatMap(input -> Stream
>.of(ops)
>.map(op -> new Object[]{op, input}))
>.parallel()
>.map(opAndInput -> {
>final Function op = opAndInput[0];
>final Input input = opAndInput[1];
>return op.accept(input);
>})
>.reduce(this::merge);
>
> Here input denotes the packages and ops denote the independent sequential
> steps performed in collectPlugins(). I don't know about the overhead of
> this call, but the above simple effort might be worth a shot. What do you
> think?
Re: Parallelization of PluginManager#collectPlugins()
The ThreadPool for ConfigurationScheduler is tied to the Configuration. So it
isn’t created until after plugins are loaded (its name should make that
obvious). Also, the number of threads it configures the thread pool for is
determined based on what it finds in the configuration. It is quite possible
that the ConfigurationScheduler might not even create an ExecutorService if it
determines there are no scheduled items configured.
Ralph
> On Mar 8, 2021, at 2:46 PM, Gary Gregory wrote:
>
> We do have a thread pool factory that decorates thread names with
> "log4j" IIRC...
>
> On Mon, Mar 8, 2021 at 4:29 PM Volkan Yazıcı wrote:
>>
>> Do we have a Log4j-specific ExecutorService somewhere we can hook into
>> PluginManager?
>>
>> On Mon, Mar 8, 2021 at 10:24 PM Carter Kozak wrote:
>>
>>> Please don't use stream.parallel! It executes tasks on the ForkJoin pool
>>> which implements work-stealing, and can result in expensive long-running
>>> operations from other components in the system that we don't expect
>>> blocking our call (anything using the fork-join pool, all stream.parallel
>>> callers). This is a good read:
>>> https://stackoverflow.com/a/54581148/7182570
>>>
>>> On Mon, Mar 8, 2021, at 16:06, Volkan Yazıcı wrote:
PluginManager#collectPlugins() performs quite some package scanning
sequentially. I have the impression that this operation can simply be
parallelized as follows:
Stream
.of(inputs)
.flatMap(input -> Stream
.of(ops)
.map(op -> new Object[]{op, input}))
.parallel()
.map(opAndInput -> {
final Function op = opAndInput[0];
final Input input = opAndInput[1];
return op.accept(input);
})
.reduce(this::merge);
Here input denotes the packages and ops denote the independent sequential
steps performed in collectPlugins(). I don't know about the overhead of
this call, but the above simple effort might be worth a shot. What do you
think?
>>>
>
